Thrust 3 · Chapter 5

Mitigation

Can paraphrase flips be reduced without degrading observed accuracy?

The answer commits at layer 16, and a candidate two-stage account runs from Feature 3818 at layer 17 to Feature 12139 at layer 29. Both are exploratory. Ablating the single feature restores the original answer in only 6 of 76 flips, and the layers the diagnosis pointed at turn out not to be the best place to intervene: early layers beat them. The adapter still works. A targeted LoRA on layers 15 to 19, touching 0.10% of parameters, cuts the pairwise flip rate from 8.5% to 3.5% on a clean patient-disjoint set, about 59%, with no observed accuracy reduction.

How I ran it

Residual-stream transplants and sparse-autoencoder feature analysis on MedGemma-4B to locate candidate mechanisms, followed by targeted low-rank adaptation of layers 15-19 evaluated on a patient-disjoint fleet (5 seeds) against a full-layer LoRA baseline.

Datasets: MIMIC-CXR, PadChest
Models: MedGemma-4B, Targeted LoRA, Full LoRA
Thesis tables: tab:thrust3_main_results, tab:thrust3_layer_ablation, tab:thrust3_padchest_transfer, tab:pd_safety_audit
Papers: Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
Code: UNHSAILLab/medical-vlm-paraphrase-consistency

The limitation that matters most

The mitigation does not preserve grounding: both adapters raise text-only agreement from 53.5% to about 77% and are statistically tied on image reliance, so they buy consistency by leaning harder on the question text; Targeted LoRA is the cheaper and more accurate route to the same consistency, not a more grounded one.

Exploratory A candidate two-stage mechanistic account localizes paraphrase sensitivity: Feature 3818 (layer 17) behaves as an operator/register feature and Feature 12139 (layer 29) as a downstream answer-selection feature, with layer 16 as a general answer-commitment locus.

Mixed Partly. Targeted LoRA on layers 15 to 19 touches 0.1% of parameters and cuts pairwise flips from 8.5% to 3.5% over five seeds, about 59%. McNemar p < 0.002 in every seed, and no observed accuracy reduction on a patient- and image-disjoint split. But it does not preserve visual grounding: both adapters raise text-only agreement from 53.5% to about 77%.

Negative result No: the best intervention layer does not match the diagnosis layer. Ablating early layers 0-10 reduces margin difference to 0.26 (86% reduction), beating the mechanistically identified layers 15-19 (0.38, 80%) by six points.

Primary results

3.5%

pairwise

Targeted LoRA cuts the flip rate from 8.5% to 3.5%, about 59%

Training a small adapter on five transformer layers cuts the paraphrase flip rate by roughly 59%, from 8.5% to 3.5%, on a held-out test that shares no patient and no image with training. The effect is significant in every one of five seeds.

Targeted LoRA · MIMIC-CXR · n = 238 · +/- 0.6 (standard deviation over five seeds)

Source, denominator, and limits

How far this goesDo not restate this as 79.5% (an old margin-difference reduction) or 69.6% (an older flip-rate reduction); on the clean patient-disjoint rerun the headline is about 59%. The base rate of 8.5% is itself lower than the 14.6% reported before the operator-preserving cleanup, because that earlier figure counted operator-changing pairs whose flips were correct behaviour. The split is disjoint by patient but NOT by institution: training and test are both MIMIC-CXR, so this does not demonstrate transfer to another hospital. Most importantly, this consistency gain does not come from better grounding, and Full LoRA reaches a slightly lower flip rate (2.9% +/- 0.3) at 7x the parameters. The displayed value is the mean over five seeds: an individual seed file differs (seed 42 alone gives 2.7% pairwise flip and 85.3% accuracy).

Metric: pairwise paraphrase flip rate, presence-of-finding endpoint
Denominator: pairwise (per question-paraphrase pair)
Sample: n = 238
Population: Patient-disjoint mitigation endpoint (238 questions)
Model: Targeted LoRA
Dataset: MIMIC-CXR
Split: patient-disjoint
Comparison: 8.5% base pairwise flip
Uncertainty: +/- 0.6 (standard deviation over five seeds)
Thesis: Chapter 5, tab:thrust3_main_results
Paper: Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
Source artifact: results/lora_fleet_patient_disjoint/eval_targeted_s{42,123,456,789,2024}.json (mean over five seeds)
Last verified: 2026-07-15

Open in Evidence Explorer

0.25 pp

question-level

No observed accuracy reduction from the targeted adapter

Accuracy on the held-out presence endpoint moves from 84.5% to 84.7% (+/- 1.0), a paired per-question difference of +0.25 percentage points with a 95% interval of [-1.00, +1.51]. There is no observed accuracy reduction: the flip-rate gain does not come out of correctness.

Targeted LoRA · MIMIC-CXR · n = 238 · 95% t interval [-1.00, +1.51] percentage points across five seeds

Source, denominator, and limits

How far this goesThe interval excludes a large accuracy cost but does NOT establish exact parity: the data are consistent with anything from a one-point loss to a 1.5-point gain. This is why the wording is 'no observed accuracy reduction' rather than 'no accuracy cost' or 'accuracy preserved' — the study was never powered for an equivalence claim, and none is made. The endpoint is also a single presence-of-finding task on one institution's data; accuracy here is not a clinical performance claim. The displayed value is the mean over five seeds: an individual seed file differs (seed 42 alone gives 2.7% pairwise flip and 85.3% accuracy).

Metric: paired per-question accuracy difference (targeted LoRA minus base), presence-of-finding endpoint
Denominator: question-level (per question)
Sample: n = 238
Population: Patient-disjoint mitigation endpoint (238 questions)
Model: Targeted LoRA
Dataset: MIMIC-CXR
Split: patient-disjoint
Comparison: 84.5% base accuracy to 84.7% +/- 1.0 adapted; Full LoRA instead gives up about 1.2 points (83.3% +/- 0.2).
Uncertainty: 95% t interval [-1.00, +1.51] percentage points across five seeds
Thesis: Chapter 5, tab:thrust3_main_results
Paper: Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
Source artifact: results/lora_fleet_patient_disjoint/eval_targeted_s{42,123,456,789,2024}.json (mean over five seeds)
Last verified: 2026-07-15

Open in Evidence Explorer

76.8%

question-level

The adapter buys consistency by leaning harder on the question text

This is the reframe that changes the story. On the clean patient-disjoint audit, text-only agreement rises from 53.5% for the base model to 76.8% (+/- 3.1) after targeted adaptation: the adapted model returns its original answer with the image deleted 23 points more often. Both adapters get much of their new consistency by relying more on the question text, not by making steadier use of the image. Targeted and Full LoRA are statistically tied on image reliance (76.8% +/- 3.1 vs 76.6% +/- 1.7).

Targeted LoRA · MIMIC-CXR · n = 241 · +/- 3.1 (standard deviation over five seeds)

Source, denominator, and limits

How far this goesThe claim that targeted adaptation preserves image dependence relative to full adaptation is NOT supported on the clean adapters: the two are tied on both image-reliance measures, and both move the model toward the text. The older ordering (targeted 66.3% text-only / 32.7% swap vs full 80.6% / 14.3%) is specific to the original-pipeline adapters and does not replicate. Targeted LoRA remains the recommendation only because at tied consistency it keeps accuracy (84.3% vs 82.4%) at 0.1% of parameters instead of 0.72%: it is the cheaper and more accurate route to the same consistency, not a more grounded one. This audit is the reason a low flip rate cannot be read as evidence of visual reasoning. The displayed value is the mean over five seeds (per-seed: 78.8, 80.5, 74.3, 77.6, 73.0).

Metric: text-only agreement (share of predictions unchanged when the image is removed), presence-of-finding endpoint
Denominator: question-level (per question)
Sample: n = 241
Population: Patient-disjoint mitigation endpoint (238 questions)
Model: Targeted LoRA
Dataset: MIMIC-CXR
Split: patient-disjoint
Comparison: 53.5% for the base model, a 23-point increase; Full LoRA reaches a statistically indistinguishable 76.6% +/- 1.7. Swap sensitivity also falls slightly (base 22.0%, targeted 19.9% +/- 1.2, full 20.9% +/- 1.3).
Uncertainty: +/- 3.1 (standard deviation over five seeds)
Thesis: Chapter 5, tab:pd_safety_audit
Paper: Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
Source artifact: results/lora_fleet_patient_disjoint/audit_targeted_s{42,123,456,789,2024}.json (mean over five seeds)
Last verified: 2026-07-15

Open in Evidence Explorer

73%

pairwise

Transplanting the residual at layer 16 flips the answer on 73% of pairs

Swapping a single layer's residual stream from a 'detecting' pair member into a 'missing' one flips the model's committed answer on 73% of pairs at layer 16, up from 35% at layer 15 and 8% at layer 14, reaching 100% by layer 20. The answer commits in a narrow band at layer 16 (median commit layer 16, 95% bootstrap CI [16, 16]). An image-token control never exceeds 0.0007, so this is not a generic patching artifact.

MedGemma-4B · PadChest · n = 1,396 · median per-pair commit layer 16, 95% bootstrap CI [16, 16]

Source, denominator, and limits

How far this goesThis is a GENERAL answer-commitment locus, not a paraphrase-specific mechanism: the same layers 16-17 band governs left/right localization (on a small 7-pair test), so it localizes where this model commits to any answer, not where paraphrase sensitivity uniquely lives. The pool is 94% positive (1,314 yes vs 82 no), so the result is established on predominantly positive items; 107 of 1,396 pairs (7.7%) carry an anatomical qualifier on one side only and are not strict paraphrases. The transplant was run in a separate codebase and the final pair-pool script was NOT retained, so the pool is reconstructible by inspection but not re-executable. Layer 16 (the causal commit) must not be conflated with layer 17 (where Feature 3818 sits), though the single-layer offset falls inside the same band.

Metric: answer flip rate under single-layer residual transplant at layer 16
Denominator: pairwise (per question-paraphrase pair)
Sample: n = 1,396
Population: Answer-commitment transplant pool (1,396 pairs)
Model: MedGemma-4B
Dataset: PadChest
Split: eval
Comparison: 8% at layer 14 and 35% at layer 15, rising to 100% by layer 20; 71% of pairs commit within layers 15-17.
Uncertainty: median per-pair commit layer 16, 95% bootstrap CI [16, 16]
Thesis: Chapter 5, fig:t3_commit_curve
Paper: Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
Source artifact: dissertation/chapters/04_thrust3.tex
Last verified: 2026-07-15

Open in Evidence Explorer

0.26

question-level

The best layers to fix are not the layers where the mechanism lives

Adapting early layers (0-10) reduces the margin difference to 0.26, an 86% cut from the 1.87 baseline, while the mechanistically indicated layers 15-19 reach only 0.38 (80%). Even a randomly chosen block (layers 5-9) does better at 0.30. The SAE analysis says where paraphrase sensitivity manifests, but the best place to intervene is upstream, before the register-sensitive representation forms. RQ3.3 is answered no.

Targeted LoRA · MIMIC-CXR · n = 355

Source, denominator, and limits

How far this goesThe comparison is not layer-count-matched: early (0-10) spans 11 layers against 5 for middle (15-19), so part of the 6-point gap may buy capacity rather than location. The endpoint is margin difference on a validation split, not flip rate on the held-out test, and these five configurations come from the ORIGINAL training pipeline (500 pairs, 3 epochs), not the clean patient-disjoint fleet that produces the headline 59% reduction, so they are reported for relative patterns only. A separate 50-config block sweep supports the same U-shaped conclusion with its global minimum at layers 1-5. Note that the deployed adapter still targets 15-19: the ablation undercuts the mechanistic rationale for that choice without changing the headline result.

Metric: mean margin difference (yes-minus-no logit gap between original and paraphrase) after adapting layers 0-10
Denominator: question-level (per question)
Sample: n = 355
Population: Layer-ablation validation split (355 questions)
Model: Targeted LoRA
Dataset: MIMIC-CXR
Split: eval
Comparison: 0.38 for the mechanistically indicated layers 15-19
Uncertainty: not reported for this value
Thesis: Chapter 5, tab:thrust3_layer_ablation
Paper: Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
Source artifact: dissertation/tables/thrust3/table_layer_ablation.tex
Last verified: 2026-07-15

Open in Evidence Explorer

Charts from this thrust

Open any of these in the evidence explorer to filter by model, dataset, or metric.

LoRA flip reduction, seed by seed

Clean patient- and image-disjoint MIMIC-CXR test, presence-of-finding endpoint: 238 questions / 991 pairs, 236 held-out subjects, zero subject and zero image overlap with training. Targeted LoRA seeds 42/123/456/789/2024; Full LoRA seeds 42/123/456.

Every adapter seed lands well below the base model's 8.5% pairwise flip rate: Targeted LoRA averages 3.5% (about a 59% reduction) and Full LoRA 2.9%, with McNemar exact p between 2.5e-7 and 1.8e-3 on every seed. The seed spread is small relative to the base-to-adapter gap.

Accuracy on the clean patient-disjoint endpoint, with 95% Wilson intervals

Presence-of-finding endpoint, n=238 questions per arm on the patient- and image-disjoint MIMIC-CXR test (236 held-out subjects, zero subject and zero image overlap with training). One point per adapter seed.

Every arm's interval overlaps every other arm's: the base model scores 84.5% and Targeted LoRA 84.7% on average, so the large flip reduction is not paid for by a large accuracy loss.

The diagnosis layer is not the best intervention layer

Five LoRA layer-range configurations (rank 16, alpha=32, combined loss, lambda=1.0) scored on the 355-question MIMIC-CXR validation split against the 1.87 no-adapter baseline.

Adapting early layers 0-10 cuts the paraphrase margin difference furthest (0.26, an 86% reduction), beating the mechanistically identified middle window 15-19 (0.38, 80%) by 6 points, and even a randomly chosen block at layers 5-9 (0.30) beats it. Where paraphrase sensitivity is diagnosed is not where it is best fixed.