Binesh Sadanandan PhD Dissertation Companion
Read thesis

Thrust 3 · Chapter 5

Mitigation

Can paraphrase flips be reduced without degrading observed accuracy?

The answer commits at layer 16, and a candidate two-stage account runs from Feature 3818 at layer 17 to Feature 12139 at layer 29. Both are exploratory. Ablating the single feature restores the original answer in only 6 of 76 flips, and the layers the diagnosis pointed at turn out not to be the best place to intervene: early layers beat them. The adapter still works. A targeted LoRA on layers 15 to 19, touching 0.10% of parameters, cuts the pairwise flip rate from 8.5% to 3.5% on a clean patient-disjoint set, about 59%, with no observed accuracy reduction. A controlled probe (FlipLens) then isolates where the sensitivity comes from: holding architecture, parameters, seed and evaluation fixed and varying only the training-phrasing distribution moves the flip rate from 4.8% under full paraphrase coverage to 67.1% under a single phrasing and 88.4% when a phrasing register is tied to the answer (Cliff's delta 1.00), which identifies the training data as a cause rather than a correlate. Part of that 4.8% is recognition of familiar wording rather than invariance: scored only on the 24 phrasings withheld from training, augmented rises to 26.6% against canonical's 65.9% and adversarial's 87.4%, and 26.6% against 65.9% is the honest size of the lever.

How this thrust was run

Residual-stream transplants and sparse-autoencoder feature analysis on MedGemma-4B to locate candidate mechanisms, followed by targeted low-rank adaptation of layers 15-19 evaluated on a patient-disjoint fleet (5 seeds) against a full-layer LoRA baseline.

Datasets
MIMIC-CXR, PadChest
Models
MedGemma-4B, Targeted LoRA, Full LoRA
Thesis tables
tab:thrust3_main_results, tab:thrust3_layer_ablation, tab:thrust3_padchest_transfer, tab:pd_safety_audit

Exploratory A candidate two-stage mechanistic account localizes paraphrase sensitivity: Feature 3818 (layer 17) behaves as an operator/register feature and Feature 12139 (layer 29) as a downstream answer-selection feature, with layer 16 as a general answer-commitment locus.

Mixed Partly. Targeted LoRA on layers 15 to 19 touches 0.1% of parameters and cuts pairwise flips from 8.5% to 3.5% over five seeds, about 59%. McNemar p < 0.002 in every seed, and no observed accuracy reduction on a patient- and image-disjoint split. But it does not preserve visual grounding: both adapters raise text-only agreement from 53.5% to about 77%.

Negative result No: the best intervention layer does not match the diagnosis layer. Ablating early layers 0-10 reduces margin difference to 0.26 (86% reduction), beating the mechanistically identified layers 15-19 (0.38, 80%) by six points.

The causality experiments

Localization shows where the flip is decided. A separate study asks why it exists: a small MedGemma-faithful probe, trained under three phrasing regimes with everything else held fixed, shows the flip rate is set by the training-phrasing distribution.

Open the causality study

Primary results

3.5%

pairwise

Targeted LoRA cuts the flip rate from 8.5% to 3.5%, about 59%

Training a small adapter on five transformer layers cuts the paraphrase flip rate by roughly 59%, from 8.5% to 3.5%, on a held-out test that shares no patient and no image with training. The effect is significant in every one of five seeds.

Targeted LoRA · MIMIC-CXR · n = 238 · +/- 0.6 (standard deviation over five seeds)

Source, denominator, and limits
Metric
pairwise paraphrase flip rate, presence-of-finding endpoint
Denominator
pairwise (per question-paraphrase pair)
Sample
n = 238
Model
Targeted LoRA
Dataset
MIMIC-CXR
Split
patient-disjoint
Comparison
8.5% base pairwise flip
Uncertainty
+/- 0.6 (standard deviation over five seeds)
Thesis
Chapter 5, tab:thrust3_main_results
Source artifact
results/lora_fleet_patient_disjoint/eval_targeted_s{42,123,456,789,2024}.json (mean over five seeds)
Last verified
2026-07-15
Open in Evidence Explorer

0.25 pp

question-level

No observed accuracy reduction from the targeted adapter

Accuracy on the held-out presence endpoint moves from 84.5% to 84.7% (+/- 1.0), a paired per-question difference of +0.25 percentage points with a 95% interval of [-1.00, +1.51]. There is no observed accuracy reduction: the flip-rate gain does not come out of correctness.

Targeted LoRA · MIMIC-CXR · n = 238 · 95% t interval [-1.00, +1.51] percentage points across five seeds

Source, denominator, and limits
Metric
paired per-question accuracy difference (targeted LoRA minus base), presence-of-finding endpoint
Denominator
question-level (per question)
Sample
n = 238
Model
Targeted LoRA
Dataset
MIMIC-CXR
Split
patient-disjoint
Comparison
84.5% base accuracy to 84.7% +/- 1.0 adapted; Full LoRA instead gives up about 1.2 points (83.3% +/- 0.2).
Uncertainty
95% t interval [-1.00, +1.51] percentage points across five seeds
Thesis
Chapter 5, tab:thrust3_main_results
Source artifact
results/lora_fleet_patient_disjoint/eval_targeted_s{42,123,456,789,2024}.json (mean over five seeds)
Last verified
2026-07-15
Open in Evidence Explorer

76.8%

question-level

The adapter buys consistency by leaning harder on the question text

On the clean patient-disjoint audit, text-only agreement rises from 53.5% for the base model to 76.8% (+/- 3.1) after targeted adaptation: the adapted model returns its original answer with the image deleted 23 points more often. Both adapters get much of their new consistency by relying more on the question text, with no steadier use of the image. Targeted and Full LoRA are statistically tied on image reliance (76.8% +/- 3.1 vs 76.6% +/- 1.7).

Targeted LoRA · MIMIC-CXR · n = 241 · +/- 3.1 (standard deviation over five seeds)

Source, denominator, and limits
Metric
text-only agreement (share of predictions unchanged when the image is removed), presence-of-finding endpoint
Denominator
question-level (per question)
Sample
n = 241
Model
Targeted LoRA
Dataset
MIMIC-CXR
Split
patient-disjoint
Comparison
53.5% for the base model, a 23-point increase; Full LoRA reaches a statistically indistinguishable 76.6% +/- 1.7. Swap sensitivity also falls slightly (base 22.0%, targeted 19.9% +/- 1.2, full 20.9% +/- 1.3).
Uncertainty
+/- 3.1 (standard deviation over five seeds)
Thesis
Chapter 5, tab:pd_safety_audit
Source artifact
results/lora_fleet_patient_disjoint/audit_targeted_s{42,123,456,789,2024}.json (mean over five seeds)
Last verified
2026-07-15
Open in Evidence Explorer

73%

pairwise

Transplanting the residual at layer 16 flips the answer on 73% of pairs

Swapping a single layer's residual stream from a 'detecting' pair member into a 'missing' one flips the model's committed answer on 73% of pairs at layer 16, up from 35% at layer 15 and 8% at layer 14, reaching 100% by layer 20. The answer commits in a narrow band at layer 16 (median commit layer 16, 95% bootstrap CI [16, 16]). An image-token control never exceeds 0.0007, so this is not a generic patching artifact.

MedGemma-4B · PadChest · n = 1,396 · median per-pair commit layer 16, 95% bootstrap CI [16, 16]

Source, denominator, and limits
Metric
answer flip rate under single-layer residual transplant at layer 16
Denominator
pairwise (per question-paraphrase pair)
Sample
n = 1,396
Model
MedGemma-4B
Dataset
PadChest
Split
eval
Comparison
8% at layer 14 and 35% at layer 15, rising to 100% by layer 20; 71% of pairs commit within layers 15-17.
Uncertainty
median per-pair commit layer 16, 95% bootstrap CI [16, 16]
Thesis
Chapter 5, fig:t3_commit_curve
Source artifact
dissertation/chapters/04_thrust3.tex
Last verified
2026-07-15
Open in Evidence Explorer

0.26

question-level

The best layers to fix are not the layers where the mechanism lives

Adapting early layers (0-10) reduces the margin difference to 0.26, an 86% cut from the 1.87 baseline, while the mechanistically indicated layers 15-19 reach only 0.38 (80%). Even a randomly chosen block (layers 5-9) does better at 0.30. The SAE analysis says where paraphrase sensitivity manifests, but the best place to intervene is upstream, before the register-sensitive representation forms. RQ3.3 is answered no.

Targeted LoRA · MIMIC-CXR · n = 355

Source, denominator, and limits
Metric
mean margin difference (yes-minus-no logit gap between original and paraphrase) after adapting layers 0-10
Denominator
question-level (per question)
Sample
n = 355
Model
Targeted LoRA
Dataset
MIMIC-CXR
Split
eval
Comparison
0.38 for the mechanistically indicated layers 15-19
Uncertainty
not reported for this value
Thesis
Chapter 5, tab:thrust3_layer_ablation
Source artifact
dissertation/tables/thrust3/table_layer_ablation.tex
Last verified
2026-07-15
Open in Evidence Explorer

Charts from this thrust

Open any of these in the evidence explorer to filter by model, dataset, or metric.

LoRA flip reduction, seed by seed

Clean patient- and image-disjoint MIMIC-CXR test, presence-of-finding endpoint: 238 questions / 991 pairs, 236 held-out subjects, zero subject and zero image overlap with training. Targeted LoRA seeds 42/123/456/789/2024; Full LoRA seeds 42/123/456.

Accuracy on the clean patient-disjoint endpoint, with 95% Wilson intervals

Presence-of-finding endpoint, n=238 questions per arm on the patient- and image-disjoint MIMIC-CXR test (236 held-out subjects, zero subject and zero image overlap with training). One point per adapter seed.

The diagnosis layer is not the best intervention layer

Five LoRA layer-range configurations (rank 16, alpha=32, combined loss, lambda=1.0) scored on the 355-question MIMIC-CXR validation split against the 1.87 no-adapter baseline.

Cite


  

  

     Permanent link
    
  




  

    
    
    
    
  

  

    

    
Type to search. Press Escape to close.