The defense story
Ask the same question twice.
One small habit of distrust runs through this dissertation: take a question the model answers, say the same thing in different words, and watch whether the answer holds. The defense tells that story in five thrusts. Measure the failure, diagnose it, explain and fix it, build the safety screen, then decide what deployment can trust. Every number below links into the evidence it comes from.
Thrust 1 · Measure
Flips are common, and the obvious suspects do not explain them.
How often do equivalent questions change model answers?
PSF-Med evaluates six medical vision-language models on 92,856 evaluated paraphrase pairs built from 26,850 chest X-ray questions spanning three countries. Pairwise flip rates on the binary yes/no subset range from 6.4% to 54.7% (an 8.5x spread), and scale does not buy consistency. Two model-dataset cells with degenerate yes-bias (>98% positive on originals) are set aside, and no consistency is claimed for them.
Thrust 2 · Diagnose
The steadier the answers, the less the model looks at the scan.
Does consistency mean the model is using the image?
Three backends are compared on a curated 107-pair set. The most consistent model, LLaVA-Rad at 6.5% flips, gives the same answer on nearly every question when you take the image away. The least consistent, MedGemma-4B at 42.1% here, is the one that reacts most when you swap the image for another. Across these three, consistency and text-reliance rise together. The set predates the equivalence audit, so these figures are illustrative. They are not estimates.
Thrust 3 · Explain & fix
A candidate circuit, a 59% fix, and a cause in the training data.
Can paraphrase flips be reduced without degrading observed accuracy?
The answer commits at layer 16, and a candidate two-stage account runs from Feature 3818 at layer 17 to Feature 12139 at layer 29. Both are exploratory. Ablating the single feature restores the original answer in only 6 of 76 flips, and the layers the diagnosis pointed at turn out not to be the best place to intervene: early layers beat them. The adapter still works. A targeted LoRA on layers 15 to 19, touching 0.10% of parameters, cuts the pairwise flip rate from 8.5% to 3.5% on a clean patient-disjoint set, about 59%, with no observed accuracy reduction. A controlled probe (FlipLens) then isolates where the sensitivity comes from: holding architecture, parameters, seed and evaluation fixed and varying only the training-phrasing distribution moves the flip rate from 4.8% under full paraphrase coverage to 67.1% under a single phrasing and 88.4% when a phrasing register is tied to the answer (Cliff's delta 1.00), which identifies the training data as a cause rather than a correlate. Part of that 4.8% is recognition of familiar wording rather than invariance: scored only on the 24 phrasings withheld from training, augmented rises to 26.6% against canonical's 65.9% and adversarial's 87.4%, and 26.6% against 65.9% is the honest size of the lever.
Figures use an illustrative, licence-cleared chest X-ray (Wikimedia Commons); the underlying pairs are credentialed data and are not shown.
Open this thrust Filter the evidence The causality experiments
Thrust 4 · Safety-evaluate
Consistency without grounding is the safety paradox.
Do consistency improvements produce safer, more image-dependent behavior?
A four-quadrant audit crossing consistency with image reliance shows that, averaged across ten model-dataset settings, 81% of each model's consistent predictions are image-invariant (range 45-100%). Consistency-optimized adapters concentrate predictions in the Dangerous quadrant (consistent but text-reliant), reaching 84.1% for Targeted LoRA on PadChest, though Chapter 6 qualifies that placement: Targeted LoRA also has the highest image-swap sensitivity of the LoRA variants (31.5% on the same 861 questions), so the text-only-agreement criterion can overcount on PadChest, where high finding base rates let the text prior and the image agree. Correctness is a separate axis: on PadChest the text-reliant cell is often more accurate than the grounded one, so the screen flags behavior. It does not flag per-prediction error.
Thrust 5 · Deploy
Abstaining buys safety.
Can uncertainty and audit signals identify predictions that should be escalated?
One forward pass is enough to rank flip risk. Predictive entropy predicts paraphrase flips at an AUROC of 0.823 and errors at 0.862 for Targeted LoRA on the PadChest flip bank, and it carries across architectures: LLaVA-Rad LoRA reaches 0.830 on PadChest and 0.905 on MIMIC. The gate is an offline readiness audit: its region term needs a radiologist's bounding box that no undiagnosed patient has, so that term cannot run online, though the MIMIC configuration, where the rule already reduces to paraphrase agreement plus an image-swap check, is deployable as written. And what it admits is mostly what the question text already answers: on the slice where text and image disagree, Targeted LoRA scores 2.9%.
The story in full
The dissertation carries the full argument; the papers carry the peer-reviewed pieces; the evidence explorer carries every number with its sample and source.