Thesis map
Each thrust answers a question the previous one opened, and several of the answers are negative. The argument view below follows that chain. The chapter view shows how the dissertation is organized instead.
Nobody had measured how often a rephrased clinical question changes a medical model’s answer, at scale, on audited paraphrases.
Chapter 3
How often do equivalent questions change model answers?
PSF-Med evaluates six medical vision-language models on 92,856 evaluated paraphrase pairs built from 26,850 chest X-ray questions spanning three countries. Pairwise flip rates on the binary yes/no subset range from 6.4% to 54.7% (an 8.5x spread), and scale does not buy consistency. Two model-dataset cells with degenerate yes-bias (>98% positive on originals) are set aside, and no consistency is claimed for them.
Measurement showed some models are strikingly consistent. That raised a harder question: is consistency evidence of good reasoning, or of ignoring the image?
Chapter 4
Does consistency mean the model is using the image?
Three backends are compared on a curated 107-pair set. The most consistent model, LLaVA-Rad at 6.5% flips, gives the same answer on nearly every question when you take the image away. The least consistent, MedGemma-4B at 42.1% here, is the one that reacts most when you swap the image for another. Across these three, consistency and text-reliance rise together. The set predates the equivalence audit, so these figures are illustrative. They are not estimates.
If consistency can be ungrounded, then reducing flips is only worth doing if it does not come from leaning further on the text. That means finding where the answer is decided.
Chapter 5
Can paraphrase flips be reduced without degrading observed accuracy?
The answer commits at layer 16, and a candidate two-stage account runs from Feature 3818 at layer 17 to Feature 12139 at layer 29. Both are exploratory. Ablating the single feature restores the original answer in only 6 of 76 flips, and the layers the diagnosis pointed at turn out not to be the best place to intervene: early layers beat them. The adapter still works. A targeted LoRA on layers 15 to 19, touching 0.10% of parameters, cuts the pairwise flip rate from 8.5% to 3.5% on a clean patient-disjoint set, about 59%, with no observed accuracy reduction. A controlled probe (FlipLens) then isolates where the sensitivity comes from: holding architecture, parameters, seed and evaluation fixed and varying only the training-phrasing distribution moves the flip rate from 4.8% under full paraphrase coverage to 67.1% under a single phrasing and 88.4% when a phrasing register is tied to the answer (Cliff's delta 1.00), which identifies the training data as a cause rather than a correlate. Part of that 4.8% is recognition of familiar wording rather than invariance: scored only on the 24 phrasings withheld from training, augmented rises to 26.6% against canonical's 65.9% and adversarial's 87.4%, and 26.6% against 65.9% is the honest size of the lever.
Open this thrust Filter the evidenceThe causality experiments
The adapter cut flips. Whether that made the model safer is a separate question, and it needs a screen that tests consistency and image dependence together.
Chapter 6
Do consistency improvements produce safer, more image-dependent behavior?
A four-quadrant audit crossing consistency with image reliance shows that, averaged across ten model-dataset settings, 81% of each model's consistent predictions are image-invariant (range 45-100%). Consistency-optimized adapters concentrate predictions in the Dangerous quadrant (consistent but text-reliant), reaching 84.1% for Targeted LoRA on PadChest, though Chapter 6 qualifies that placement: Targeted LoRA also has the highest image-swap sensitivity of the LoRA variants (31.5% on the same 861 questions), so the text-only-agreement criterion can overcount on PadChest, where high finding base rates let the text prior and the image agree. Correctness is a separate axis: on PadChest the text-reliant cell is often more accurate than the grounded one, so the screen flags behavior. It does not flag per-prediction error.
If a consistent model can still be ungrounded, deployment cannot rely on consistency. Something has to flag the predictions a clinician should look at.
Chapter 7
Can uncertainty and audit signals identify predictions that should be escalated?
One forward pass is enough to rank flip risk. Predictive entropy predicts paraphrase flips at an AUROC of 0.823 and errors at 0.862 for Targeted LoRA on the PadChest flip bank, and it carries across architectures: LLaVA-Rad LoRA reaches 0.830 on PadChest and 0.905 on MIMIC. The gate is an offline readiness audit: its region term needs a radiologist's bounding box that no undiagnosed patient has, so that term cannot run online, though the MIMIC configuration, where the rule already reduces to paraphrase agreement plus an image-swap check, is deployable as written. And what it admits is mostly what the question text already answers: on the slice where text and image disagree, Targeted LoRA scores 2.9%.
| Chapter | Title | Thrust | What it establishes |
|---|---|---|---|
| 1 | Introduction | — | Frames paraphrase sensitivity in medical Vision-Language Models, states the overarching question and its five-thrust decomposition, and motivates the restriction to binary presence questions about chest X-rays. |
| 2 | A Scoping Review of Trustworthiness Evaluation in Medical Vision-Language Models | — | Surveys how the field evaluates trustworthiness in medical VLMs and shows that the practices each thrust targets, such as paraphrase invariance testing and image-reliance checks, are routinely omitted. |
| 3 | Thrust 1: Measuring Paraphrase Sensitivity in Medical VLMs | Measurement | Builds the PSF-Med benchmark (92,856 judge-filtered evaluation pairs from 26,850 questions across MIMIC-CXR, PadChest, and VinDr-CXR) and measures pairwise flip rates from 6.4% to 54.7% across six models on the binary subset, with two degenerate yes-bias cells set aside. |
| 4 | Thrust 2: Diagnosing the Origin of Paraphrase Sensitivity | Diagnosis | Uses text-only evaluation and controlled image swaps on a 107-pair diagnostic set, plus an attention-bounding-box analysis on a separate 200-pair PadChest subsample, to show that, across the three models tested, the most consistent backends rely least on the image. |
| 5 | Thrust 3: Mechanistic Diagnosis and Targeted Mitigation | Mitigation | Locates a general answer-commitment locus at layer 16 and candidate SAE features (L17/#3818, L29/#12139), then shows a targeted LoRA on layers 15-19 (0.10% of parameters) cuts the pairwise flip rate about 59% on a patient-disjoint set without observed accuracy reduction; a safety re-audit finds the adapters gain consistency by leaning harder on question text, with no grounding restored. |
| 6 | Thrust 4: Safety Evaluation | Safety | Introduces a four-quadrant audit crossing consistency with image reliance: averaged across ten model-dataset settings, 81% of consistent predictions are image-invariant, and consistency-optimized adapters concentrate predictions in the consistent-but-text-reliant quadrant. Correctness is a separate axis, so the screen is not a per-prediction safety verdict. |
| 7 | Thrust 5: Calibrated Uncertainty and Deployment Gating | Deployment | Shows a single-forward-pass entropy signal predicts paraphrase flips (AUROC 0.823) and errors (AUROC 0.862) and transfers across architectures, but the deployment gate is an offline readiness audit whose region term requires radiologist bounding boxes that are not available at inference; admitted cases skew text-answerable over grounded, and the deep ensemble fails out of distribution without early warning. |
| 8 | Conclusion | — | Synthesizes the five thrusts under the central claim that a low paraphrase-flip rate is not sufficient evidence of reliable visual reasoning, and lays out future work including a same-polarity mechanistic screen and the designed but not yet conducted clinician deployment consultation. |
| — | Appendices: Supplementary Material; Extended Model Details and Hyperparameters; Reproducibility Information | — | Supplementary analyses and tables, extended model and hyperparameter details, and reproducibility information for the experiments in Chapters 3 through 7. Published as a separate 42-page PDF. |
Low paraphrase sensitivity is insufficient evidence of grounded visual reasoning. Consistency must be evaluated together with correctness, image dependence, and uncertainty.
The mitigation cuts flips but does not preserve grounding. The mechanism is a candidate account, its circuit unproven. The deployment audit admits the cases a text prior already answers.
From the synthesis table in Chapter 8. The chip states how strongly the evidence supports the answer.
Paraphrase sensitivity is common in medical vision-language models: pairwise flip rates on the binary yes/no subset span 6.4% to 54.7% across six models and three chest X-ray datasets, an 8.5x spread.
flip-rate-range · flip-rate-medgemma4b-mimic · flip-rate-medgemma4b-padchest · benchmark-eval-pairs
Scale does not buy paraphrase consistency: MedGemma-27B is the most consistent of the three MedGemma variants on MIMIC (6.4%) and VinDr (8.1%) but not on PadChest, where MedGemma-4B edges it (13.4% vs 13.9%), and no stable consistency ordering by parameter count exists across datasets.
Low paraphrase sensitivity can coexist with output-level image-removal invariance: LLaVA-Rad reaches 96% text-only agreement at low flip rates while MedGemma-4B is more image-dependent but flips more, so a low flip rate is insufficient evidence of grounded visual reasoning.
text-only-llavarad-107 · text-only-medgemma4b-107 · swap-sensitivity-medgemma4b-107 · flip-textonly-correlation-107
A candidate two-stage mechanistic account localizes paraphrase sensitivity: Feature 3818 (layer 17) behaves as an operator/register feature and Feature 12139 (layer 29) as a downstream answer-selection feature, with layer 16 as a general answer-commitment locus.
feature3818-operator-preserving-top1 · feature3818-patch-recovery · layer16-commit-rate
Partly. Targeted LoRA on layers 15 to 19 touches 0.1% of parameters and cuts pairwise flips from 8.5% to 3.5% over five seeds, about 59%. McNemar p < 0.002 in every seed, and no observed accuracy reduction on a patient- and image-disjoint split. But it does not preserve visual grounding: both adapters raise text-only agreement from 53.5% to about 77%.
lora-flip-reduction-pd · lora-accuracy-delta-pd · lora-param-fraction · lora-textonly-increase-pd
No: the best intervention layer does not match the diagnosis layer. Ablating early layers 0-10 reduces margin difference to 0.26 (86% reduction), beating the mechanistically identified layers 15-19 (0.38, 80%) by six points.
Reducing flips can create a false sense of safety: averaged across ten model-dataset settings, 81% of each model's consistent predictions are image-invariant (range 45-100%), so consistency and grounding routinely come apart.
image-invariant-share · quadrant-dangerous-targeted-padchest · quadrant-dangerous-fulllora-mimic · definitional-correlation
No: attention does not reliably indicate grounding. True-box coverage only marginally exceeds a displaced box (0.296 vs 0.261 for Base), and patch-rank correlation between attention and causal occlusion importance is near zero across all three MedGemma variants.
Partially: the worst-calibrated model also shows the largest demographic disparities. Full LoRA has the worst calibration (ECE of about 0.25 or higher for every group) and the largest sex gaps, while Targeted LoRA has the smallest sex ECE gap (0.012) but the steepest age accuracy gradient (13 pp).
fairness-ece-sex-gap-targeted · fairness-age-gradient-targeted
Yes: single-pass predictive entropy predicts paraphrase flips (AUROC 0.823) as well as errors (AUROC 0.862) on PadChest, and the flip bridge replicates across architectures (LLaVA-Rad LoRA 0.830 PadChest, 0.905 MIMIC).
No. Gates admit the cases the text already answers over the ones grounded in the image. Admitted-case accuracy does rise, to 96.8% against 91.5% for Targeted LoRA on PadChest. Yet that same model scores 2.9% on the slice where text and image disagree, and Qwen2-VL gets 0 of 279 such questions right under every rule tried.
gate-grounded-slice-failure · gate-admission-targeted-padchest
No: no single internal monitor transfers across model families. Gemma, LLaVA-Rad, and Qwen2-VL each require different monitors, and the multi-pass alternatives fail in family-specific ways: adapter-only Monte Carlo dropout is an uninformative epistemic probe and the MedGemma deep ensemble collapses out of distribution.
mcdropout-mi · ensemble-ood-failure · gate-grounded-slice-failure
A low paraphrase-flip rate is not sufficient evidence of reliable visual reasoning: safe deployment requires joint evaluation of semantic invariance, correctness, image-dependence, and calibration, because optimizing for consistency alone creates a false sense of safety.
image-invariant-share · flip-rate-range · lora-flip-reduction-pd · lora-textonly-increase-pd · quadrant-dangerous-targeted-padchest · entropy-flip-auroc
Paraphrase sensitivity persists in frontier general-purpose models that were never medically finetuned: four configurations of three such models flip on 3.1% to 9.5% of clinically equivalent rephrasings, and enabling reasoning on Claude Opus 5 raised its flip rate rather than lowering it (9.53% against 8.36%, p = 0.022). On PadChest, the one population where a frontier run and the published medical cells are directly comparable, the frontier rate is 3.08% against 12.0% to 32.8%.
frontier-flip-persists · frontier-reasoning-no-benefit · frontier-padchest-like-for-like