Thesis map
Each thrust answers a question the previous one opened, and several of the answers are negative. The argument view below follows that chain. The chapter view shows how the dissertation is organized instead.
Nobody had measured how often a rephrased clinical question changes a medical model’s answer, at scale, on audited paraphrases.
Chapter 3
How often do equivalent questions change model answers?
Six models change their answer on 6.4% to 54.7% of clinically equivalent rewordings, and scale does not buy consistency.
Measurement showed some models are strikingly consistent. That raised a harder question: is consistency evidence of good reasoning, or of ignoring the image?
Chapter 4
Does consistency mean the model is using the image?
Across three backends, the most consistent model is the one that responds least when the image is taken away.
If consistency can be ungrounded, then reducing flips is only worth doing if it does not come from leaning further on the text. That means finding where the answer is decided.
Chapter 5
Can paraphrase flips be reduced without degrading observed accuracy?
A targeted LoRA on five layers cuts the flip rate about 59% without observed accuracy loss, and a controlled probe identifies the training-phrasing distribution as the cause.
Open this thrust Filter the evidenceThe causality experiments
The adapter cut flips. Whether that made the model safer is a separate question, and it needs a screen that tests consistency and image dependence together.
Chapter 6
Do consistency improvements produce safer, more image-dependent behavior?
Averaged over ten model-dataset settings, 81% of consistent predictions are unchanged when the image is removed; consistency and grounding are different axes.
If a consistent model can still be ungrounded, deployment cannot rely on consistency. Something has to flag the predictions a clinician should look at.
Chapter 7
Can uncertainty and audit signals identify predictions that should be escalated?
One forward pass ranks flip risk (AUROC 0.823), and the deployment gate is an offline readiness audit rather than an online tool.
| Chapter | Title | Thrust | What it establishes |
|---|---|---|---|
| 1 | Introduction | — | Frames paraphrase sensitivity in medical Vision-Language Models, states the overarching question and its five-thrust decomposition, and motivates the restriction to binary presence questions about chest X-rays. |
| 2 | A Scoping Review of Trustworthiness Evaluation in Medical Vision-Language Models | — | Surveys how the field evaluates trustworthiness in medical VLMs and shows that the practices each thrust targets, such as paraphrase invariance testing and image-reliance checks, are routinely omitted. |
| 3 | Thrust 1: Measuring Paraphrase Sensitivity in Medical VLMs | Measurement | Builds the PSF-Med benchmark (92,856 judge-filtered evaluation pairs from 26,850 questions across MIMIC-CXR, PadChest, and VinDr-CXR) and measures pairwise flip rates from 6.4% to 54.7% across six models on the binary subset, with two degenerate yes-bias cells set aside. |
| 4 | Thrust 2: Diagnosing the Origin of Paraphrase Sensitivity | Diagnosis | Uses text-only evaluation and controlled image swaps on a 107-pair diagnostic set, plus an attention-bounding-box analysis on a separate 200-pair PadChest subsample, to show that, across the three models tested, the most consistent backends rely least on the image. |
| 5 | Thrust 3: Mechanistic Diagnosis and Targeted Mitigation | Mitigation | Locates a general answer-commitment locus at layer 16 and candidate SAE features (L17/#3818, L29/#12139), then shows a targeted LoRA on layers 15-19 (0.10% of parameters) cuts the pairwise flip rate about 59% on a patient-disjoint set without observed accuracy reduction; a safety re-audit finds the adapters gain consistency by leaning harder on question text, with no grounding restored. |
| 6 | Thrust 4: Safety Evaluation | Safety | Introduces a four-quadrant audit crossing consistency with image reliance: averaged across ten model-dataset settings, 81% of consistent predictions are image-invariant, and consistency-optimized adapters concentrate predictions in the consistent-but-text-reliant quadrant. Correctness is a separate axis, so the screen is not a per-prediction safety verdict. |
| 7 | Thrust 5: Calibrated Uncertainty and Deployment Gating | Deployment | Shows a single-forward-pass entropy signal predicts paraphrase flips (AUROC 0.823) and errors (AUROC 0.862) and transfers across architectures, but the deployment gate is an offline readiness audit whose region term requires radiologist bounding boxes that are not available at inference; admitted cases skew text-answerable over grounded, and the MedGemma deep ensemble fails out of distribution without early warning. |
| 8 | Conclusion | — | Synthesizes the five thrusts under the central claim that a low paraphrase-flip rate is not sufficient evidence of reliable visual reasoning, and lays out future work including a same-polarity mechanistic screen and the designed but not yet conducted clinician deployment consultation. |
| — | Appendices: Supplementary Material; Extended Model Details and Hyperparameters; Reproducibility Information | — | Supplementary analyses and tables, extended model and hyperparameter details, and reproducibility information for the experiments in Chapters 3 through 7. Included in the final dissertation PDF. |
From the synthesis table in Chapter 8. The chip states how strongly the evidence supports the answer.
Paraphrase sensitivity is common in medical vision-language models: pairwise flip rates on the binary yes/no subset span 6.4% to 54.7% across six models and three chest X-ray datasets, an 8.5x spread.
flip-rate-range · flip-rate-medgemma4b-mimic · flip-rate-medgemma4b-padchest · benchmark-eval-pairs
Scale does not buy paraphrase consistency: MedGemma-27B is the most consistent of the three MedGemma variants on MIMIC (6.4%) and VinDr (8.1%) but not on PadChest, where MedGemma-4B edges it (13.4% vs 13.9%), and no stable consistency ordering by parameter count exists across datasets.
Low paraphrase sensitivity can coexist with output-level image-removal invariance: LLaVA-Rad reaches 96% text-only agreement at low flip rates while MedGemma-4B is more image-dependent but flips more, so a low flip rate is insufficient evidence of grounded visual reasoning.
text-only-llavarad-107 · text-only-medgemma4b-107 · swap-sensitivity-medgemma4b-107 · flip-textonly-correlation-107
A candidate two-stage mechanistic account localizes paraphrase sensitivity: Feature 3818 (layer 17) behaves as an operator/register feature and Feature 12139 (layer 29) as a downstream answer-selection feature, with layer 16 as a general answer-commitment locus.
feature3818-operator-preserving-top1 · feature3818-patch-recovery · layer16-commit-rate
Partly. Targeted LoRA on layers 15 to 19 touches 0.1% of parameters and cuts pairwise flips from 8.5% to 3.5% over five seeds, about 59%. McNemar p < 0.002 in every seed, and no observed accuracy reduction on a patient- and image-disjoint split. But it does not preserve visual grounding: both adapters raise text-only agreement from 53.5% to about 77%.
lora-flip-reduction-pd · lora-accuracy-delta-pd · lora-param-fraction · lora-textonly-increase-pd
No: the best intervention layer does not match the diagnosis layer. Ablating early layers 0-10 reduces margin difference to 0.26 (86% reduction), beating the mechanistically identified layers 15-19 (0.38, 80%) by six points.
Reducing flips can create a false sense of safety: averaged across ten model-dataset settings, 81% of each model's consistent predictions are image-invariant (range 45-100%), so consistency and grounding routinely come apart.
image-invariant-share · quadrant-dangerous-targeted-padchest · quadrant-dangerous-fulllora-mimic · definitional-correlation
No: attention does not reliably indicate grounding. True-box coverage only marginally exceeds a displaced box (0.296 vs 0.261 for Base), and patch-rank correlation between attention and causal occlusion importance is near zero across all three MedGemma variants.
Partially: the worst-calibrated model also shows the largest demographic disparities. Full LoRA has the worst calibration (ECE of about 0.25 or higher for every group) and the largest sex gaps, while Targeted LoRA has the smallest sex ECE gap (0.012) but the steepest age accuracy gradient (13 pp).
fairness-ece-sex-gap-targeted · fairness-age-gradient-targeted
Yes: single-pass predictive entropy predicts paraphrase flips (AUROC 0.823) as well as errors (AUROC 0.862) on PadChest, and the flip bridge replicates across architectures (LLaVA-Rad LoRA 0.830 PadChest, 0.905 MIMIC).
No. Gates admit the cases the text already answers over the ones grounded in the image. Admitted-case accuracy does rise, to 96.8% against 91.5% for Targeted LoRA on PadChest. Yet that same model scores 2.9% on the slice where text and image disagree, and Qwen2-VL gets 0 of 279 such questions right under every rule tried.
gate-grounded-slice-failure · gate-admission-targeted-padchest
No: no single internal monitor transfers across model families. Gemma, LLaVA-Rad, and Qwen2-VL each require different monitors, and the multi-pass alternatives fail in family-specific ways: adapter-only Monte Carlo dropout is an uninformative epistemic probe and the MedGemma deep ensemble collapses out of distribution.
mcdropout-mi · ensemble-ood-failure · gate-grounded-slice-failure
A low paraphrase-flip rate is not sufficient evidence of reliable visual reasoning: safe deployment requires joint evaluation of semantic invariance, correctness, image-dependence, and calibration, because optimizing for consistency alone creates a false sense of safety.
image-invariant-share · flip-rate-range · lora-flip-reduction-pd · lora-textonly-increase-pd · quadrant-dangerous-targeted-padchest · entropy-flip-auroc
Paraphrase sensitivity persists in frontier general-purpose models that were never medically finetuned: four configurations of three such models flip on 3.1% to 9.5% of clinically equivalent rephrasings, and enabling reasoning on Claude Opus 5 raised its flip rate rather than lowering it (9.53% against 8.36%, p = 0.022). On PadChest, the one population where a frontier run and the published medical cells are directly comparable, the frontier rate is 3.08% against 12.0% to 32.8%.
frontier-flip-persists · frontier-reasoning-no-benefit · frontier-padchest-like-for-like