Paraphrase Sensitivity in Medical Vision-Language Models

Medical Vision-Language Models (VLMs) answer clinical questions about radiological images, but their reliability under natural language variation is poorly understood. This dissertation investigates Paraphrase Sensitivity Failure (PSF), where a medical VLM changes its diagnostic answer when a clinically equivalent question is rephrased, across five research thrusts.

Thrust 1 (Measurement) introduces PSF-Med, a benchmark of 26,850 chest X-ray questions with roughly 92,000 candidate paraphrases spanning MIMIC-CXR (United States), PadChest (Spain), and VinDr-CXR (Vietnam), filtered by a rubric-based equivalence audit (GPT-5-mini judge, 91.6 to 94.4% cross-family agreement). On the binary yes/no subset, six medical VLMs flip on 6.4 to 54.7% of pairs once two degenerate yes-bias cells are set aside.

Thrust 2 (Diagnosis) shows that a low flip rate is not evidence of visual grounding: a text-reliant model (LLaVA-Rad, 21.7% flips, 89.9% text-only agreement) is stable because it leans on the question text, while a more image-dependent model (MedGemma-4B, 73.9% flips) is far more fragile under rephrasing. The relationship is a tendency across models, not a universal law.

Thrust 3 (Mitigation) applies Sparse Autoencoders to MedGemma-4B and identifies Feature 3818 (layer 17) as an upstream operator gate and Feature 12139 (layer 29) as a downstream decision gate. A targeted Low-Rank Adaptation on layers 15 to 19, trained on operator-preserving paraphrases, reduces the presence-of-finding flip rate on a patient- and image-disjoint held-out test (pairwise 8.5% to 3.5% over five seeds, about 59%; paired McNemar p < 0.002 in every seed) with no observed accuracy reduction (84.5% to 84.7% ± 1.0; paired difference +0.25 percentage points, 95% interval [−1.00, +1.51], which excludes a large accuracy cost without establishing exact parity), modifying only 0.1% of parameters, so the reduction holds on patients and studies never seen in training.

Thrust 4 (Safety Evaluation) finds that across ten model-dataset settings, 81% of each model's consistent predictions are image-invariant (unchanged when the image is removed); correctness is a separate axis, since the image-invariant cell is often more accurate because high finding base rates make the text prior usually correct. Attention heatmaps are coarse localizers, not faithful causal explanations (patch saliency versus causal importance ρ ≈ 0).

Thrust 5 (Deployment Gating) establishes that single-pass predictive entropy predicts both errors and flips (AUROC 0.82 to 0.86), yet selective-prediction gates often admit text-answerable rather than image-grounded cases, and no single monitor transfers across model families.

The central thesis is that a low paraphrase-flip rate is not sufficient evidence of reliable visual reasoning: safe deployment requires joint evaluation of semantic invariance, correctness, image-dependence, and calibration, because optimizing for consistency alone creates a false sense of safety.