Paraphrase Sensitivity in Medical Vision-Language Models

Medical vision-language models are judged reliable when they answer the same clinical question the same way. They often fail even this basic test: across six models and three clinical populations, the answer changes on 6.4% to 54.7% of rephrased questions. The natural response is to train for consistency, and that is where the harder problem appears. A model can be consistent because it reads the chest radiograph the same way each time, or because it ignores the radiograph and answers from the language of the question. When we look, the second case is common: across ten model-dataset settings, a mean of 81% of each model's consistent answers are unchanged when the image is removed. In this thesis we study paraphrase sensitivity in medical vision-language models along both of these layers, and we show that neither inconsistency nor consistency, on its own, tells us whether a model is using the image.

We approach the problem in five stages: measure the failure, diagnose where it comes from, reduce it, evaluate whether the reduction is safe, and test what a deployment-time monitor can and cannot catch. We build PSF-Med, a benchmark of 26,850 chest X-ray questions and 92,856 evaluation pairs across three clinical populations, and show that six medical vision-language models flip on 6.4% to 54.7% of paraphrase pairs. We separate consistency from image dependence with controlled image-removal and image-swap experiments, and find that across ten model-dataset settings a mean of 81% of each model's consistent predictions are unchanged when the image is removed. We apply sparse autoencoders to locate where the failure is expressed, and use that account to design a targeted intervention that reduces the flip rate by about 59% on patients and images never seen in training while modifying 0.1% of the parameters. We then re-evaluate that intervention with the same controls used to indict the base models, and characterize a single-forward-pass uncertainty signal for deployment. To check that this failure begins on the language side rather than in the image, we train a small model that mirrors MedGemma's architecture, a compact Gemma-3 decoder over a frozen MedSigLIP-448 encoder, so that every paraphrase of a question is answered from identical image features and any answer change is language-side by construction, and we train it on two hospitals while holding out two others. Broadening the training-phrasing distribution lowers its flip rate from 67.1% to 4.8%, and to 26.6% when the test wording is withheld from training, while a rank-1 edit at its early layers restores the flipped answer. The yes-minus-no margin, the scalar a binary answer discards, catches a paraphrase flip in one forward pass at an area under the curve of 0.923 to 0.974, whereas an image-unreliant answer carries no single-pass signal.

The methods and the benchmark have been reviewed outside the university, with five papers drawn from this work accepted at peer-reviewed venues. The central conclusion is that safety claims for medical vision-language models require joint evaluation of semantic invariance, correctness, image dependence, and calibration, reported on margins rather than on binary answers alone, because a model optimized for consistency alone will answer the same way whether or not the patient's radiograph shows the finding.