Paraphrase Sensitivity in Medical Vision-Language Models

Medical vision-language models often give different answers to clinically equivalent questions. Across six models and three clinical populations, answers change on 6.4% to 54.7% of paraphrase pairs. Training for consistency can reduce these changes, but consistent answers need not depend on the radiograph. They may instead follow learned associations between question wording and answers. Across ten model-dataset settings, a mean of 81% of each model's consistent answers are unchanged when the image is removed. Both paraphrase sensitivity and image dependence are studied and reported in this dissertation. We show that consistency alone does not establish whether a model uses the image.

The study covers measurement, diagnosis, mitigation, safety evaluation, and monitoring at deployment. PSF-Med contains 26,850 chest X-ray questions and 92,856 evaluation pairs across three clinical populations. Six medical vision-language models flip on 6.4% to 54.7% of these pairs. Image-removal and image-swap experiments then test whether consistent answers depend on visual input: across ten model-dataset settings, a mean of 81% of each model's consistent predictions are unchanged when the image is removed. Sparse-autoencoder analysis identifies internal features associated with the failure and guides a targeted intervention. It reduces flip rate by about 59% on patients and images excluded from training while modifying 0.1% of the parameters.

We evaluate the intervention with the image controls used for the base models. We also study an uncertainty signal that requires a single forward pass at deployment. To test the role of language processing, we train a small, domain-adapted model based on MedGemma's architecture. It combines a compact Gemma-3 decoder with a frozen MedSigLIP-448 encoder. Every paraphrase of a question receives identical image features, so any answer change arises on the language side. We train the model on two hospitals and hold out two others. Broadening the training-phrasing distribution lowers its flip rate from 67.1% to 4.8%, and to 26.6% when the test wording is withheld from training, while a rank-1 edit at its early layers restores the flipped answer. The yes-minus-no margin retains information lost in the binary answer. In one forward pass, it predicts paraphrase flips with an area under the curve of 0.923 to 0.974. The tested signals do not identify image-unreliant answers in a single pass.

Six papers reporting this work have been accepted at peer-reviewed venues. These results show why safety claims require joint evaluation of semantic invariance, correctness, image dependence, and calibration, using margins as well as binary answers. A model trained for consistency alone may give the same answer whether or not the patient's radiograph shows the finding.