Doctoral dissertation · 2026
Paraphrase Sensitivity in Medical Vision‑Language Models
Measurement, Mechanisms, Mitigation, and Deployment Safety
Advisor: Vahid Behzadan, Ph.D.
In brief
Equivalent clinical questions should produce the same answer, yet six medical vision-language models changed their responses on 6.4% to 54.7% of tested paraphrase pairs. Consistency alone was also misleading: across ten model-dataset settings, a mean of 81% of consistent predictions stayed unchanged when the image was removed. This dissertation measures both failures, tests their causes, evaluates a targeted intervention, and studies what uncertainty signals can detect before deployment.
How this thesis addresses the FDA's asks
The defense deck connects the FDA's proposed tests to work in this thesis. Each row pairs an ask with the study that addresses it.
Based on defense slide 66, “How my work answers FDA's Asks,” and the FDA discussion paper.
| FDA element | What the FDA asks | How this thesis responds |
|---|---|---|
| R.1 · Consistent answers |
Check that a device gives the same answer to questions with the same meaning. |
PSF-Med measures this failure. It covers 26,850 source questions and 92,856 evaluated pairs. On the binary subset, pairwise flips range from 6.4% to 54.7%. The range excludes settings where models almost always answer yes. |
| R.1 · Diagnostic failures |
Treat changes in a diagnosis as failures, even when only the wording changes. |
The four-quadrant screen checks image use too. Across 10 model and dataset settings, a mean of 81% of consistent predictions stayed the same when the image was removed. A steady answer can still ignore the scan. |
| S.3 · Confidence and deferral |
Check that confidence matches what the device can support, and that it defers when needed. |
Entropy flags answers that may fail. For Targeted LoRA on PadChest, it ranks flips at AUROC 0.823 and errors at 0.862 in one pass. Tests with corrupted images also show loss of conformal coverage. The gate is an offline audit; clinical deferral still needs testing. Chapter 7 · Flip detection · Error detection · Coverage tests |
| R.2 · Patient groups |
Check performance across patient groups and levels of health literacy. |
The thesis audits sex and age groups on PadChest. It finds gaps in accuracy and how well confidence matches errors. No model leads on every measure. Health literacy remains untested. |
| VI.C · Re-test after changes |
Re-test a changed model against its earlier benchmark to check for shifts in safety and performance. |
The LoRA re-audit shows why this matters. Targeted LoRA lowers pairwise flips to 3.5% with no observed accuracy reduction on the patient-disjoint test. But follow-up checks find more reliance on text. A gain in consistency can hide a loss of image use. |
This mapping shows how the research informs the FDA's discussion. The paper sets no new rules, and these studies do not establish clinical readiness or FDA approval.
Dissertation and supporting material
The dissertation, research questions, experiments, explanations, and reproducibility record.