26,850
PSF-Med covers 26,850 original clinical questions
The failure this thesis is about
Consistency and grounding are different properties. Plotting them against each other gives four behaviors, and the dangerous one looks like success: the model answers rephrased questions the same way, and it answers the same way when you take the image away.
Choose a quadrant to see what it means, or read the four definitions below.
The four quadrants, as text
- Dangerous: Consistent but image-invariant
- The model gives the same answer to rephrased questions and the same answer when the image is removed or swapped. Consistency here is a text prior, not visual reasoning. This is the failure the dissertation is named for. See the evidence.
- Ideal: Consistent and image-dependent
- The model gives the same answer to rephrased questions, and its answer changes when the image changes. This is the target behavior. See the evidence.
- Worst: Inconsistent and image-invariant
- The answer moves with the wording but not with the image. Neither stable nor grounded. See the evidence.
- Fragile: Inconsistent but image-dependent
- The model uses the image, but rephrasing still changes the answer. Unstable, yet grounded. See the evidence.
The work in numbers
Every number here links to the file it was computed from and the sample it covers.
92,856
PSF-Med evaluates 92,856 question-paraphrase pairs
81% (range 45-100%)
81% of consistent predictions are image-invariant
6.4% to 54.7%
Paraphrase flip rates span 6.4% to 54.7% across six medical VLMs
Follow the argument
Each thrust exists because the previous one raised a question it could not answer.
- Measurement How often do equivalent questions change model answers?
- Diagnosis Does consistency mean the model is using the image?
- Mitigation Can paraphrase flips be reduced without degrading observed accuracy?
- Safety Do consistency improvements produce safer, more image-dependent behavior?
- Deployment Can uncertainty and audit signals identify predictions that should be escalated?
Selected evidence
Five results that carry the argument, including the two that went against my hypothesis.
6.4% to 54.7%
pairwiseParaphrase flip rates span 6.4% to 54.7% across six medical VLMs
Rephrasing a clinical yes/no question in a way that preserves its meaning changes the answer on anywhere from 1 in 16 to more than half of paraphrase pairs, depending on which model and which patient population you test. Paraphrase sensitivity is not a quirk of one weak model: every model tested flips, and the spread between the best and worst model-dataset cell is 8.5-fold.
Multiple models · Multiple datasets · A range over 18 model-dataset cells, so it has no single denominator. Per-dataset binary-subset denominators are MIMIC-CXR 1,539 questions / 5,076 pairs, PadChest 8,445 / 36,244, and VinDr-CXR 2,807 / 8,612, each evaluated on every model. The endpoints are MedGemma-27B on MIMIC (6.4%) and RadFM on VinDr (54.7%).
Source, denominator, and limits
How far this goesTwo of the 18 cells are set aside as degenerate yes-bias and excluded from this range: LLaVA-Rad on PadChest (99.6% positive predictions on originals) and MedGemma-1.5-4B on VinDr (100%). Both post a near-zero flip rate (0.8%) that reflects a single-class prior rather than paraphrase stability, so a model that answers 'yes' to everything would look maximally consistent here. The range also confounds model and population: it spans three different countries' patient distributions, so a cell's position reflects both the model and where the data came from.
- Metric
- pairwise paraphrase flip rate (range across six models x three datasets)
- Denominator
- pairwise (per question-paraphrase pair)
- Sample
- A range over 18 model-dataset cells, so it has no single denominator. Per-dataset binary-subset denominators are MIMIC-CXR 1,539 questions / 5,076 pairs, PadChest 8,445 / 36,244, and VinDr-CXR 2,807 / 8,612, each evaluated on every model. The endpoints are MedGemma-27B on MIMIC (6.4%) and RadFM on VinDr (54.7%).
- Population
- Binary yes/no subset
- Model
- Multiple models
- Dataset
- Multiple datasets
- Split
- eval
- Comparison
- An 8.5x spread between the most consistent cell (MedGemma-27B on MIMIC) and the least consistent (RadFM on VinDr).
- Uncertainty
- not reported for this value
- Thesis
- Chapter 3, Table 3.3
- Paper
- PSF-Med: A Clinician-Audited Benchmark for Paraphrase Sensitivity in Medical Vision-Language Models
- Source artifact
results/uai/revision/psf_binary_recompute_v2.json- Last verified
- 2026-07-15
81% (range 45-100%)
model-dataset81% of consistent predictions are image-invariant
Averaged across ten model-dataset settings, 81% of each model's consistent predictions are unchanged when the image is removed (range 45-100%), so a low paraphrase-flip rate is not evidence of grounded visual reasoning.
Multiple models · Multiple datasets · n = 10 · unweighted mean over 10 settings; range 45-100%
Source, denominator, and limits
How far this goesStated deliberately as a level, not a correlation: the r = -0.86 between flip rate and the image-invariant fraction is largely definitional (the fraction is by construction a subset of consistent predictions, and (1-flip) alone correlates r = +0.86) and falls to r = -0.15 once conditioned on consistency. The level, not the slope, is the finding.
- Metric
- mean share of consistent predictions that are image-invariant (text-only agreement)
- Denominator
- model-dataset (per model-dataset setting)
- Sample
- n = 10
- Model
- Multiple models
- Dataset
- Multiple datasets
- Split
- eval
- Comparison
- range 45-100% across the ten model-dataset settings
- Uncertainty
- unweighted mean over 10 settings; range 45-100%
- Thesis
- Chapter 6, tab:quadrant_counts
- Source artifact
dissertation/tables/thrust4/table_quadrant_counts.tex- Last verified
- 2026-07-15
76.8%
question-levelThe adapter buys consistency by leaning harder on the question text
This is the reframe that changes the story. On the clean patient-disjoint audit, text-only agreement rises from 53.5% for the base model to 76.8% (+/- 3.1) after targeted adaptation: the adapted model returns its original answer with the image deleted 23 points more often. Both adapters get much of their new consistency by relying more on the question text, not by making steadier use of the image. Targeted and Full LoRA are statistically tied on image reliance (76.8% +/- 3.1 vs 76.6% +/- 1.7).
Targeted LoRA · MIMIC-CXR · n = 241 · +/- 3.1 (standard deviation over five seeds)
Source, denominator, and limits
How far this goesThe claim that targeted adaptation preserves image dependence relative to full adaptation is NOT supported on the clean adapters: the two are tied on both image-reliance measures, and both move the model toward the text. The older ordering (targeted 66.3% text-only / 32.7% swap vs full 80.6% / 14.3%) is specific to the original-pipeline adapters and does not replicate. Targeted LoRA remains the recommendation only because at tied consistency it keeps accuracy (84.3% vs 82.4%) at 0.1% of parameters instead of 0.72%: it is the cheaper and more accurate route to the same consistency, not a more grounded one. This audit is the reason a low flip rate cannot be read as evidence of visual reasoning. The displayed value is the mean over five seeds (per-seed: 78.8, 80.5, 74.3, 77.6, 73.0).
- Metric
- text-only agreement (share of predictions unchanged when the image is removed), presence-of-finding endpoint
- Denominator
- question-level (per question)
- Sample
- n = 241
- Model
- Targeted LoRA
- Dataset
- MIMIC-CXR
- Split
- patient-disjoint
- Comparison
- 53.5% for the base model, a 23-point increase; Full LoRA reaches a statistically indistinguishable 76.6% +/- 1.7. Swap sensitivity also falls slightly (base 22.0%, targeted 19.9% +/- 1.2, full 20.9% +/- 1.3).
- Uncertainty
- +/- 3.1 (standard deviation over five seeds)
- Thesis
- Chapter 5, tab:pd_safety_audit
- Source artifact
results/lora_fleet_patient_disjoint/audit_targeted_s{42,123,456,789,2024}.json (mean over five seeds)- Last verified
- 2026-07-15
0.823
question-levelSingle-pass entropy predicts paraphrase flips (AUROC 0.823)
Softmax predictive entropy from one forward pass predicts whether a prediction will flip under paraphrasing with AUROC 0.823 for Targeted LoRA on PadChest: uncertainty is a deployable single-pass proxy for paraphrase fragility.
Targeted LoRA · PadChest · n = 861 · 95% CI [0.778, 0.864]; p = 4.3e-29
Source, denominator, and limits
How far this goesSoftmax entropy, temperature-scaled entropy and absolute margin are rank-equivalent for a binary softmax, so they are one signal reported three ways, not three independent methods; only the deep ensemble is genuinely distinct, and it fails (0.552). Operating thresholds are model-specific and do not transfer.
- Metric
- flip-prediction AUROC from softmax predictive entropy
- Denominator
- question-level (per question)
- Sample
- n = 861
- Population
- PadChest flip bank (861 questions)
- Model
- Targeted LoRA
- Dataset
- PadChest
- Split
- ood
- Comparison
- operator-preserving slice 0.826 (95% CI [0.780, 0.870]); deep ensemble 0.552; cross-architecture LLaVA-Rad LoRA 0.830 (PadChest) / 0.905 (MIMIC)
- Uncertainty
- 95% CI [0.778, 0.864]; p = 4.3e-29
- Thesis
- Chapter 7, tab:bridge_auroc
- Paper
- Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models
- Source artifact
dissertation/tables/thrust4/table_bridge_auroc.tex- Last verified
- 2026-07-15
2.9%
question-level2.9% accuracy on the slice where the image should matter
Targeted LoRA scores only 2.9% accuracy on the PadChest text-disagrees slice - the cases where the image-conditioned answer must depart from the text-only answer - so gates raise admitted-case accuracy by selecting text-answerable cases rather than image-grounded ones (RQ5.3 = no).
Targeted LoRA · PadChest · n = 241
Source, denominator, and limits
How far this goesA negative result, and it should be read as one: apparent safety gains from gating come from routing around the image-relevant portion of the workload, not from solving it. Severity is dataset-dependent (the same slice on MIMIC-CXR sits at 68-86% accuracy), so the lesson is that high-answerability environments amplify the text shortcut, not that all medical VLMs always fail when the image matters.
- Metric
- accuracy on the text-disagrees slice
- Denominator
- question-level (per question)
- Sample
- n = 241
- Population
- PadChest flip bank (861 questions)
- Model
- Targeted LoRA
- Dataset
- PadChest
- Split
- ood
- Comparison
- the strict gating rule raises this only to 25.0% at 2% intra-slice coverage; Qwen2-VL gets 0 of 279 text-disagrees questions correct under every evaluated rule
- Uncertainty
- not reported for this value
- Thesis
- Chapter 7
- Paper
- Consistency Is Not Safety: Deployment Audits Reveal Text-Driven Selection in Medical Vision-Language Models
- Source artifact
dissertation/chapters/06_thrust5.tex- Last verified
- 2026-07-15
Where to start
Pick a depth. Nothing is hidden either way, so choosing one only changes the order I suggest.
The argument and the one result that carries it.
One result per thrust, measurement through deployment.
Every thrust, every population, every caveat.