Every result, traceable to its source

What this shows. Every model changes its yes/no answer on a meaningful share of clinically equivalent rephrasings, and the spread across models is 8.5-fold (6.4% for MedGemma-27B on MIMIC-CXR to 54.7% for RadFM on VinDr-CXR). No model is stable on all three populations, and rank order changes with the dataset.

Population. Six base models (MedGemma-1.5-4B, MedGemma-4B, MedGemma-27B, LLaVA-Rad, CheXone, RadFM) on the equivalence-filtered PSF-Med binary yes/no subset. Denominator is the paraphrase pair: MIMIC-CXR 5,076 pairs over 1,539 questions, PadChest 36,244 over 8,445, VinDr-CXR 8,612 over 2,807. RadFM's PadChest cell covers 8,092 questions / 34,733 pairs.

What this shows. Counting a question as unstable if any of its roughly 3.3 paraphrases flips roughly doubles every rate: MedGemma-4B moves from 8.3% pairwise to 18.1% on MIMIC-CXR and from 13.4% to 32.2% on PadChest. Per-question exposure is what a clinician meets, and it is far higher than the pairwise number.

Population. Six base models on the equivalence-filtered PSF-Med binary yes/no subset. Denominator is the question: MIMIC-CXR 1,539, PadChest 8,445, VinDr-CXR 2,807. RadFM's PadChest cell covers 8,092 questions. MedGemma-27B / PadChest has no cell in the source file.

What this shows. Consistency and correctness do not travel together. The most consistent cells include the degenerate ones, and on VinDr-CXR (all-positive) a model that always answers yes scores 100% accuracy at a 0.8% flip rate. Reading either axis alone ranks models wrongly.

Population. Six base models x three datasets on the equivalence-filtered PSF-Med binary yes/no subset (MedGemma-27B / PadChest has no cell). x = pairwise flip rate over paraphrase pairs; y = weak-proxy accuracy over original questions in the same run.

What this shows. A high bar means the model would give the same answer with no image at all. On the 107-pair set LLaVA-Rad reaches 96.3%, so it is effectively a text-only model that accepts an image. On the patient-disjoint audit both adapters buy their consistency by leaning harder on the question text: text-only agreement rises about 23 points above the base model, and Targeted and Full LoRA are statistically tied.

Population. Group 1: the 107-pair three-backend diagnostic set (MedGemma-4B, MedGemma-27B, LLaVA-Rad), denominator = pair. Group 2: the patient- and image-disjoint MIMIC-CXR safety re-audit, presence-of-finding endpoint, 241 questions, 239 held-out subjects, zero subject and zero image overlap with training; Targeted LoRA is the mean over seeds 42/123/456/789/2024 and Full LoRA over seeds 42/123/456.

What this shows. The Dangerous cell (consistent but text-reliant) is the largest in eight of the ten settings, and it grows when consistency is optimised: the base model on MIMIC-CXR is 25.5% Dangerous while Targeted LoRA is 59.2% and Full LoRA 78.6%. Averaged across the ten settings, 81% of each model's consistent predictions are image-invariant (range 45-100%).

Population. Curated behavioural flip banks, question-level. MedGemma-4B base / Targeted LoRA / Full LoRA on MIMIC-CXR (n=98, the subset with a text-only baseline) and PadChest (n=861); LLaVA-Rad base and LLaVA-Rad LoRA on MIMIC-CXR (n=88) and PadChest (n=732). These flip banks are distinct from the PSF-Med benchmark and must not be pooled with it.

What this shows. Replacing the radiograph with one showing the opposite finding changes the answer for at most 39.5% of questions, and for LLaVA-Rad only 14.5%. Even the most image-sensitive model here leaves roughly six answers in ten unchanged when the visual evidence is contradicted.

Population. The 861-question PadChest flip bank, question-level denominator. Swap sensitivity = fraction of questions whose answer changes when the image is replaced with one showing the opposite finding, computed by the safety-evaluation pipeline (results/miccai) alongside flip rate and text-only agreement.

What this shows. Every adapter seed lands well below the base model's 8.5% pairwise flip rate: Targeted LoRA averages 3.5% (about a 59% reduction) and Full LoRA 2.9%, with McNemar exact p between 2.5e-7 and 1.8e-3 on every seed. The seed spread is small relative to the base-to-adapter gap.

Population. Base MedGemma-4B versus Targeted LoRA (layers 15-19, r=16, alpha=32, 4.38M params) and Full LoRA (all 34 layers, 30.5M params) on the clean patient-disjoint MIMIC-CXR test. Primary endpoint = presence-of-finding questions: 238 questions, 991 paraphrase pairs; denominator is the paraphrase pair. Base is a single run and is identical across seeds by construction.

What this shows. Every arm's interval overlaps every other arm's: the base model scores 84.5% and Targeted LoRA 84.7% on average, so the large flip reduction is not paid for by a large accuracy loss.

Population. Base MedGemma-4B, Targeted LoRA (5 seeds) and Full LoRA (3 seeds) on the clean patient- and image-disjoint MIMIC-CXR test, presence-of-finding primary endpoint. Denominator = 238 original questions per arm (question-level accuracy on the original phrasing). Interval method: 95% Wilson score interval on the binomial proportion of correct originals, computed per arm from the per-question block, z = 1.96, no continuity correction, unpaired.

What this shows. Swapping a single layer's residual state at the answer position flips the model's committed answer on 73% of pairs at layer 16 and saturates by layer 20, while the image-token control never exceeds 0.07%. The commit is narrow-band: the median per-pair commit layer is 16 with a 95% bootstrap CI of [16, 16].

Population. MedGemma-4B, 1,396 detecting/missing minimal pairs spanning 91 findings drawn from the PadChest paraphrase population. Denominator = transplant pair. Control series = a pre-question image-token position, which a causal mask makes unable to receive the later donor state.

What this shows. Adapting early layers 0-10 cuts the paraphrase margin difference furthest (0.26, an 86% reduction), beating the mechanistically identified middle window 15-19 (0.38, 80%) by 6 points, and even a randomly chosen block at layers 5-9 (0.30) beats it. Where paraphrase sensitivity is diagnosed is not where it is best fixed.

Population. MedGemma-4B with five LoRA layer ranges (early 0-10, random block 5-9, all 0-33, middle 15-19, late 25-33), rank 16, alpha=32, combined loss lambda=1.0, evaluated on the 355-question MIMIC-CXR validation split. Denominator = validation question. Baseline = the un-adapted model, margin difference 1.87.

What this shows. Feature 3818 has the strongest association with flipping of any layer-17 feature (rank-biserial -0.462): when it is active the model flips less often. It encodes presence-versus-exclusion phrasing, which is the candidate mechanism the dissertation follows up.

Population. Targeted LoRA (layers 15-19) on the 861-question PadChest flip bank, 5,112 original/paraphrase records, 305 flipped questions (35.4% question-level flip rate in this screen). GemmaScope 2 sparse autoencoder at layer 17; denominator = feature. Association is the rank-biserial correlation between a feature's activation and whether the pair flipped.

What this shows. Questions whose answer flips under paraphrase sit at visibly higher single-pass predictive entropy than stable ones, which is what lets one forward pass rank flip risk (AUROC 0.823).

Population. Targeted LoRA (MedGemma-4B, layers 15-19) on the 861-question PadChest flip bank, one question per row, single-pass softmax entropy over the yes/no answer distribution.

What this shows. Cheap predictive entropy on the original question predicts whether that question will flip under paraphrasing at AUROC 0.823, so uncertainty and paraphrase sensitivity are measuring related failure. The expensive deep ensemble is the worst predictor at 0.552.

Population. Targeted LoRA (layers 15-19) on the 861-question PadChest flip bank; denominator = question. Positive class = the question flips under at least one paraphrase. Temperature scaling is scored on the 732 questions outside its 15% calibration holdout.

What this shows. Targeted LoRA dominates: its risk stays near zero out to high coverage (AUGRC 0.014 against 0.047 for the base model and 0.153 for Full LoRA), so confidence-based deferral works for it and barely works for Full LoRA, whose curve rises almost immediately.

Population. Base MedGemma-4B, Targeted LoRA and Full LoRA on the 861-question PadChest flip bank, softmax-entropy method, one prediction per question. Selection score = max(p_yes, 1-p_yes); at coverage k/n the selective risk is errors among the k most confident, divided by k. Denominator = question.

What this shows. Filtering on agreement plus image reliance lifts accuracy in five of six cells (base on PadChest: 78.6% to 96.9%), but admits only 14-42% of questions. The price of a trustworthy answer is refusing most of them.

Population. Base MedGemma-4B, Targeted LoRA and Full LoRA on the curated behavioural flip banks: MIMIC-CXR n=98 and PadChest n=861 questions. Denominator = question. Admission rule: agree(p1, p2) AND (swap_changed OR roi_matters), where roi_matters (region-only versus background-only crops disagree) is PadChest-only.

What this shows. Roughly half of everything an LLM generates as a 'paraphrase' of a clinical question is not clinically equivalent, and the rate depends strongly on the source: PadChest keeps only 35.7% against 78.6% for VinDr-CXR. Without this audit a benchmark would be scoring models on questions that genuinely changed meaning.

Population. All 122,778 construction-stage candidate paraphrase pairs adjudicated against the equivalence rubric, by source dataset. Denominator = candidate pair. Verdicts recomputed on 2026-07-15 from data/judge_full_results.json and data/vindr_judge/gpt_results.json; the total reproduces the 122,778 reported in Chapter 3 exactly.

What this shows. No transformation type is safe: even plain lexical substitution flips 11-17% of pairs. Negation-pattern rephrasings are the worst where a usable pool survives the audit (34.7% on VinDr-CXR), roughly triple the lexical rate on the same dataset.

Population. Mean over six base models (MedGemma-4B, MedGemma-1.5-4B, MedGemma-27B, LLaVA-Rad, CheXone, RadFM) on the equivalence-filtered PSF-Med binary yes/no subset, split by paraphrase transformation type. Denominator = paraphrase pair; per-cell n is pairs per model.

What this shows. Targeted LoRA is the most accurate arm in every band but has the steepest age gradient: 83.3% for patients under 40 against 96.3% for those over 80, a 13-point spread. The arm with the smallest between-sex calibration gap is therefore also the one with the widest age disparity, so no model is uniformly the most equitable.

Population. Base MedGemma-4B, Targeted LoRA and Full LoRA on the 861-question PadChest flip bank, softmax-entropy method, stratified by PatientAge from the PadChest master table. Denominator = question. Bands: under 40 (n=48), 40-60 (n=233), 60-80 (n=416), over 80 (n=164).

What this shows. Targeted LoRA's undetected-failure risk stays low and nearly flat under corruption (0.014 clean to 0.027 at severity 5) while the base model nearly doubles (0.047 to 0.088). Full LoRA is the worst at every severity (0.153 to 0.186): it is flat only because it starts badly.

Population. Base MedGemma-4B, Targeted LoRA and Full LoRA on the 861-question PadChest flip bank under synthetic corruption, softmax-entropy method. Denominator = question. AUGRC (Traub et al., NeurIPS 2024) is bounded in [0, 0.5] and reads as average undetected-failure risk; severity 0 is one clean run, severities 1/3/5 are the mean over five corruption types.

What this shows. Frontier models that were never medically finetuned still change their yes/no answer on 3.1% to 9.5% of clinically equivalent rephrasings. That is lower than the 6.4% to 54.7% span the six medical models post on the binary subset, but it is nowhere near zero, so paraphrase sensitivity is not a defect of small domain-tuned models that frontier scale has removed. Two details carry most of the weight. Reasoning does not help: Claude Opus 5 flips more with reasoning on (9.53%) than with it off (8.36%), a 1.17 percentage-point increase in the wrong direction (two-proportion z = 2.29, p = 0.022, 95% CI on the difference 0.17 to 2.18 points). And on PadChest, the one dataset where a frontier run and the published medical cells cover the same population, Kimi K3's 3.08% sits against 12.0% to 32.8% for the medical models, so the frontier advantage on a like-for-like population is real and large.

Population. Three frontier general-purpose multimodal models with no medical finetuning, evaluated in a post-draft addendum on clinically equivalent paraphrase pairs from the PSF-Med binary yes/no subset. Denominator is the paraphrase pair. Claude Opus 5 covers 6,209 pairs with reasoning on and 6,194 with reasoning off, GPT-5.6 Sol 6,195, each spanning MIMIC-CXR, PadChest and VinDr-CXR; Kimi K3 covers 2,274 PadChest pairs only.