Defense companion
Experiments in the defense deck
Every experiment, control, and analysis represented in the 71-slide main defense presentation, with its sample size, method, headline result, and supporting material. For the same material arranged by thesis claim, see the thesis map.
Defense slides (PDF, 71 pages) Filterable results and charts
About the slide referencesThe hosted PDF contains numbered slides 1–71 only. The appendix divider and every A-numbered appendix slide are excluded. Each deck reference opens the matching page in the PDF.
33 experiments
| Deck | Experiment / analysis | Sample size | Method | Headline result(s) | Evidence on this site |
|---|---|---|---|---|---|
| 11–13 | PRISMA-ScR scoping review of trustworthiness evaluation of medical VLMs | 516 records screened → 34 studies included | PRISMA-ScR protocol; per-study mapping of datasets, metrics and controls across 8 trustworthiness dimensions; output is the 8-item MiTEC checklist | 30/34 (88.2%) report no image-reliance control; 3/34 (<9%) report even a text-only baseline | JMIR-AI scoping review (paper card) |
| 18 | PSF-Med benchmark construction and equivalence audit | 26,850 source questions; 122,778 candidate pairs → 92,856 evaluated pairs | GPT-4-generated paraphrase candidates; embedding prefilter + clinical rule checks + 10-category rubric audit, GPT-5-mini as primary judge | 50.3% of candidate pairs retained; the evaluated benchmark is 92,856 pairs over MIMIC-CXR, PadChest, VinDr-CXR | Audit-retention chart · 92,856 evaluated pairs · 26,850 questions |
| 20 | Headline flip rates across models and populations | 92,856 pairs × 6 models × 3 datasets | Pairwise flip rate on the equivalence-filtered binary subset; greedy decoding (temperature 0) | Pairwise flip 6.4–54.7% (two degenerate cells set aside); MedGemma-4B query-level 18.1% MIMIC / 32.2% PadChest; cross-model Jaccard 0.13–0.18; RadFM 13.7 → 32.8 → 54.7% across MIMIC → PadChest → VinDr | Flip-rate range 6.4–54.7% · Pairwise flip chart · Query-level flip chart · Live leaderboard |
| 22 | Transformation-type analysis | Benchmark pairs split by linguistic phenomenon | Flip rate by transformation type after the equivalence audit | The four main types sit within ~2 points on MIMIC (8.7–11.1%); negation flips 34.7% only on VinDr | Transformation-type flip chart |
| 10 | Paraphrase embedding-similarity analysis Reported in the PSF-Med paper; not a site artifact |
Benchmark pairs, flipped vs non-flipped | Cosine similarity and Euclidean distance between original and paraphrase, distributions split by flip | Means differ significantly (tiny p) but the distributions overlap almost completely; surface distance is not a usable flip predictor | PSF-Med paper card |
| 21 | Frontier reasoning ablation, matched pairs The matched-ablation numbers are in the MMFM-BIOMED paper; the site chart shows the larger Aug-2026 arm |
1,960 matched paraphrase pairs × 3 frontier models | Same pairs with reasoning disabled, then enabled; paired within-model comparison | GPT-5-mini 7.76 → 8.21% (p = 0.57, ns); Claude Haiku 4.5 17.19 → 22.65% (+5.46 pp, p = 1.4e-5); Gemini 3 Flash 12.65 → 14.95% (+2.30 pp, p = 0.025) | MMFM-BIOMED paper card · Frontier flip chart |
| 21 | Matched-set frontier comparison (apples-to-apples) | 5,613 (question, paraphrase) pairs readable by all seven arms; 1,172 fully matched questions | Every arm scored on the identical pair set — a pair enters only if all arms answered both the original and that paraphrase readably; pairwise flip with Wilson 95% intervals | Pooled: Haiku 11.28 > Sol 9.51 ≈ Opus-on 9.12 > Opus-off 7.79 > Kimi 5.77 > GPT-5-mini 5.08 ≈ Muse 4.69 — the unrestricted ordering is unchanged. Exact McNemar on the identical pairs: Opus reasoning-on is significantly worse than off (+1.33 pp, p = 7.6e-3); Sol worse than Opus-off (p = 8.7e-4), tied with Opus-on (p = 0.48); Haiku worse than both (p ≤ 2e-3); Kimi ≈ GPT-5-mini (p = 0.09) | Full matched-set accounting · Unrestricted frontier chart |
| 24–25 | Image-reliance controls on the 107-pair diagnostic set | 107 pairs × 3 models × 4 image conditions | Real image / text-only / gray placeholder / opposite-label swap; metrics: text-only agreement, swap sensitivity, image-agnostic rate | MedGemma-4B: flip 42.1%, text-only 66.4%, swap 30.8%, image-agnostic 43.0%; MedGemma-27B: 14.0 / 85.0 / 19.6 / 60.7; LLaVA-Rad: 6.5 / 96.3 / 10.3 / 85.0 — the most consistent model is the least image-reliant | LLaVA-Rad text-only 96.3% · MedGemma-4B text-only 66.4% · Swap sensitivity 30.8% · Flip × text-only correlation · Text-only agreement chart · Image-swap chart |
| 26 | Pixel-edit control (visual-pathway stability) Reported in the Consistent-but-Dangerous paper; not a site artifact |
500-question PadChest sample | JPEG q95 recompression and 5% global contrast shift; greedy decoding rules out sampling noise | Answer changes: Base 1.2% (0.6% contrast-only), Targeted LoRA 2.0%, Full LoRA 1.4% — over an order of magnitude below the 42.1% paraphrase flip rate on the same model | Consistent but Dangerous (paper card) |
| 27 | Attention on flipped vs stable answers | Balanced annotated sample (200 annotated PadChest questions) | Ground-truth box coverage and top-attention precision, split by flip | Coverage 8.4% on flips vs 14.2% on stable (−41%); precision of the strongest attention 29.0% → 10.6% (−63%) | Coverage 8.4% vs 14.2% |
| 30–32 | Sparse-autoencoder transfer to MedGemma | Layer-17 residual stream (2,560-d → 16,384 features); medical and general prompt sets | GemmaScope 2 JumpReLU SAEs transferred with no retraining; reconstruction scored on medical prompts | R² = 0.9972 (medical) vs 0.9974 (general); 0.28% unexplained variance; ~77 of 16,384 features fire per token | Layer-17 feature chart |
| 33 | Two-stage candidate circuit (feature 3818 → 12139) | FlipBank pairs; 76 operator-preserving flips; 37 mediation pairs | SAE feature deltas at the final token; causal activation patching; mediation test 3818@L17 → 12139@L29 | Register gate 3818: 344.5 units on presence framing vs 0 on exclusion; patch moves the margin −0.625 → +2.0 (28% recovery vs 8% for 10 matched controls); 37/37 direction-coherent; top L17 delta in 37/76 flips (48.7%) but ablation restores only 6/76; 12139 is the top delta in 76/76 | Patch recovery 28% · Top-1 in 48.7% of operator-preserving flips · Claim: two-stage candidate |
| 34 | Domain-adapted transcoders (medgemma-circuit-tools) and the instrument-change test | Two top-k transcoders (layers 17 and 29); 2,560 → 20,480 features, top-k 64; 50,000 steps on 1.6M ReXGradient-160K documents | Transcoders trained from scratch on MedGemma-4B MLP activations from radiology reports; then substituted for Gemma Scope 2 with identical pairs, arms and statistics | Explained variance 0.996 in-distribution, and the EV ordering inverts at the domain boundary (register layer medically specialized, decision layer generic); the correlational register signal transfers (|Δf| AUROC up to 0.885 frozen / 0.948 split-half) but single-feature causal restoration collapses from 17–58% to 0–3% — the single-feature two-stage circuit reads as an instrument artifact | Weights, code and full tables · Claim this instruments |
| 35 | Lens-free residual transplant (commit localization) | 1,396 transplant pairs, 91 findings | Transplant the correctly-answering residual stream into the flipped run at each layer; image-token-position and reverse-direction controls | Flips restored 8% → 35% → 73% across layers 14 → 15 → 16, saturating by 19; median commit layer 16, 95% bootstrap CI [16, 16]; 71% commit within layers 15–17; image-token control ≤ 0.0007 at every layer | 73% commit at layer 16 · Commit-by-layer chart · Interactive commit demo |
| 36 | Layer-window sweep (negative result) | 5-layer windows across all 34 layers + 20 random subsets | LoRA adaptation per window; flip rate per window | Mechanistically identified window L15–19 = 5.08%; early-to-mid windows go lower; full-model adaptation 3.11% — locating is not intervening | Layer-ablation chart · Early vs middle result · Claim: locate ≠ intervene |
| 37 | Jacobian-lens readout of the commit layer | Six shared PadChest cases; per-layer, per-token readout on MedGemma-4B and the FlipLens probe | At every layer, read the yes/no margin the model is disposed to produce (first-order; no training, no patching) | Two phrasings travel together, then split at the causally established commit layer (~16) on MedGemma; on the probe, flipping clusters diverge from stable ones from the input layer onward (2.2× divergence; 8.9× on the earlier probe); the lens ports to Qwen2-VL unchanged, and a weak readout there marks lens faithfulness, not model robustness | Interactive Jacobian-lens demo · The causal commit it corroborates |
| 39–40 | FlipLens (BabyMedGemma) probe training and transfer | 86,288 binary presence questions (NIH ChestX-ray14 + PadChest); MIMIC-CXR and VinDr-CXR held out entirely | Frozen MedSigLIP-448 encoder + 14.1M Gemma-3 decoder; every split balanced per finding so wording predicts the answer at chance | In-distribution accuracy 74.8% / AUROC 0.827; MIMIC 0.671 / 0.743; VinDr 0.686 / 0.756; text-only 50.0% (chance) — it genuinely reads the image and transfers to two unseen hospitals | Probe transfer AUROC 0.756 · FlipLens weights |
| 41 | Training-phrasing regime experiment | 3 regimes × 8 seeds; 48-template paraphrase bank; 24 phrasings withheld from training | Only the training-phrasing distribution changes (augmented / canonical / adversarial); architecture, parameters, seed and evaluation held fixed | Flip rate (all / unseen phrasings): augmented 4.8 / 26.6%, canonical 67.1 / 65.9%, adversarial 88.4 / 87.4%; accuracy 75.3 / 66.8 / 58.1%; Mann–Whitney p = 1.6e-4, Cliff's δ = 1.00; text-only accuracy 49.4–50.7% throughout | Regime flip rates (4.8 / 67.1 / 88.4%) · Causality page |
| 42 | Blind / shuffled / seeing probe variants Reported in the CHIL paper (probe measurement analysis); not a site artifact |
Balanced held-out set, 5 seeds, 3 probe variants | Probe given no pooled image summary vs a shuffled one vs the correct one; accuracy vs margin AUROC | Accuracy flat across blind / shuffled / seeing (50.1 / 52.2 / 52.6%); margin AUROC 0.500 / 0.502 / 0.604 — binary accuracy cannot certify image use | CHIL paper card |
| 43 | Single-pass detection asymmetry | Probe evaluations in-distribution + MIMIC + VinDr; 345 patient-seed evaluations | Absolute answer margin (1 pass) vs sampling self-consistency (k passes) vs a trained hidden-state probe (1 pass + fit); image swap (2 passes) for reliance | Flip detection AUROC 0.923 / 0.974 / 0.974 (margin) vs 0.709–0.786 (self-consistency) vs 0.826–0.838 (hidden-state probe); image-unreliance: margin 0.470–0.519 (chance) vs image swap 0.827–0.907; 0/345 identifiable per-case grounded-to-unreliant events | Interactive margin demo |
| 38 | Targeted LoRA mitigation | Patient- and image-disjoint MIMIC presence test: 238 questions, 991 pairs; 5 seeds | LoRA rank 16 on layers 15–19 only (4.38M params, 0.10%), trained on operator-preserving paraphrases; ~620 min on one A100 | Pairwise flip 8.5 → 3.5% (−59% ± 0.6 pp; McNemar p ≤ 1.8e-3 in every seed); query-level 19.8 → 8.1%; accuracy 84.5 → 84.7% (paired difference [−1.00, +1.51] pp); PadChest accuracy 85.2 → 91.6% (+6.4 pp) | Pairwise flip 8.5 → 3.5% · Accuracy delta interval · 0.10% of parameters · PadChest 91.6% · Seed-by-seed chart · Patient-disjoint accuracy chart |
| 46, 49 | Four-quadrant safety screen and per-finding rules | All scored predictions across 10 model–dataset settings; per-finding shares on PadChest | Behavioral axes paraphrase consistency × image reliance give Ideal / Fragile / Dangerous / Worst cells | Mean 81% of consistent predictions are image-invariant; the Dangerous cell dominates for most models; text-driven share: vertebral degenerative changes 95.7%, cardiomegaly 80.4%, interstitial pattern 5.0%, infiltrates / vascular hilar 0.0% | Four-quadrant chart · 81% image-invariant · Claim: consistency ≠ safety |
| 47 | Convergent validation of the screen labels The KL-divergence AUROC 0.76 is reported in the iMIMIC paper; not a site artifact |
Predictions scored by the quadrant screen | Image-off vs image-on KL divergence; swap invariance; null-image control — none reuses the quadrant label | KL divergence separates Dangerous from Ideal at AUROC 0.76; swap and null-image checks agree with the behavioral labels (the screen is not circular) | iMIMIC paper card |
| 48 | Attention fidelity vs causal use | 637 annotated PadChest boxes | True-box vs displaced-box vs random-far attention coverage; Spearman ρ of attention rank vs causal importance; ROI-deletion grounding score (region ablation minus same-size random ablation) | True-box coverage 29.6% (Base) / 38.5% (Full LoRA) vs displaced 26.1 / 33.8 and random-far 5.5 / 7.4; ρ = −0.09 / +0.06 (attention does not rank cause); ROI-deletion 2.97 [2.67, 3.26] / 1.18 / 0.57, all excluding zero | ρ = −0.09 attention vs cause · True vs shifted box coverage · Claim: heatmaps not faithful |
| 50 | Calibration–fairness audit | PadChest demographic strata (861 questions) | Sex- and age-stratified accuracy and ECE per model variant | Temperature-scaled ECE 13.6 / 3.6 / 22.9% (Base / Targeted / Full); sex accuracy gap 2.0 / 2.7 / 4.3 pp; age range 4.3 / 13.0 / 8.2 pp — the best-calibrated model has the steepest age gradient; no model is uniformly most equitable | Accuracy by age band · 13 pp age gradient · ECE sex gap 0.012 · Claim: partly |
| 53–55 | Uncertainty-method comparison for flip and error detection | PadChest, Targeted LoRA (n = 861) | Softmax entropy, yes/no margin, temperature scaling, MC Dropout (K = 10), deep ensemble (5 seeds) | Softmax entropy AUROC 0.823 flip / 0.862 error in one pass; the margin gives the identical ranking; temperature scaling fixes ECE but moves no ranking; MC Dropout 0.823 / 0.856 at 10× cost (no gain); deep ensemble 0.751 error, AURC 0.130 vs 0.019 | Entropy flip AUROC 0.823 · Entropy error AUROC 0.862 · Bridge AUROC chart · Entropy distribution · Interactive entropy demo |
| 57 | Temperature scaling and calibration | 15% calibration holdout, PadChest | Fit one scalar temperature per model | ECE: Base 0.16 → 0.14, Targeted LoRA 0.11 → 0.04, Full LoRA 0.27 → 0.23; ranking unchanged (monotone transform); the best-calibrated variant also has the steepest age gradient | UNSURE paper card · ECE results (safety page) |
| 58–60 | Selective prediction, AUGRC corruption sweep, conformal coverage | PadChest; corruption severities 1–5; 15% conformal holdout | Sort by entropy and admit the lowest fraction; AUGRC under image corruption; conformal (APS score) at a 90% target | At 40% coverage: error 0.6% and flips 2.3%, vs 8.6% / 13.2% at full coverage; AUGRC: Targeted 0.014 → 0.027, Base 0.047 → 0.088, Full 0.153 → 0.186 (severity 5); conformal: 93.7% clean → 89.9% at severity 5 (Base under-covers at 88.7%) | Risk–coverage chart · AUGRC degradation chart · AUGRC 0.016 (Targeted) · Conformal 88.7% (Base, severity 5) |
| 56 | Deep-ensemble negative result and entropy decomposition | 5 LoRA seeds | Deep ensemble across seeds; mutual-information decomposition MI = H[p̄] − mean H[pₖ]; MC Dropout comparison | Seeds span 52–92% PadChest accuracy (mean 72.6%); ensemble error AUROC 0.751 vs 0.862 single-pass; ensemble MI 0.057 nats (real but contaminated) vs MC Dropout MI ≈ 0.0001 (uninformative probe) — neither licenses an aleatoric reading | Ensemble OOD failure (52%) · MC Dropout MI ≈ 0.0001 |
| 59–60 | Admission-gate audit | PadChest audit; 279 text-image disagreement cases (Qwen2-VL) | Architecture-aware audit of gating rules: null-image rule, strict all-paraphrase-agreement rule, and looser variants | Null-image rule admits 72% at 98.9% accuracy; strict rule admits 10% at 96.7%; even the strict subset stays 94.4% swap-invariant; the adapter scores 2.9% on text-image disagreements and Qwen2-VL 0/279 — gates raise accuracy by selecting the text-answerable slice | Deployment-gate chart · Admission 33.0% · Grounded-slice 2.9% · Claim: gates admit the text slice · Interactive gate demo |
| 61 | Cross-architecture entropy transfer | MIMIC + PadChest; LLaVA-Rad LoRA; operator-preserving ex-negation slice | Single-pass predictive entropy as a flip predictor on a second architecture family | AUROC 0.823 (MedGemma Targeted LoRA / PadChest), 0.905 (LLaVA-Rad / MIMIC), 0.830 (LLaVA-Rad / PadChest); ex-negation slice 0.826 [0.780, 0.870]; mean entropy 0.585 nats on flipped vs 0.419 on stable predictions | Bridge AUROC chart · Entropy flip AUROC |
| 62 | Per-family monitor comparison (RQ5.4) | Gemma / LLaVA-Rad / Qwen2-VL probes | Internal probes per architecture family | Gemma shows late readout-misalignment (AUC 0.81–0.92); LLaVA-Rad fails earlier in transport; Qwen2-VL sits near chance on both signatures — each family needs its own monitor | Claim: no monitor transfers |
| 18 | Frontier judges for the equivalence audit | 122,778-pair audit; 500-pair stratified cross-check | GPT-4 generates paraphrase candidates; GPT-5-mini is the primary rubric judge; Claude Haiku 4.5 runs an independent cross-family check | Cross-family judge agreement 91.6–94.4% | Judge agreement 91.6–94.4% |
Only analyses represented in the main defense presentation are listed. Appendix-only analyses are deliberately excluded.