| PSF-Med main benchmark |
92,856 pairs |
The full evaluated benchmark: 92,856 final evaluation pairs built from 26,850 original clinical questions (MIMIC-CXR 8,938 pairs, PadChest v2 59,573, VinDr-CXR 24,345; mean 3.5 paraphrases per question). Source of the headline flip rates. |
The 92,856 evaluation pairs are numerically close to the ~92,000 construction-stage candidate paraphrases by coincidence and are NOT a subset of the 61,761 audit-retained core (PadChest v1 audited pairs were superseded by regenerated v2). Never equate pipeline stages, and never compare these rates with curated flip-bank rates. |
| Binary yes/no subset |
49,932 pairs |
The binary presence-question subset of the main benchmark used for headline pairwise flip rates: MIMIC-CXR 1,539 questions / 5,076 pairs; PadChest 8,445 / 36,244; VinDr-CXR 2,807 / 8,612. |
Pairwise flip rates on this subset are roughly half the query-level rates on the same questions (MedGemma-4B PadChest: 13.4% pairwise vs 32.2% query-level). Never mix the two denominators, and never compare these rates against curated flip-bank rates. |
| Mechanistic FlipBank (158 pairs) |
158 pairs |
A 158-pair curated set of confirmed flips (94 yes-to-no, 64 no-to-yes, 142 images, 23 findings) used for the sparse-autoencoder mechanistic analyses on MedGemma-4B. |
Every pair in this set flips by construction, so any flip statistic computed on it is 100% by design; it exists to study mechanisms, never to estimate flip prevalence or to compare against benchmark rates. |
| Three-backend diagnostic set (107 pairs) |
107 pairs |
A curated 107-pair set of semantically close question-paraphrase pairs (BioClinicalBERT cosine similarity > 0.95) on which MedGemma-4B, MedGemma-27B, and LLaVA-Rad were all evaluated for the text-only, image-swap, and attention diagnostics. Constructed before the equivalence audit. |
MedGemma-4B flips 42.1% on this curated near-paraphrase set versus 8.3% on the benchmark; the two rates measure different things and must never be compared. The set also predates the equivalence audit, so its flip labels contain operator changes. |
| PadChest flip bank (861 questions) |
861 questions |
An 861-question PadChest bank curated to be flip-prone, used for the four-quadrant audit, uncertainty quantification, the entropy bridge, and the deployment gate on MedGemma variants. |
This bank was curated to be flip-prone: Base MedGemma-4B flips 73.9% of it versus 13.4% pairwise on the PadChest benchmark. Quadrant shares, AUROCs, and gate numbers computed here must never be quoted as benchmark statistics. |
| MIMIC uncertainty set (98 questions) |
98 questions |
The full 98-question MIMIC-CXR presence-question test set used for the four-quadrant audit and uncertainty analyses of the MedGemma variants; not truncated. |
Only 98 questions with no demographic attributes on disk: quadrant and calibration estimates carry wide intervals and cannot be pooled with the 861-question PadChest bank or the main benchmark. |
| LLaVA-Rad PadChest set (732 questions) |
732 questions |
The 732-question subset of the PadChest flip bank on which LLaVA-Rad variants are evaluated (the temperature-scaling calibration holdout also leaves 732 of 861 for evaluation). |
A 732-question subset of the 861-question PadChest bank: LLaVA-Rad numbers computed here are not directly comparable to MedGemma numbers computed on the full 861 questions. |
| Patient-disjoint mitigation endpoint (238 questions) |
238 questions |
The clean mitigation evaluation set: 238 presence-of-finding questions / 991 pairs over 236 held-out subjects and 238 held-out images, with zero subject and zero image overlap with LoRA training data. |
This is the only clean mitigation endpoint: its ~59% pairwise flip reduction (8.5% to 3.5%) supersedes the retracted 79.5% (a margin-difference reduction) and 69.6% (an older flip reduction) figures, and must not be compared with numbers from the original non-disjoint pipeline. |
| Attention analysis subsample (200 pairs) |
200 pairs |
A 200-pair attention-analysis subsample balanced at 100 flip / 100 no-flip pairs, drawn from a 49,522-pair annotated pool. |
Balanced by construction at 50% flips: any aggregate flip statistic on this set is a design choice, not an observation, and attention contrasts here cannot be projected onto benchmark populations. |
| PadChest grounding subset (637 boxes) |
637 boxes |
The 637 all-positive PadChest cases with radiologist bounding boxes used for attention-grounding coverage (true vs shifted vs random-far box) and ROI causal scoring. |
All 637 cases are positive findings with radiologist boxes: coverage and ROI scores say nothing about negative findings, and boxes exist for only 637 of the 861 PadChest bank questions and none of MIMIC, which is also why the gate's region term is not deployable online. |
| Clinician adjudication sample (1,200 pairs) |
1,200 pairs |
A 1,200-pair stratified sample adjudicated by three clinician reviewers (400 pairs triple-reviewed; inter-reviewer agreement 98.5%, kappa 0.97) to validate the LLM equivalence judge (72.3% raw agreement, kappa 0.52). |
Drawn from the pre-regeneration v1 pool: only 95 of 174 MIMIC, 45 of 658 PadChest, and 0 of 43 VinDr pairs remain in the evaluated benchmark. It validates the equivalence judge, not the benchmark's flip rates; the earlier 0.9 pp sensitivity claim is retracted. |
| Canonical SAE screen (41,822 pairs) |
41,822 pairs |
The canonical sparse-autoencoder screening pool: 4,542 MIMIC pairs (436 flips, 9.6%) and 37,280 PadChest pairs (2,833 flips, 7.6%) used for the two-stage L17/#3818 to L29/#12139 validation, mediation, and composition analyses. |
171 of the 436 verified MIMIC flips (39%) are negation-pattern or operator changes, so restoration and mediation rates conflate correct operator handling with genuine same-polarity paraphrase sensitivity; do not read them as benchmark-level causal effect sizes. |
| Answer-commitment transplant pool (1,396 pairs) |
1,396 pairs |
The 1,396-pair / 91-finding pool used for the residual-transplant study that locates the answer-commitment locus at layer 16 (flip rate 8% at L14, 73% at L16, 100% by L20; median commit layer 16, 95% bootstrap CI [16, 16]). |
The pool is 94% positive (1,314 yes vs 82 no), 7.7% of pairs carry one-sided qualifiers, and the pair-pool construction script was not retained; commit-layer statistics are specific to this pool and are evidence of a general commitment locus, not a paraphrase-specific mechanism. |
| Layer-ablation validation split (355 questions) |
355 questions |
The 355-question MIMIC-CXR validation split used to compare LoRA layer ranges (early 0-10, random 5-9, all 0-33, middle 15-19, late 25-33) on margin difference. |
A validation split used for model selection, not the held-out test set or the patient-disjoint endpoint: margin-difference results here (early layers 0.26 beating the mechanistic middle 0.38) must not be quoted alongside patient-disjoint endpoint numbers. |
| PadChest transfer set (250 questions) |
250 questions |
A 250-question class-balanced PadChest set (125 yes / 125 no) used as the out-of-distribution transfer test for the targeted LoRA (accuracy 85.2% to 91.6%, margin difference 1.02 to 0.25). |
Balanced 50/50 by construction, so the base model's 7.6% flip rate here sits far below its 13.4% pairwise PadChest benchmark rate; the transfer gain shows up in accuracy and margin sharpening, not in flip reduction, and neither number generalizes to the unbalanced benchmark. |
| Four-quadrant audit settings (10 model-dataset settings) |
Ten model-dataset settings; per-setting question counts differ (MIMIC 98 or 88, PadChest 861 or 732), so there is no single pooled n. |
The ten model-dataset settings of the four-quadrant safety screen, pooling the MIMIC flip banks (n=98 for MedGemma variants, n=88 for LLaVA-Rad variants) and the PadChest flip banks (n=861 and n=732) across five models. |
The 81% image-invariant share is a level averaged over ten settings whose underlying flip banks were curated to be flip-prone; it is not a benchmark statistic, and the r=-0.86 consistency-grounding correlation over the same ten points is largely definitional (conditioned on consistency it falls to r=-0.15). |