Measurement

How far this goesTwo of the 18 cells are set aside as degenerate yes-bias and excluded from this range: LLaVA-Rad on PadChest (99.6% positive predictions on originals) and MedGemma-1.5-4B on VinDr (100%). Both post a near-zero flip rate (0.8%) that reflects a single-class prior rather than paraphrase stability, so a model that answers 'yes' to everything would look maximally consistent here. The range also confounds model and population: it spans three different countries' patient distributions, so a cell's position reflects both the model and where the data came from.

How far this goesThe MIMIC binary subset is not a pure presence set: it mixes 956 presence questions, 177 view-identification questions and 406 abnormality questions, so the rate averages over question types that may not be equally fragile. This is the pairwise denominator (flipped paraphrases over paraphrase pairs); the query-level rate on the same questions is more than double at 18.1%, and the two must never be interchanged. Do not compare this figure to the 42.1% MedGemma-4B flip rate on the 107-pair diagnostic set of Chapter 4, which is a different, deliberately hard population.

How far this goes92,856 is NOT the same thing as the roughly 92,000 candidate paraphrases generated at construction time; the two numbers are close by coincidence and describe different pipeline stages. The construction stage produced about 92,000 candidates (mean 4.7 per question) before filtering; 92,856 is the post-audit evaluation set (mean 3.5 per question). It is also not a subset of the 61,761 audit-retained core, because the audited PadChest v1 pairs were superseded by a separately filtered PadChest v2 set. Sources that report 'about 92,000 pairs' as the evaluation size are blurring two stages.

How far this goesThe 92,856-pair evaluation set is NOT a subset of the 61,761-pair retained core: the audit's retained PadChest v1 pairs were superseded by a regenerated, separately judge-filtered PadChest v2 set (59,573 pairs), which combined with audit-retained MIMIC (8,938) and VinDr (24,345) gives the 92,856 evaluated pairs. So the 50.3% describes the audit population, not the shipped benchmark. These tallies supersede an earlier snapshot of the same audit (60,712 retained / 60,766 rejected / 1,300 uncertain) taken while the VinDr run was still completing. Retention is a judge verdict, not a clinician verdict; the clinician sample agrees with this judge only 72.3% of the time.

How far this goesThis validates the judge, NOT the benchmark's flip rates. The 1,200-pair sample was drawn before the PadChest v2 regeneration, so only 95 of 174 MIMIC pairs, 45 of 658 PadChest pairs and 0 of 43 VinDr pairs are in the evaluated benchmark at all. It therefore cannot be used to restate any flip rate; on the 95 overlapping MIMIC pairs, restricting to clinician-confirmed equivalents moves per-model flip rates by only -2.5 to +1.1 percentage points, within sampling noise. An earlier '0.9 pp sensitivity' claim from this sample was unreproducible and has been removed. The sample also deliberately over-samples adversarial and uncertain pairs, so 72.3% is a lower bound on agreement over a natural pair mix, not an unbiased estimate.