Diagnosis

How far this goesThe 107-pair diagnostic set is NOT comparable to the main benchmark: MedGemma-4B flips on 42.1% of these pairs against 8.3% on the benchmark, roughly 5x, because the set was curated for semantic closeness on a small pool. Read these numbers as a within-set contrast between backends, never against a benchmark flip rate. The set also predates the equivalence audit, so some pairs are operator changes rather than true paraphrases. Text-only agreement is a behavioural screen, not a per-prediction verdict: a model can agree with itself without the image and still be right, because base rates make the text prior usually correct.

How far this goesThe 107-pair diagnostic set is NOT comparable to the main benchmark: MedGemma-4B flips on 42.1% of these pairs against 8.3% on the benchmark, roughly 5x. Do not read the 42.1% as a benchmark result or compare this text-only rate to rates measured on other evaluation sets. The set predates the equivalence audit, so some pairs are operator changes. Lower text-only agreement means less image-invariance, which is desirable, but it does not establish that the model uses the image *correctly*: MedGemma-4B also has the worst robust accuracy of the three backends here (36.4%).

How far this goesThe 107-pair diagnostic set is NOT comparable to the main benchmark: MedGemma-4B flips on 42.1% of these pairs against 8.3% on the benchmark, roughly 5x, so no rate here transfers to a benchmark claim. Swap sensitivity has no correctness axis: a model that reacts to a swapped image is using the image, but the reaction can be in either direction and may be wrong. Even the most reactive backend here ignores the swap on 69% of pairs. The set predates the equivalence audit.

How far this goesThree points cannot establish a law. An r computed on n=3 carries almost no inferential content: almost any three monotone points yield |r| near 1, and a single additional model could destroy it. This is ILLUSTRATIVE of a tendency, not evidence of a deterministic trade-off, and it must never be reported as if it were fitted on the benchmark. The related r = -0.86 correlation in the safety chapter is separately known to be largely definitional, falling to r = -0.15 once conditioned on consistency. The 107-pair set is also not comparable to the main benchmark, where MedGemma-4B flips on 8.3% of pairs rather than 42.1%.

How far this goesNo displaced-box or random-location control was run at this n=200, so the absolute coverage values are meaningless on their own: only the flip vs no-flip contrast is load-bearing. The larger 637-box grounding study in Chapter 6 shows why this matters, since attention there only marginally beats a deliberately shifted box, meaning it is a coarse localizer rather than a faithful one. The flip/no-flip split also uses pairs from before the equivalence audit, so some 'flip' cases are operator changes rather than paraphrases and the gap may be partly confounded by transformation type; the direction holds but the magnitude is suggestive. This chapter's diagnostic sets, including this one, are separate curated populations and are not comparable to the main benchmark, where MedGemma-4B flips on 8.3% of pairs against 42.1% on the companion 107-pair set.