Failure cases

What a paraphrase flip looks like

Five curated cases where semantically equivalent clinical questions produce contradictory answers. The benchmark tells you how often this happens. These show you what it looks like when it does.

What this gallery is, and is not These are hand-picked illustrations, not benchmark measurements. Two of the three models shown here (GPT-5-mini and Claude Haiku 4.5) are general-purpose models that sit outside the dissertation's six medical vision-language models, so their answers do not appear in any thesis table. Read this page for the shape of the failure, and the evidence explorer for the rates.

Cases

Show only flipped paraphrases

Case 01 · Curated demonstration set (MIMIC-CXR)

Pneumothorax

3 flipped answers 3 of 3 models flip

The radiograph is not shown

This case uses a MIMIC-CXR study. The PhysioNet licence for MIMIC-CXR prohibits redistributing images, so the radiograph is not published here. Anyone with credentialed access can obtain it from PhysioNet.

What this case demonstrates is the question and the answers, and both are below. The image itself is not what the flip is about: the image never changes across a case, only the wording does.

Original question

Is there evidence of pneumothorax in this image?

Reference answer

Why this case mattersThe image shows no pneumothorax, yet a single more specific rewording about pleural air pushes all three models in this illustration into the same false positive.

Why these questions count as equivalentEach paraphrase keeps the same clinical operator (presence of a finding) and the same finding (pneumothorax), varying only syntax, scope wording, specificity, or synonyms, which is the class of rewrite the equivalence audit rubric treats as clinically equivalent.

MedGemma-4B

Correct until rephrased

Answer to the original question: No

Does this image demonstrate a pneumothorax?

Syntactic restructuring No Same as the original Matches the reference answer
Is there evidence of pneumothorax, unilateral or bilateral, on this image?

Scope quantification No Same as the original Matches the reference answer
Is there radiographic evidence of pleural air consistent with pneumothorax in this chest radiograph?

Specificity modulation Yes Flipped from the original Conflicts with the reference answer
Is there air in the pleural space suggesting pneumothorax on this image?

Lexical substitution No Same as the original Matches the reference answer

GPT-5-mini

Correct until rephrased

Answer to the original question: No

Does this image demonstrate a pneumothorax?

Syntactic restructuring No Same as the original Matches the reference answer
Is there evidence of pneumothorax, unilateral or bilateral, on this image?

Scope quantification No Same as the original Matches the reference answer
Is there radiographic evidence of pleural air consistent with pneumothorax in this chest radiograph?

Specificity modulation Yes Flipped from the original Conflicts with the reference answer
Is there air in the pleural space suggesting pneumothorax on this image?

Lexical substitution No Same as the original Matches the reference answer

Claude Haiku 4.5

Correct until rephrased

Answer to the original question: No

Does this image demonstrate a pneumothorax?

Syntactic restructuring No Same as the original Matches the reference answer
Is there evidence of pneumothorax, unilateral or bilateral, on this image?

Scope quantification No Same as the original Matches the reference answer
Is there radiographic evidence of pleural air consistent with pneumothorax in this chest radiograph?

Specificity modulation Yes Flipped from the original Conflicts with the reference answer
Is there air in the pleural space suggesting pneumothorax on this image?

Lexical substitution No Same as the original Matches the reference answer

The claim this illustrates · Benchmark flip rates · The PSF-Med paper

Case 02 · Curated demonstration set (MIMIC-CXR)

Apical Scarring

6 flipped answers 3 of 3 models flip

The radiograph is not shown

What this case demonstrates is the question and the answers, and both are below. The image itself is not what the flip is about: the image never changes across a case, only the wording does.

Original question

Is there scarring in the lung apices?

Reference answer

Yes

Why this case mattersOn one image the three models in this illustration disagree with each other and with themselves: one recovers the correct answer only after a rewording, and another changes on every rewording.

Why these questions count as equivalentEach paraphrase keeps the presence-of-finding operator and the same finding in the same anatomical region (apical scarring, with fibrosis as an accepted synonym), varying only syntax, scope wording, or lexical choice, the rewrite classes the equivalence audit rubric treats as clinically equivalent.

MedGemma-4B

Wrong and unstable

Answer to the original question: No

Is there fibrosis in the apical regions of the lungs?

Lexical substitution No Same as the original Conflicts with the reference answer
Can scarring be seen in the apices of the lungs?

Syntactic restructuring Yes Flipped from the original Matches the reference answer
Is there any scarring in the apices of either lung?

Scope quantification No Same as the original Conflicts with the reference answer
Do the lung apices show evidence of scarring?

Syntactic restructuring No Same as the original Conflicts with the reference answer

GPT-5-mini

Correct until rephrased

Answer to the original question: Yes

Is there fibrosis in the apical regions of the lungs?

Lexical substitution No Flipped from the original Conflicts with the reference answer
Can scarring be seen in the apices of the lungs?

Syntactic restructuring Yes Same as the original Matches the reference answer
Is there any scarring in the apices of either lung?

Scope quantification Yes Same as the original Matches the reference answer
Do the lung apices show evidence of scarring?

Syntactic restructuring Yes Same as the original Matches the reference answer

Claude Haiku 4.5

Wrong and unstable

Answer to the original question: No

Is there fibrosis in the apical regions of the lungs?

Lexical substitution Yes Flipped from the original Matches the reference answer
Can scarring be seen in the apices of the lungs?

Syntactic restructuring Yes Flipped from the original Matches the reference answer
Is there any scarring in the apices of either lung?

Scope quantification Yes Flipped from the original Matches the reference answer
Do the lung apices show evidence of scarring?

Syntactic restructuring Yes Flipped from the original Matches the reference answer

The claim this illustrates · Benchmark flip rates · The PSF-Med paper

Case 03 · Curated demonstration set (MIMIC-CXR)

Atelectasis

4 flipped answers 2 of 3 models flip

The radiograph is not shown

What this case demonstrates is the question and the answers, and both are below. The image itself is not what the flip is about: the image never changes across a case, only the wording does.

Original question

Is there evidence of atelectasis in this image?

Reference answer

Yes

Why this case mattersSubstituting the synonym pulmonary collapse for atelectasis changes the answer for two of the three models in this illustration, even though the question asks the same clinical thing.

Why these questions count as equivalentEach paraphrase keeps the presence-of-finding operator and the same finding (atelectasis, with pulmonary collapse as an accepted synonym), varying only syntax, scope wording, specificity, or lexical choice, the rewrite classes the equivalence audit rubric treats as clinically equivalent.

MedGemma-4B

Correct until rephrased

Answer to the original question: Yes

Is there evidence of pulmonary collapse on this image?

Lexical substitution No Flipped from the original Conflicts with the reference answer
Does this image show evidence of atelectasis?

Syntactic restructuring Yes Same as the original Matches the reference answer
Is there any atelectasis visible on this image?

Scope quantification Yes Same as the original Matches the reference answer
Is there evidence of atelectasis in the lungs on this image?

Specificity modulation Yes Same as the original Matches the reference answer
Are there radiographic signs of pulmonary collapse on this image?

Lexical substitution No Flipped from the original Conflicts with the reference answer

GPT-5-mini

Correct until rephrased

Answer to the original question: Yes

Is there evidence of pulmonary collapse on this image?

Lexical substitution No Flipped from the original Conflicts with the reference answer
Does this image show evidence of atelectasis?

Syntactic restructuring Yes Same as the original Matches the reference answer
Is there any atelectasis visible on this image?

Scope quantification Yes Same as the original Matches the reference answer
Is there evidence of atelectasis in the lungs on this image?

Specificity modulation Yes Same as the original Matches the reference answer
Are there radiographic signs of pulmonary collapse on this image?

Lexical substitution No Flipped from the original Conflicts with the reference answer

Claude Haiku 4.5

Consistent and correct

Answer to the original question: Yes

Is there evidence of pulmonary collapse on this image?

Lexical substitution Yes Same as the original Matches the reference answer
Does this image show evidence of atelectasis?

Syntactic restructuring Yes Same as the original Matches the reference answer
Is there any atelectasis visible on this image?

Scope quantification Yes Same as the original Matches the reference answer
Is there evidence of atelectasis in the lungs on this image?

Specificity modulation Yes Same as the original Matches the reference answer
Are there radiographic signs of pulmonary collapse on this image?

Lexical substitution Yes Same as the original Matches the reference answer

The claim this illustrates · Benchmark flip rates · The PSF-Med paper

Case 04 · Curated demonstration set (MIMIC-CXR)

Hyperinflation

4 flipped answers 2 of 3 models flip

The radiograph is not shown

What this case demonstrates is the question and the answers, and both are below. The image itself is not what the flip is about: the image never changes across a case, only the wording does.

Original question

Is there hyperinflation in the lungs?

Reference answer

Yes

Why this case mattersOne model in this illustration never changes its answer yet is wrong every time, which is exactly why a low flip rate on its own is not evidence of image-grounded reasoning.

Why these questions count as equivalentEach paraphrase keeps the presence-of-finding operator and the same finding (hyperinflation, with hyperexpansion and overinflation as accepted synonyms), varying only syntax, scope wording, specificity, or lexical choice, the rewrite classes the equivalence audit rubric treats as clinically equivalent.

MedGemma-4B

Correct until rephrased

Answer to the original question: Yes

Is there pulmonary hyperexpansion in the lung fields?

Lexical substitution Yes Same as the original Matches the reference answer
Does the imaging show hyperinflation of the lungs?

Syntactic restructuring No Flipped from the original Conflicts with the reference answer
Is there any hyperinflation in either lung?

Scope quantification No Flipped from the original Conflicts with the reference answer
Is there hyperinflation in the pulmonary parenchyma?

Specificity modulation No Flipped from the original Conflicts with the reference answer
Is there overinflation of the lungs?

Lexical substitution Yes Same as the original Matches the reference answer

GPT-5-mini

Consistent but wrong

Answer to the original question: No

Is there pulmonary hyperexpansion in the lung fields?

Lexical substitution No Same as the original Conflicts with the reference answer
Does the imaging show hyperinflation of the lungs?

Syntactic restructuring No Same as the original Conflicts with the reference answer
Is there any hyperinflation in either lung?

Scope quantification No Same as the original Conflicts with the reference answer
Is there hyperinflation in the pulmonary parenchyma?

Specificity modulation No Same as the original Conflicts with the reference answer
Is there overinflation of the lungs?

Lexical substitution No Same as the original Conflicts with the reference answer

Claude Haiku 4.5

Correct until rephrased

Answer to the original question: Yes

Is there pulmonary hyperexpansion in the lung fields?

Lexical substitution Yes Same as the original Matches the reference answer
Does the imaging show hyperinflation of the lungs?

Syntactic restructuring No Flipped from the original Conflicts with the reference answer
Is there any hyperinflation in either lung?

Scope quantification Yes Same as the original Matches the reference answer
Is there hyperinflation in the pulmonary parenchyma?

Specificity modulation Yes Same as the original Matches the reference answer
Is there overinflation of the lungs?

Lexical substitution Yes Same as the original Matches the reference answer

The claim this illustrates · Benchmark flip rates · The PSF-Med paper

Case 05 · Curated demonstration set (MIMIC-CXR)

Thoracic Aortic Tortuosity

4 flipped answers 2 of 3 models flip

The radiograph is not shown

What this case demonstrates is the question and the answers, and both are below. The image itself is not what the flip is about: the image never changes across a case, only the wording does.

Original question

Is there evidence of tortuosity of the thoracic aorta in this image?

Reference answer

Why this case mattersThe three models in this illustration fail in three different ways on one image: repeated false positives after rewording, stable but wrong throughout, and a single late flip.

Why these questions count as equivalentEach paraphrase keeps the presence-of-finding operator and the same finding in the same structure (tortuosity of the thoracic aorta), varying only syntax, scope wording, or lexical choice, the rewrite classes the equivalence audit rubric treats as clinically equivalent.

MedGemma-4B

Correct until rephrased

Answer to the original question: No

Is there imaging evidence of thoracic aortic tortuosity on this study?

Lexical substitution No Same as the original Matches the reference answer
Does the image show tortuosity of the thoracic aorta?

Syntactic restructuring No Same as the original Matches the reference answer
Is there focal or diffuse tortuosity of the thoracic aorta visible on this image?

Scope quantification No Same as the original Matches the reference answer
On this image, is the thoracic aorta tortuous?

Syntactic restructuring Yes Flipped from the original Conflicts with the reference answer

GPT-5-mini

Correct until rephrased

Answer to the original question: No

Is there imaging evidence of thoracic aortic tortuosity on this study?

Lexical substitution No Same as the original Matches the reference answer
Does the image show tortuosity of the thoracic aorta?

Syntactic restructuring Yes Flipped from the original Conflicts with the reference answer
Is there focal or diffuse tortuosity of the thoracic aorta visible on this image?

Scope quantification Yes Flipped from the original Conflicts with the reference answer
On this image, is the thoracic aorta tortuous?

Syntactic restructuring Yes Flipped from the original Conflicts with the reference answer

Claude Haiku 4.5

Consistent but wrong

Answer to the original question: Yes

Is there imaging evidence of thoracic aortic tortuosity on this study?

Lexical substitution Yes Same as the original Conflicts with the reference answer
Does the image show tortuosity of the thoracic aorta?

Syntactic restructuring Yes Same as the original Conflicts with the reference answer
Is there focal or diffuse tortuosity of the thoracic aorta visible on this image?

Scope quantification Yes Same as the original Conflicts with the reference answer
On this image, is the thoracic aorta tortuous?

Syntactic restructuring Yes Same as the original Conflicts with the reference answer

The claim this illustrates · Benchmark flip rates · The PSF-Med paper

Images and licensing

This site publishes no chest X-ray images. These five cases are built on MIMIC-CXR studies, and the PhysioNet Credentialed Health Data Licence that governs MIMIC-CXR prohibits redistributing them, so the radiographs are described rather than shown. The same holds for the rest of the corpora behind the benchmark: MIMIC-CXR and VinDr-CXR need credentialed PhysioNet access, and PadChest is distributed by BIMCV under its own registration. Anyone reproducing this work obtains the images from those sources directly. The one exception is the dissertation PDF itself, which reproduces a small number of figure panels inside the document.

Full data and image notice