Binesh Sadanandan PhD Dissertation Companion
Read thesis

Paraphrase Sensitivity in Medical Vision-Language Models

Measurement, Mechanisms, Mitigation, and Deployment Safety

Reword a chest X-ray question without changing its meaning, and a medical AI can change its answer. A benchmark counts how often a model is correct. This dissertation measures something else: whether the answer holds when the question is rephrased.

Start with the evidence Read the dissertation How a medical VLM works

Binesh Sadanandan · Doctor of Philosophy in Engineering and Applied Science
Advisor: Vahid Behzadan, Ph.D.

Exhibit A — One question, five ways Replaying a real case
Original question

Is there evidence of pneumothorax in this image?

MedGemma-4B answers No

Reference answer: No. The image shows no pneumothorax.

Case from the failure gallery, which compares MedGemma-4B with two general-purpose reference models; the six-model benchmark is a separate evaluation. The image shows no pneumothorax, yet a single more specific rewording about pleural air pushes all three models into the same false positive.

Exhibit B — the mechanism, live

One line down the network. Then it forks.

The Jacobian lens reads MedGemma-4B's answer margin at every layer, at the word where the answer is decided. Two phrasings of one question travel as a single line — until the answer commits.

Replaying the lens readout

Can an interstitial pattern at the apex be identified?Is there interstitial pattern?

These two are not strict paraphrases. The first asks about an interstitial pattern at the apex; the second asks about an interstitial pattern anywhere. The qualifier narrows the question, so this pair changes scope along with the wording. It is shown because it is where the divergence is easiest to watch, and it is one of the 107 pairs of 1,396, 7.7%, that carry a one-sided qualifier. At the last layer the two readings sit at +15.9 (Yes) and -5.9 (No).

Jacobian lens on MedGemma-4B, read at layers 0-32 of the 34-layer decoder, margins at the answer token. The full readout, every word · The experiments that pin the commit to layer 16

92,856
evaluated question-paraphrase pairs
6
medical models tested
3
countries: US, Spain, Vietnam
5
accepted papers
01 · The core finding

Remove the image. The answer stays.

The dissertation's test: ask the same question with and without the radiograph.

Is there evidence of pneumothorax in this image?

model
No

Is there evidence of pneumothorax in this image?

model
Noidentical
81% of consistent predictions do not change when the image is removed, averaged across ten model-dataset settings. A stable answer is not evidence the model looked.

Sample and source

02 · The instrument

The four-quadrant screen

Cross that test with paraphrase consistency, and every prediction lands in one of four cells.

Choose a quadrant to see what it means, or read the four definitions below.

The four quadrants, as text
Dangerous: Consistent but image-invariant
The model gives the same answer to rephrased questions and the same answer when the image is removed or swapped. Consistency here is a text prior, with no visual reasoning behind it. This is the failure the paper Consistent but Dangerous is named for. See the evidence.
Ideal: Consistent and image-dependent
The model gives the same answer to rephrased questions, and its answer changes when the image changes. This is the target behavior. See the evidence.
Worst: Inconsistent and image-invariant
The answer moves with the wording but not with the image. Neither stable nor grounded. See the evidence.
Fragile: Inconsistent but image-dependent
The model uses the image, but rephrasing still changes the answer. Unstable, yet grounded. See the evidence.
What tuning for consistency does to the screen
Base MedGemma-4B
Each dot is one of 861 PadChest questions. Base MedGemma-4B puts 22.5% of predictions in the Dangerous cell; the consistency-tuned adapter puts 84.1% there. Fewer flips, more text reliance. These fractions are measured on the original-pipeline Targeted LoRA, not the clean patient-disjoint adapters behind the 3.5% flip rate quoted above; the screen has not been re-run on those adapters, so the cell fractions must not be attributed to them, though their re-audit does show the same direction (text-only agreement 53.5% to about 77%).

These are behavioral categories. Correctness is a separate axis. The Dangerous cell — consistent, but identical without the image — is the one accuracy benchmarks cannot see.

03 · The argument

Five thrusts, five findings

Each thrust answers a question the previous one opened.

  1. Measurement How often do equivalent questions change model answers? All six models flip, and scale does not close the gap. 6.4–54.7% pairwise flip range
  2. Some models are strikingly consistent. Is that reasoning, or ignoring the image?
  3. Diagnosis Does consistency mean the model is using the image? No. The most consistent backend barely reads the image. 96.3% LLaVA-Rad answers kept with the image removed, 107-pair diagnostic set
  4. If consistency can be ungrounded, can flips be cut without leaning harder on the text?
  5. Mitigation Can paraphrase flips be reduced without degrading observed accuracy? Yes, but the consistency is bought from the question text. 8.5% → 3.5% pairwise flips, patient-disjoint
  6. The adapter cut flips. Did it make the model safer?
  7. Safety Do consistency improvements produce safer, more image-dependent behavior? No. Tuning for consistency concentrates predictions in the Dangerous cell. 81% of consistent predictions are image-invariant
  8. If consistency cannot be trusted, what flags the predictions a clinician should check?
  9. Deployment Can uncertainty and audit signals identify predictions that should be escalated? Partly. Entropy ranks flips well, but the gate admits text-answerable cases. 0.823 AUROC from one forward pass

For patients

A reworded question can flip a finding from present to absent: the difference between a study flagged for review and one that is not.

For clinicians

A tool that answers the same question two ways cannot be trusted at the bedside, however high its accuracy.

For evaluation

Accuracy-only benchmarks are structurally blind to this failure: a model can score well and still flip.

04 · Where to start

Where to start

Three ways in, depending on how long you have.

Five-minute overview

The argument and the one result that carries it.

  1. 1Consistency says nothing about grounding
  2. 281% of consistent answers do not change without the image
  3. 3What the dissertation recommends

Fifteen-minute evidence tour

One result per thrust, measurement through deployment.

  1. 1How often answers flip
  2. 2Consistency without the image
  3. 3What the adapter actually buys
  4. 4Attention says nothing about grounding
  5. 5What the deployment audit lets through

Full technical path

Every thrust, every population, every source.

  1. 1Thesis map
  2. 2Evidence
  3. 3Five ways into the argument
  4. 4The causality experiments
  5. 5Papers and code
  6. 6Read the dissertation
05 · The dissertation

Read the dissertation

227 pages, eight chapters and appendices. The defense is expected in August 2026.

Read the dissertation (PDF, 227 pages) Executive summary (PDF, 4 pages) Questions and answers

This PDF is a draft circulated ahead of the August 2026 defense; it is not the final archival dissertation and its contents may change before deposit.

Cite


  

  

     Permanent link
    
  




  

    
    
    
    
  

  

    

    
Type to search. Press Escape to close.