How a medical vision-language model works, end to end
MedGemma-4B is too large to take apart on a bench, so we built a smaller one with the same anatomy. FlipLens pairs the same frozen MedSigLIP-448 image encoder with a 14.1M-parameter Gemma-3 decoder, small enough that every token and gradient fits in one view. Same machine, bench scale. This page walks it: how it is built, trained, tuned, and run.
429M parameters, frozen encoder
14.1M parameters, trained decoder
256 image tokens per film
141 words in the vocabulary
74.8% accuracy
FlipLens is a research probe for mechanism. It is not a medical question-answering system, and it is not for clinical use.
Act 1 · The anatomy
One film, one question, one sequence
A chest radiograph goes in as pixels. The frozen encoder cuts it into 4,096 patches, a 64 by 64 grid, reads them all, and pools them to 256 image tokens. Those tokens are prepended to the tokenized question, so the decoder reads one sequence: image first, then words. The decoder is a small Gemma-3: root-mean-square normalization (RMSNorm), rotary position embeddings (RoPE), and grouped-query attention (GQA) with six query heads over two key-value heads, under a causal mask. The snowflake marks what training never touches.
Act 2 · Training
Predict the next token, over and over
Training adjusts only the decoder; the encoder stays frozen, so the image features never change across paraphrases of a question. Each example pairs a radiograph with a presence question and its answer, and the objective is the usual one: at every position, predict the next token. Whatever the model learns about wording, it learns in the decoder.
istherecardiomegaly?→yeshigher scoreno
Schematic. The decoder scores the vocabulary at the answer position; the higher of yes and no is the answer. Training moves those scores toward the label.
The probe's point is that the training-phrasing distribution is a knob. Train the same model three ways on 86,288 questions from two hospitals, everything else fixed, and the flip rate moves:
Canonicalone fixed phrasing67.1%
Augmentedevery paraphrase4.8%
Adversarialregister tied to the answer88.4%
Flip rates over 8 seeds per regime. Every split is balanced per finding, so a text-only model scores exactly 0.500 and no regime can exploit an answer prior. Augmented separates from both narrow regimes at the maximum effect size (Mann-Whitney p = 1.6e-4, Cliff's delta 1.00), and the two narrow regimes separate from each other too (p = 3.1e-4, delta 0.97), so tying a phrasing register to the answer is a distinct and larger harm than narrow coverage alone. Accuracy follows the same ordering (75.3%, 66.8%, 58.1%), so the shortcut costs diagnostic performance and not only consistency. Scored only on the 24 phrasings withheld from training, augmented rises to 26.6% against 65.9% and 87.4%, which is the honest size of the lever.
Act 3 · Finetuning
Most weights never move
The probe is itself a finetuning story: of its roughly 443M parameters, only the 14.1M in the decoder ever train. The deployed model takes the same idea further with a low-rank adapter (LoRA): every original weight stays frozen and a small trainable correction is added at layers 15 to 19, 0.1% of parameters. On the clean patient-disjoint split that cuts pairwise flips from 8.5% to 3.5%, about 59%.
About 443M parameters in all. The three warm cells are the 14.1M that train.A low-rank update: the frozen weight matrix W plus the product of two small trainable matrices A and B gives the correction. A and B together hold far fewer numbers than W, so the update costs little and the base model stays untouched. The adapter result
Act 4 · Inference
One forward pass, read at one position
Answering is a single forward pass. The film is encoded once, the question is tokenized, the two are fused, and the decoder reads the sequence once. Everything is decided at the answer position, where the decoder puts a score on each vocabulary token.
1
Tokenize
The question becomes ids from a 141-piece vocabulary built from chest X-ray presence questions.
2
Encode once
The frozen encoder turns the film into 256 image tokens, computed once per film and reused.
3
Fuse
Image tokens go first, question tokens after. One sequence, read left to right.
4
Decode
A short stack of blocks: grouped-query attention with 6 query and 2 key-value heads and rotary positions, a feed-forward stage, RMSNorm throughout, and a causal mask that hides the future.
5
Read
At the answer position the decoder scores the vocabulary. Whichever of yes and no scores higher is the answer.
Stage 1 of 5: Tokenize
Because the pass is a stack of layers, you can ask what each layer prefers. A Jacobian lens reads the yes-or-no margin each layer is disposed to produce; for two phrasings of one question the readouts track together early, then commit to opposite answers. In the probe, flipping pairs diverge about 2.2 times more than stable ones at layer 0 (point-biserial correlation with the flip 0.26), and a rank-1 patch at layers 0 to 4 restores the flipped answer with net recovery near 1.0 while norm-matched controls do nothing (disruption 0.000). The deployed model shows the same shape at layer 16. Read every layer, word by word · The experiments that localize the cause
What the small model licenses
FlipLens is a controlled probe. It is not a medical question-answering system. It grounds weakly, its 141-piece vocabulary covers presence questions only, and it is not for clinical use. What it adds is the one experiment a frozen deployed model cannot run: vary the training-phrasing distribution and read the effect. The claim it supports is sufficiency plus localization. It does not prove that MedGemma-4B has this exact origin, and the probe's early-layer locus does not map onto the deployed model's layer 16. It also carries a measurement lesson worth stating next to the 74.8% above: on a balanced set a blind model, a shuffled-grounding model, and a genuinely seeing one all score about 0.50 accuracy while their areas under the curve separate (0.500, 0.502, 0.604), so grounding must be read from the margin, not from a binary answer.