Thrust 5 · Chapter 7

Deployment

Can uncertainty and audit signals identify predictions that should be escalated?

One forward pass is enough to rank flip risk. Predictive entropy predicts paraphrase flips at an AUROC of 0.823 and errors at 0.862 for Targeted LoRA on the PadChest flip bank, and it carries across architectures: LLaVA-Rad LoRA reaches 0.830 on PadChest and 0.905 on MIMIC. The gate is a different story. It is an offline readiness audit rather than something a clinic could run, since its region term needs a radiologist's bounding box that no undiagnosed patient has. And what it admits is mostly what the question text already answers: on the slice where text and image disagree, Targeted LoRA scores 2.9%.

How I ran it

Compare uncertainty methods (softmax entropy, Monte Carlo dropout, deep ensemble, temperature scaling) for flip and error prediction, calibration, corruption stress, and descriptive conformal coverage, then evaluate a multi-signal admit rule combining paraphrase agreement, swap sensitivity, and region evidence.

Datasets: MIMIC-CXR, PadChest
Models: MedGemma-4B, Targeted LoRA, Full LoRA, LLaVA-Rad, LLaVA-Rad LoRA, Qwen2-VL
Thesis tables: tab:bridge_auroc, tab:gate_results, tab:conformal_shift, tab:clean_calibration
Papers: Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models, Consistency Is Not Safety

The limitation that matters most

The gate is an offline readiness audit: its region term requires a radiologist bounding box that does not exist for an undiagnosed patient (available for 637 of 861 PadChest questions and none of MIMIC), and what it admits are text-answerable cases, not grounded ones, so the answer to whether gates select grounded predictions is no.

Confirmed Yes: single-pass predictive entropy predicts paraphrase flips (AUROC 0.823), not only errors (AUROC 0.862), on PadChest, and the flip bridge replicates across architectures (LLaVA-Rad LoRA 0.830 PadChest, 0.905 MIMIC).

Negative result No. Gates admit the cases the text already answers, not the ones grounded in the image. Admitted-case accuracy does rise, to 96.8% against 91.5% for Targeted LoRA on PadChest. Yet that same model scores 2.9% on the slice where text and image disagree, and Qwen2-VL gets 0 of 279 such questions right under every rule tried.

Negative result No: no single internal monitor transfers across model families. Gemma, LLaVA-Rad, and Qwen2-VL each require different monitors, and the multi-pass alternatives fail in family-specific ways: adapter-only Monte Carlo dropout is an uninformative epistemic probe and the MedGemma deep ensemble collapses out of distribution.

Try this thrust yourself

Both signals in this chapter are interactive. Drag the entropy threshold and watch coverage trade against error, then switch the audit rule's conditions off and on and see what each one admits. Both run on the same 861 questions this chapter reports.

The entropy demo The audit demo

Primary results

0.823

question-level

Single-pass entropy predicts paraphrase flips (AUROC 0.823)

Softmax predictive entropy from one forward pass predicts whether a prediction will flip under paraphrasing with AUROC 0.823 for Targeted LoRA on PadChest: uncertainty is a deployable single-pass proxy for paraphrase fragility.

Targeted LoRA · PadChest · n = 861 · 95% CI [0.778, 0.864]; p = 4.3e-29

Source, denominator, and limits

How far this goesSoftmax entropy, temperature-scaled entropy and absolute margin are rank-equivalent for a binary softmax, so they are one signal reported three ways, not three independent methods; only the deep ensemble is genuinely distinct, and it fails (0.552). Operating thresholds are model-specific and do not transfer.

Metric: flip-prediction AUROC from softmax predictive entropy
Denominator: question-level (per question)
Sample: n = 861
Population: PadChest flip bank (861 questions)
Model: Targeted LoRA
Dataset: PadChest
Split: ood
Comparison: operator-preserving slice 0.826 (95% CI [0.780, 0.870]); deep ensemble 0.552; cross-architecture LLaVA-Rad LoRA 0.830 (PadChest) / 0.905 (MIMIC)
Uncertainty: 95% CI [0.778, 0.864]; p = 4.3e-29
Thesis: Chapter 7, tab:bridge_auroc
Paper: Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models
Source artifact: dissertation/tables/thrust4/table_bridge_auroc.tex
Last verified: 2026-07-15

Open in Evidence Explorer

33.0% admitted at 96.8% accuracy

question-level

The readiness audit admits 33.0% of Targeted LoRA PadChest predictions at 96.8% accuracy

The deployment-readiness rule (paraphrase agreement AND image reliance) admits 33.0% of Targeted LoRA's PadChest predictions, and the admitted slice is 96.8% accurate versus 91.5% unfiltered, with a 4.2% residual flip rate on uninspected paraphrases.

Targeted LoRA · PadChest · n = 861

Source, denominator, and limits

How far this goesThis is an OFFLINE readiness audit, not an online gate: the roi_matters term requires a radiologist bounding box for the finding, which a deployed system does not have for an undiagnosed patient (boxes exist for 637 of 861 PadChest records and none of MIMIC). Dropping the region term - the only online-deployable variant - cuts admission from 33.0% to 26.9%.

Metric: admission rate of the offline readiness audit
Denominator: question-level (per question)
Sample: n = 861
Population: PadChest flip bank (861 questions)
Model: Targeted LoRA
Dataset: PadChest
Split: ood
Comparison: accuracy 96.8% gated vs 91.5% on all; Base admits 41.6% (96.9% gated), Full LoRA 26.7% (83.9% gated)
Uncertainty: not reported for this value
Thesis: Chapter 7, tab:gate_results
Paper: Consistency Is Not Safety: Deployment Audits Reveal Text-Driven Selection in Medical Vision-Language Models
Source artifact: dissertation/tables/thrust4/table_gate_results.tex
Last verified: 2026-07-15

Open in Evidence Explorer

2.9%

question-level

2.9% accuracy on the slice where the image should matter

Targeted LoRA scores only 2.9% accuracy on the PadChest text-disagrees slice - the cases where the image-conditioned answer must depart from the text-only answer - so gates raise admitted-case accuracy by selecting text-answerable cases rather than image-grounded ones (RQ5.3 = no).

Targeted LoRA · PadChest · n = 241

Source, denominator, and limits

How far this goesA negative result, and it should be read as one: apparent safety gains from gating come from routing around the image-relevant portion of the workload, not from solving it. Severity is dataset-dependent (the same slice on MIMIC-CXR sits at 68-86% accuracy), so the lesson is that high-answerability environments amplify the text shortcut, not that all medical VLMs always fail when the image matters.

Metric: accuracy on the text-disagrees slice
Denominator: question-level (per question)
Sample: n = 241
Population: PadChest flip bank (861 questions)
Model: Targeted LoRA
Dataset: PadChest
Split: ood
Comparison: the strict gating rule raises this only to 25.0% at 2% intra-slice coverage; Qwen2-VL gets 0 of 279 text-disagrees questions correct under every evaluated rule
Uncertainty: not reported for this value
Thesis: Chapter 7
Paper: Consistency Is Not Safety: Deployment Audits Reveal Text-Driven Selection in Medical Vision-Language Models
Source artifact: dissertation/chapters/06_thrust5.tex
Last verified: 2026-07-15

Open in Evidence Explorer

0.0001

question-level

Adapter-only Monte Carlo dropout detects almost no epistemic uncertainty (MI = 0.0001 nats)

Monte Carlo dropout on the LoRA adapters yields mean mutual information of 0.0001 nats for Targeted LoRA on PadChest: the perturbation barely moves the model, so the probe detects almost none of the predictive entropy as epistemic.

Targeted LoRA · PadChest · n = 861

Source, denominator, and limits

How far this goesNear-zero mutual information does NOT establish that the remaining uncertainty is aleatoric: a probe that perturbs 0.1% of the weights (4.38M adapter parameters of roughly 4.4B) cannot see epistemic uncertainty carried by the frozen 99.9%. The defensible conclusion is that adapter-only dropout is an uninformative epistemic probe for this model, which is also why it adds nothing over a single softmax pass.

Metric: mean mutual information (epistemic component) under Monte Carlo dropout, p = 0.05 on adapters
Denominator: question-level (per question)
Sample: n = 861
Population: PadChest flip bank (861 questions)
Model: Targeted LoRA
Dataset: PadChest
Split: ood
Comparison: deep ensemble mean MI 0.057 nats, but that signal is corrupted by member OOD failure; Monte Carlo dropout flip AUROC equals plain softmax entropy (0.823 vs 0.823)
Uncertainty: not reported for this value
Thesis: Chapter 7
Paper: Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models
Source artifact: dissertation/chapters/06_thrust5.tex
Last verified: 2026-07-15

Open in Evidence Explorer

52.0% worst member vs 92.1% best

question-level

Deep ensemble fails out-of-distribution: worst member at 52.0% accuracy

The five-seed LoRA deep ensemble fails on PadChest: per-member accuracy spans 52.0% (seed 456) to 92.1% (seed 42), and the ensemble's selective prediction collapses to AURC 0.130 versus 0.019 for single-pass entropy, admitting only 23.8% of cases at 5% risk versus 88.5%.

Targeted LoRA · PadChest · n = 861 · member accuracies 52.0 / 67.1 / 68.5 / 78.7 / 92.1% across the 5 seeds

Source, denominator, and limits

How far this goesTrain-validation accuracy of 86.9-88.9% gave no early warning of which seeds would fail, so the failure is undetectable before deployment shift. The ensemble works in-distribution on MIMIC (best calibration and accuracy there), and the LLaVA-Rad ensemble does not collapse: the failure is specific to adapters trained on the same narrow 500-pair MIMIC distribution, not to ensembling in general.

Metric: worst ensemble-member accuracy on PadChest
Denominator: question-level (per question)
Sample: n = 861
Population: PadChest flip bank (861 questions)
Model: Targeted LoRA
Dataset: PadChest
Split: ood
Comparison: best member (seed 42) 92.1%; ensemble AURC 0.130 vs 0.019 single-pass entropy; probability averaging recovers 72.6% aggregate accuracy but not a usable confidence ranking
Uncertainty: member accuracies 52.0 / 67.1 / 68.5 / 78.7 / 92.1% across the 5 seeds
Thesis: Chapter 7
Paper: Predictive Entropy as a Joint Screen for Error and Paraphrase Instability in Medical Vision-Language Models
Source artifact: results/uai/revision/ensemble_diagnostics.json
Last verified: 2026-07-15

Open in Evidence Explorer

Charts from this thrust

Open any of these in the evidence explorer to filter by model, dataset, or metric.

Predictive entropy of stable versus flipped predictions

Targeted LoRA on the 861-question PadChest flip bank: 747 stable questions and 114 that flipped under paraphrase.

Questions whose answer flips under paraphrase sit at visibly higher single-pass predictive entropy than stable ones, which is what lets one forward pass rank flip risk (AUROC 0.823).

Predicting a paraphrase flip from uncertainty on the original question

Targeted LoRA on the 861-question PadChest flip bank (temperature scaling on the 732-question subset that survives its 15% calibration holdout). AUROC for predicting whether the question will flip.

Cheap predictive entropy on the original question predicts whether that question will flip under paraphrasing at AUROC 0.823, so uncertainty and paraphrase sensitivity are measuring related failure. The expensive deep ensemble is the worst predictor at 0.552.

Risk-coverage: what accepting fewer predictions buys you

PadChest flip bank, n=861 questions per model. Selective risk (error rate among accepted) against coverage, ranking by softmax confidence, for base MedGemma-4B, Targeted LoRA and Full LoRA. Curves recomputed from the post-image-fix run and downsampled to 50 points each.

Targeted LoRA dominates: its risk stays near zero out to high coverage (AUGRC 0.014 against 0.047 for the base model and 0.153 for Full LoRA), so confidence-based deferral works for it and barely works for Full LoRA, whose curve rises almost immediately.

Offline readiness audit: admission rate against accuracy

Six cells: base MedGemma-4B, Targeted LoRA and Full LoRA on the MIMIC-CXR flip bank (n=98) and the PadChest flip bank (n=861). The audit admits a prediction only if the first two paraphrases agree with the original and the answer shows image reliance.

Filtering on agreement plus image reliance lifts accuracy in five of six cells (base on PadChest: 78.6% to 96.9%), but admits only 14-42% of questions. The price of a trustworthy answer is refusing most of them.

Does failure detection survive image corruption?

AUGRC against corruption severity (0 = clean, then 1/3/5 averaged over five corruption types) on the 861-question PadChest flip bank, softmax entropy, for base MedGemma-4B, Targeted LoRA and Full LoRA. Bars are bootstrap 95% intervals. Lower is better.

Targeted LoRA's undetected-failure risk stays low and nearly flat under corruption (0.014 clean to 0.027 at severity 5) while the base model nearly doubles (0.047 to 0.088). Full LoRA is the worst at every severity (0.153 to 0.186): it is flat only because it starts badly.