Reproducibility
What it cost, and what you need to run it again
This work took about ~4,800 GPU-hours on NVIDIA A100 80GB hardware. Most of that went on evaluation rather than training, and the headline mitigation is one of the cheapest things in the dissertation to reproduce: about twenty minutes on a single GPU. This page itemizes the cost, pins the software and seeds, and says plainly which artifacts are public and which need a request.
Figures on this page are transcribed from the dissertation's appendices (dissertation/chapters/08_appendix.tex, Appendix B (Computational Resources, Reproducibility) and Appendix C (Software Environment, Hardware, Random Seeds)).
Where the compute went
Total across all thrusts on a shared cluster, including failed and exploratory runs, not only the runs that appear in the dissertation. Every run was tracked in Weights & Biases.
6 models, 3 datasets, roughly 93k paraphrase pairs; rubric-based judge audit of 122,778 pairs; regeneration of the PadChest paraphrase set
5 models, 2 datasets; softmax entropy, Monte Carlo dropout, temperature scaling, deep ensembles across 5 seeds, and text-only baselines
5 corruption types, 3 severities, across models, datasets, and uncertainty methods
Multiple model families; image-swap, text-only, and null-image controls
Attention extraction, patch occlusion, region-of-interest ablation, and laterality across models
Includes the five-seed replication study
Sparse-autoencoder screening and transfer, activation and residual-transplant patching, canonical validation, and lens fitting
Total: ~4,800 GPU-hours (~200 GPU-days). The line items above sum to 4,800 hours; the reported total is rounded and includes runs that produced nothing publishable.
What it costs to reproduce one result
The aggregate above is the cost of the whole programme, including exploration. Reproducing a single headline result is far cheaper, and that asymmetry is deliberate: the interventions in this dissertation are small.
| Task | Cost | Detail |
|---|---|---|
| Reproduce the targeted-LoRA mitigation | ~20 minutes on one A100 | ~24 GB peak memory, 4.38M trainable parameters (0.10%). This is the headline result and it is cheap to reproduce. |
| Reproduce the full-LoRA comparison | ~45 minutes on one A100 | 30.5M trainable parameters (0.72%) |
| Run sparse-autoencoder extraction | single A100, ~4 GB | SAE inference only; the Gemma Scope weights are downloaded, not trained |
| Score the deployment gate on one prediction | 5 to 6 forward passes, ~600-900 ms | K = 3 paraphrases plus the image-reliance checks, on an A100 |
Environment, hardware, and seeds
Software
- Python
- 3.10.12
- PyTorch
- 2.6.0 with CUDA 12.4
- Transformers
- 4.57.3
- PEFT
- 0.7.0
- scikit-learn
- 1.3.2
- scipy
- 1.11.4
- Sparse autoencoders
- custom implementation over Gemma Scope weights
Hardware
- Training: single NVIDIA A100 80GB, 64 GB system RAM
- Inference: NVIDIA A100 80GB or NVIDIA A6000 48GB
- Multi-GPU: 8x A100 for the corruption sweep
- Sparse-autoencoder extraction: single A100, about 4 GB
Seeds
Primary seed 42; the replication study uses 42, 123, 456, 789, 2024.
The primary seed is 42 for all main results; the replication study uses all five. Model weights are loaded from Hugging Face checkpoints with commit hashes recorded in the experiment configurations.
What is already public
These need no request. The underlying chest X-ray images are never redistributed here: they come from their original providers under their own agreements.
| Artifact | What it is | Link |
|---|---|---|
| PSF-Med benchmark code | Benchmark construction and evaluation code for PSF-Med: paraphrase generation, equivalence auditing, and flip-rate evaluation (Thrust 1). | Open |
| PSF-Med dataset (Hugging Face) | The released PSF-Med paraphrase benchmark data on Hugging Face; image files are not redistributed and must be obtained from the original credentialed sources. | Open |
| Mechanistically-guided LoRA weights (Hugging Face collection) | Targeted and full LoRA adapter checkpoints for MedGemma-4B from the CHIL 2026 paper, trained at n=12K. | Open |
| Targeted MedGemma-4B LoRA model | The n=12K targeted LoRA adapter for MedGemma-4B used in the paraphrase-consistency mitigation experiments. | Open |
| Paraphrase-consistency mitigation code | Training and evaluation code for the targeted LoRA mitigation and its safety re-audit (Thrust 3). | Open |
| VSF-Med project site | Project page for VSF-Med, the vulnerability scoring framework companion work on medical VLM safety. | Open |
| VSF-Med code | Implementation of the VSF-Med vulnerability scoring framework and its evaluation pipeline. | Open |
| MedMCQA robustness study code | Code for the companion study of prompt sensitivity and chain-of-thought prompting on medical question answering. | Open |
| Predictive entropy study code | Code and reproducibility materials for the UNSURE 2026 predictive-entropy screen for errors and paraphrase instability. | Open |
| Attention Without Grounding code | Code and data instructions for the iMIMIC 2026 causal evaluation of visual explanations in medical VLMs. | Open |
Requesting the rest
Public artifacts need no request; the link is on this page. A request is only needed for the parts that are not yet public, and it exists so that access is recorded and so that the code can be sent with the context needed to run it.
| Artifact | What it is | Access |
|---|---|---|
| Weights & Biases experiment tracking | Experiment tracking workspace for the uncertainty-quantification and robustness runs. | On request |
| medical-vlm-robustness research repository | The full research monorepo behind the dissertation (pipelines, analyses, dissertation sources); private during the pre-defense period. | private |
Send a request
The button opens a mail draft addressed to the author with the questions below already in it. Nothing is submitted from this page and nothing is logged here: you will see the message before it is sent.
- Your name, role, and institution
- Which artifact you want (benchmark code, LoRA training, safety pipeline, mechanistic analyses, or a specific figure or table)
- What you intend to do with it
- Whether you already hold the credentials for the underlying image datasets (PhysioNet for MIMIC-CXR and VinDr-CXR, BIMCV for PadChest)
Open a request draft All papers and resources
Requests go to contact@bineshkumar.me. Access to the chest X-ray images themselves cannot be granted by the author: MIMIC-CXR and VinDr-CXR are credentialed through PhysioNet, and PadChest through BIMCV.