PSF-Med: A Clinician-Audited Benchmark for Paraphrase Sensitivity in Medical Vision-Language Models
Published in MMFM-BIOMED Workshop, CVPR 2026, 2026
| Dataset | Code | arXiv |
PSF-Med is a benchmark of 92,856 final evaluation question-paraphrase pairs, built from 26,850 chest X-ray questions across three countries. Rewording a clinically equivalent question flips the diagnosis on 6.4% to 54.7% of binary pairs across six medical Vision-Language Models (VLMs). The release carries questions, paraphrases, and audit verdicts, not images.
Recommended citation: Sadanandan, B., Behzadan, V., Jayan, L., & Kurup, A. G. (2026). PSF-Med: A clinician-audited benchmark for paraphrase sensitivity in medical vision-language models. MMFM-BIOMED Workshop, CVPR 2026. arXiv:2602.21428.
Download Paper