FairVLM: Enhancing Fairness and Prompt Sensitivity in Vision Language Models for Medical Image Segmentation

Vision-language models (VLMs) have demonstrated substantial promise in medical image segmentation by utilizing radiology reports as prompts to segment regions of interest. However, VLM deployment in clinical settings is challenged by two intertwined issues: i) demographic bias, where performance varies across demographic groups, and ii) prompt sensitivity, where semantically similar prompts yield inconsistent outputs. These challenges are interconnected; demographic underrepresentation can worsen a model’s sensitivity to prompts, and prompt instability can more heavily affect certain demographic groups. In this study, we present FairVLM, a unified framework that addresses both demographic disparity and prompt sensitivity in VLMs. FairVLM integrates three key components: (1) Semantic-Retaining Counterfactual Prompting (SRCP), which generates clinically consistent and diverse prompt variations via large language models; (2) Demographic-Aware Feature Normalization (DAFN), a lightweight module that mitigates latent representation bias across demographic groups; and (3) a Fairness-Calibrated Loss (FCL) that explicitly penalizes performance disparities while encouraging prompt consistency. Extensive evaluations on the Harvard-FairSeg dataset show that FairVLM significantly improves equity-scaled segmentation. It also reduces demographic disparity (DI) by over 65\% and relative performance gap (RPG) by over 60\%, while maintaining or boosting overall accuracy. FairVLM is robust to prompt changes, with less than 0.5\% performance drop across varied prompts, and also generalizes well on unseen datasets. These findings present FairVLM as a new state-of-the-art, robust, and adaptable framework for a fair, prompt-invariant vision-language model. Code and data are available at https://github.com/.../FairVLM.