Intra-Class Probabilistic Embeddings for Uncertainty Estimation in Vision-Language Models

Vision-language models (VLMs), such as CLIP, have gained popularity for their strong open vocabulary classification performance, but they are prone to assigning high confidence scores to misclassifications, limiting their reliability in safety-critical applications. We introduce a training-free, post-hoc uncertainty estimation method for contrastive VLMs that can be used to detect erroneous predictions. Our approach assesses visual feature consistency within a class, using feature projection combined with multivariate Gaussians to create class-specific probabilistic embeddings. At inference, test embeddings are scored using the log-probability under each class distribution, with a softmax-normalized density used as the new confidence score. Our method is VLM-agnostic, requires no fine-tuning, robust to label shift, and works effectively with as few as 300 training images per class. Extensive experiments on ImageNet, Flowers102, and Food101 show state-of-the-art error detection performance, significantly outperforming both deterministic and probabilistic VLM baselines. Our code will be made publicly available upon acceptance.