Gradient-Free Classifier Guidance for Diffusion Model Sampling

Unguided sampling in diffusion models is known to generate images of wide variations, albeit with a trade-off in image fidelity. Guided sampling methods, such as classifier guidance (CG) and classifier-free guidance (CFG), focus on sampling in well-learned high-probability regions to generate images of high fidelity, but each has its limitations. CG is computationally expensive due to the costly classifier gradient descent process, while CFG, being gradient-free, is more efficient but compromises class label alignment compared to CG. In this work, we introduce Gradient-Free Classifier Guidance (GFCG), a novel method utilizing a pre-trained classifier solely in inference mode, entirely avoiding gradient calculations. GFCG introduces two innovative mechanisms: (1) adaptive reference class selection, dynamically determining an appropriate undesired reference class; and (2) adaptive guidance scaling, dynamically adjusting the guidance strength based on classifier confidence during sampling. Experiments on both class-conditioned and text-to-image generation demonstrate that the proposed GFCG method consistently improves class prediction accuracy while preserving diversity. We also show that GFCG is complementary to other guided sampling methods. When combined with Autoguidance (ATG), GFCG attains record performance on ImageNet 512 $\times$ 512, with a $FD_{DINOv2}$ of 23.39, while maintaining a high Precision of 94.0% compared to ATG's 89.9%.