IPTQ-ViT: Post-Training Quantization of Non-linear Functions for Integer-only Vision Transformers

Previous Quantization-Aware Training (QAT) methods for Vision Transformers rely on expensive retraining to recover accuracy loss in non-linear layer quantization, limiting their use in resource-constrained environments. In contrast, existing Post-Training Quantization (PTQ) methods either partially quantize non-linear functions or adjust activation distributions to maintain accuracy but fail to achieve fully integer-only inference. In this paper, we introduce IPTQ-ViT, a PTQ framework for fully integer-only Vision Transformers without retraining. We present novel approximation functions: a polynomial-based GELU optimized for vision data and a bit-shifting-based Softmax designed to improve approximation accuracy in PTQ. In addition, we propose a unified metric integrating quantization sensitivity, perturbation, and computational cost to select the optimal approximation function per activation layer. IPTQ-ViT outperforms previous PTQ methods, achieving up to 6.44%p (avg. 1.78%p) top-1 accuracy improvement for image classification, 1.0 mAP for object detection. IPTQ-ViT is the first fully integer-only PTQ method for Vision Transformers, surpassing partially integer-based PTQ methods in both W8A8 and W4A8 quantization and achieving comparable accuracy to QAT methods.