One-shot Portrait Stylizaiton via Geometric Alignment

Portrait stylization aims to cast vivid artistic style drawn from style examples to portrait photos. This task has recently been extensively studied with machine learning algorithms, but it is still difficult for existing methods to stylize portraits from a single style reference, severely limiting these methods for real-world applications. In this paper, we propose a portrait stylization method that learns style reference from a single artistic portrait image. Unlike previous StyleGAN based methods that heavily rely on the quality of GAN inversion or diffusion based methods that introduce computational expensive operations and fall short of precise control, our method achieves high-quality stylization with small computation and parameter budget. Specifically, we employ geometric alignment to build spatial correlation between content images and style reference. A content LoRA and a style LoRA are then jointly optimized based on a pre-trained diffusion backbone respectively, with orthogonal adaptation used to disentangle the content and style information. During inference, the style LoRA is integrated into the diffusion backbone and ControlNet is further combined to facilitate better spatial and identity control. We illustrate abundant stylized portraits with multiple styles. Qualitative comparison, quantitative validation and user study all prove that our method outperforms existing methods, and ablation study demonstrates the effectiveness of each components. Code and pre-trained model will be made publicly available upon paper acceptance.