HEART-PFL: Stable Personalized Federated Learning under Heterogeneity with Hierarchical Directional Alignment and Adversarial Knowledge Transfer

Personalized Federated Learning (PFL) aims to deliver effective client-specific models under heterogeneous data distributions while preserving privacy. Despite recent progress, existing approaches face two key limitations: (i) shallow, one-way prototype alignment that underutilizes hierarchical semantics and risks suppressing client-specific cues, and (ii) brittle server-side knowledge transfer that propagates teacher bias and destabilizes global updates.We propose HEART-PFL, a dual-sided framework that addresses these challenges through Hierarchical Directional Alignment (HDA) and Adversarial Knowledge Transfer (AKT). On the client side, HDA performs depth-aware alignment by enforcing cosine similarity in early layers for directional consistency and mean-squared matching in deeper layers for semantic precision, thereby leveraging hierarchical features without erasing personalization. On the server side, AKT strengthens ensemble knowledge transfer with bidirectional, symmetric-KL distillation on both clean and adversarial proxy samples, mitigating personalization bias and enhancing the stability of global model updates.Implemented with lightweight adapters requiring only 1.46M trainable parameters, HEART-PFL achieves state-of-the-art personalized accuracy on CIFAR-100, Flowers-102, and Caltech-101 (63.42\%, 84.23\%, and 95.67\%, respectively) under Dirichlet non-IID partitions, while remaining robust to out-of-domain proxy data. Ablations corroborate that HDA yields hierarchy-aware gains, AKT improves robustness and server-side stability, and their combination delivers the strongest—and most stable—personalization.