Federated Distillation with Structural Perturbation for Robust Fine-Tuning of LLMs

Abstract

This paper proposes a federated fine-tuning framework that integrates differentiated distillation guidance and modular structure injection to address key challenges in distributed fine-tuning of large language models. These challenges include unstable semantic transfer, sensitivity to structural perturbations, and low communication efficiency. Without transmitting raw data, the framework introduces a differentiated distillation mechanism to guide local client models in aligning with the global semantic structure. This reduces representation drift under non-independent and identically distributed multi-task settings. Meanwhile, a modular structure injection mechanism applies structural perturbations to key components such as attention layers and feedforward networks. This guides the model to learn robust representations under local variation, enhancing the consistency and stability of cross-task representations. The two mechanisms are designed to be decoupled yet jointly optimized. They can be flexibly embedded into mainstream pre-trained language models and enable communication-efficient distributed knowledge optimization under the federated learning framework. Experiments on multiple task-incremental subsets verify the effectiveness of the proposed method. Through comprehensive main experiments, ablation studies, and hyperparameter sensitivity analyses, the model is evaluated across multiple dimensions, including semantic retention, structural stability, generalization ability, and parameter efficiency. The results show that the proposed method outperforms existing representative approaches and demonstrates strong practical value and adaptability.