一句话：提出 Attention Pruning，用代理模拟退火搜索注意力头剪枝参数，自动降低语言模型的性别偏见并兼顾效用。
大语言模型在敏感社会场景中会编码并延续训练数据中的社会偏见，而修改训练数据往往不可行，因此需要后处理公平性修复。本文研究通过剪枝注意力头进行偏见缓解，但寻找最优剪枝参数是组合优化问题，需在公平性与效用间权衡。作者提出 Attention Pruning，一种公平感知的代理模拟退火方法：用代理深度神经网络建模注意力头状态与公平性/效用指标之间的关系，从而避免直接搜索巨大的LLM参数空间。实验表明，该方法在性别偏见上最高降低40%，优于当前最佳方法，同时兼顾模型效用。
This paper explores pruning attention heads as a post-processing bias mitigation method for large language models (LLMs). As LLMs expand into sensitive social contexts, fairness concerns become crucial since these models encode and perpetuate societal biases from their training data. While modifying training datasets is prohibitive, post-processing techniques like pruning attention heads can provide feasible approaches to improve fairness. However, identifying optimal parameters to prune presents a combinatorial challenge requiring efficient solutions that balance model fairness and utility. We introduce Attention Pruning, a fairness-aware surrogate simulated annealing approach that uses surrogate deep neural networks to efficiently model the relationship between attention head states and fairness/utility metrics, enabling optimization without directly searching the massive LLM parameter space. Our experimental evaluation shows that Attention Pruning achieves up to \(40\%\) reduction in gender bias, outperforming the state-of-the-art. Warning: This paper contains content that some readers may find offensive and harmful.
@inproceedings{abs-2503-15815,
title = {Attention Pruning: Automated Fairness Repair of Language Models via Surrogate Simulated Annealing},
author = {Vishnu Asutosh Dasu and Md Rafi Ur Rashid and Vipul Gupta and Saeid Tizpaz-Niari and Gang Tan},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773115}
}