一句话：实证揭示LLM在故障定位中对输入方法顺序存在显著偏置，拆分小上下文可近乎消除差距。
本文研究大型语言模型在软件故障定位（FL）中受输入顺序与上下文规模影响的问题。作者在包含 Java 与 Python 项目的两个基准上，用 Kendall Tau 距离构造不同方法排序，包括真实故障方法在最前与最后的“perfect”和“worst”顺序，并测试反转代码顺序及上下文切分。结果显示偏置显著：Java Top-1 准确率从 57% 降至 20%，Python 从 38% 降至约 3%；拆分为更小上下文后，两基准性能差距从 22% 和 6% 缩小至 1%。重命名方法名排除数据泄漏后趋势不变；按 DepGraph 排名排序可达 48% Top-1，优于 CallGraphDFS 等简单排序。
Large Language Models (LLMs) show great promise in software engineering tasks like Fault Localization (FL) and Automatic Program Repair (APR). This study investigates the impact of input order and context size on LLM performance in FL, a crucial step for many downstream software engineering tasks. We test different orders for methods using Kendall Tau distances, including "perfect" (where ground truths come first) and "worst" (where ground truths come last), using two benchmarks that consist of both Java and Python projects. Our results indicate a significant bias in order; Top-1 FL accuracy in Java projects drops from 57% to 20%, while in Python projects, it decreases from 38% to approximately 3% when we reverse the code order. Breaking down inputs into smaller contexts helps reduce this bias, narrowing the performance gap in FL from 22% and 6% to just 1% on both benchmarks. We then investigated whether the bias in order was caused by data leakage by renaming the method names with more meaningful alternatives. Our findings indicated that the trend remained consistent, suggesting that the bias was not due to data leakage. We also look at ordering methods based on traditional FL techniques and metrics. Ordering using DepGraph’s ranking achieves 48% Top-1 accuracy, which is better than more straightforward ordering approaches like CallGraphDFS. These findings underscore the importance of how we structure inputs, manage contexts, and choose ordering methods to improve LLM performance in FL and other software engineering tasks.
@inproceedings{paperbot11708,
title = {Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization},
author = {Md Nakhla Rafi and Dong Jae Kim and Tse-Hsun (Peter) Chen and Shaowei Wang},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3764524}
}