一句话：系统复现 ICSE/ASE 2024 中 18 项商业 LLM 实证研究，发现无一能完全复现，并分析阻碍因素。
本文关注以 LLM 为中心的实证软件工程研究能否复现。作者分析 ICSE 2024 与 ASE 2024 的 85 篇 LLM 研究，其中 18 篇提供研究制品并使用 OpenAI 模型，故尝试复现这 18 项研究。结果仅 5 项制品足够完整且可执行；对这 5 项均未能完全复现，2 项部分可复现，3 项不可复现。研究进一步讨论阻碍复现的因素，并建议加强研究制品评审与更稳健的研究设计，以提高未来 LLM 实证出版物的可复现价值。
Large Language Models have gained remarkable interest in industry and academia. The increasing interest in LLMs in academia is also reflected in the number of publications on this topic over the last years. For instance, alone 78 of the around 425 publications at ICSE 2024 performed experiments with LLMs. Conducting empirical studies with LLMs remains challenging and raises questions on how to achieve reproducible results, for both researchers and practitioners. One important step towards excelling in empirical research on LLM and their application is to first understand to what extent current research results are eventually reproducible and what factors may impede reproducibility. This investigation is within the scope of our work. We contribute an analysis of the reproducibility of LLM-centric studies, provide insights into the factors impeding reproducibility, and discuss suggestions on how to improve the current state. In particular, we studied the 85 articles describing LLM-centric studies, published at ICSE 2024 and ASE 2024. Of the 85 articles, 18 provided research artefacts and used OpenAI models. We attempted to replicate those 18 studies. Of the 18 studies, only five were sufficiently complete and executable. For none of the five studies, we were able to fully reproduce the results. Two studies seemed to be partially reproducible, and three studies did not seem to be reproducible. Our results highlight not only the need for stricter research artefact evaluations but also for more robust study designs to ensure the reproducible value of future publications.
@inproceedings{abs-2510-25506,
title = {Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies},
author = {Florian Angermeir and Maximilian Amougou and Mark Kreitz and Andreas Bauer and Matthias Linhuber and Davide Fucci and Fabiola Moyon and Daniel Mendez and Gorschek Tony},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773207}
}