paperbot · PL 论文追踪

全部论文 待读 收藏

P2 How (not) to Analyze Software Engineering Experiments: From Anti-Patterns to Solutions

如何(不)分析软件工程实验:从反模式到解决方案
ICSE Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering2026匹配度 1/10 · 该论文讨论软件工程实验数据分析中的统计反模式与实践指南,与研究画像关注的 PL 与 AI 交叉、静态分析、浮点程序分析、贝叶斯程序分析或 LLM/Agent 工具改进等方向基本无关。引用 0 inbox
Sira Vegas
crossref: 2026-09-12

AI 速览

一句话:本教程总结软件工程实验数据分析中的常见反模式与陷阱,并基于25年经验给出提升结果可靠性的实践指南。

软件工程中的实验研究虽已广泛开展,但其实验设计与数据分析的质量始终是影响结果可信度的关键难题。作者指出,统计方法的误用问题长期存在,根源在于软件工程实证研究本身的复杂性,以及该领域特有的一些性质使部分实验问题与其他学科的理解方式不同。本教程聚焦实验数据分析环节,并非系统讲授数据分析方法,而是回顾已发表软件工程实验中识别出的典型问题、反模式和错误做法,并给出相应的解决方案与建议。内容基于作者超过25年运行实验的经验,旨在帮助研究者避免常见陷阱,提高实验结果的可靠性与可重复性。

问题
软件工程实验在统计方法使用和数据分析上长期存在误用与反模式,导致实验结果的可信度受到质疑。
方法
以教程形式回顾已发表软件工程实验中暴露出的关键分析问题,归纳常见反模式并给出基于经验的解决方案与指导。
评估
基于作者超过25年运行软件工程实验的经验,结合已发表实验中的问题案例进行总结与经验性指导,而非开展新的对照实验。
原文摘要(Abstract)

Experimentation is a key aspect of science and engineering, yet it remains one of the major stumbling blocks in software engineering. Although many experiments are conducted today, ensuring their quality—whether they involve human subjects or not—remains a persistent concern for the trustworthiness of results. Researchers have raised concerns about the correct use of statistical methods for many years, these issues often persist due to two main factors: the inherent complexity of empirical studies in our field, and the unique characteristics of software engineering, which lead to some experimentation issues being conceived differently than in other disciplines. This tutorial focuses on the analysis of experimental data, helping participants avoid common pitfalls and anti-patterns while improving the quality and reliability of their results. It is not intended as a data analysis course, but rather reviews key issues identified in published software engineering experiments, providing guidance based on over 25 years of experience running experiments.

链接与引用

DOI 原文 · PDF(开放获取) ·

BibTeX
@inproceedings{paperbot11712,
  title = {How (not) to Analyze Software Engineering Experiments: From Anti-Patterns to Solutions},
  author = {Sira Vegas},
  booktitle = {ICSE},
  year = {2026},
  doi = {10.1145/3774748.3776598}
}