一句话:本教程总结软件工程实验数据分析中的常见反模式与陷阱,并基于25年经验给出提升结果可靠性的实践指南。
软件工程中的实验研究虽已广泛开展,但其实验设计与数据分析的质量始终是影响结果可信度的关键难题。作者指出,统计方法的误用问题长期存在,根源在于软件工程实证研究本身的复杂性,以及该领域特有的一些性质使部分实验问题与其他学科的理解方式不同。本教程聚焦实验数据分析环节,并非系统讲授数据分析方法,而是回顾已发表软件工程实验中识别出的典型问题、反模式和错误做法,并给出相应的解决方案与建议。内容基于作者超过25年运行实验的经验,旨在帮助研究者避免常见陷阱,提高实验结果的可靠性与可重复性。
Experimentation is a key aspect of science and engineering, yet it remains one of the major stumbling blocks in software engineering. Although many experiments are conducted today, ensuring their quality—whether they involve human subjects or not—remains a persistent concern for the trustworthiness of results. Researchers have raised concerns about the correct use of statistical methods for many years, these issues often persist due to two main factors: the inherent complexity of empirical studies in our field, and the unique characteristics of software engineering, which lead to some experimentation issues being conceived differently than in other disciplines. This tutorial focuses on the analysis of experimental data, helping participants avoid common pitfalls and anti-patterns while improving the quality and reliability of their results. It is not intended as a data analysis course, but rather reviews key issues identified in published software engineering experiments, providing guidance based on over 25 years of experience running experiments.