一句话：通过系统文献检索与主题综合，为程序理解实验中最常报告的有效性威胁构建证据档案，揭示多数引用缺乏证据支持。
程序理解实验常把某些因素（如编程经验、程序长度、理解度量）视为普遍的有效性威胁，但这类假设未必有证据支持。本文开展两阶段研究，结合系统文献检索、滚雪球抽样与主题综合，先识别代码理解原始研究中最常讨论的有效性威胁类别，再为三类最常报告的威胁构建“证据档案”。结果显示，409篇原始研究中仅31篇在提及有效性威胁时引用了支持证据；对三类最常见威胁所引用的18项研究中，17项未达到作者的证据标准。作者因此主张，许多威胁的影响取决于具体研究设计与情境，应在特定研究语境中讨论并以证据支撑，并呼吁更多此类元研究，其方法也可推广到其他软件工程领域。
Searching for clues, gathering evidence, and reviewing case files are all techniques used by criminal investigators to draw sound conclusions and avoid wrongful convictions. Medicine, too, has a long tradition of evidence-based practice, in which administering a treatment without evidence of its efficacy is considered malpractice. Similarly, in software engineering (SE) research, we can develop sound methodologies and mitigate threats to validity by basing study design decisions on evidence. Echoing a recent call for the empirical evaluation of design decisions in program comprehension experiments, we conducted a 2-phases study consisting of systematic literature searches, snowballing, and thematic synthesis. We found out (1) which validity threat categories are most often discussed in primary studies of code comprehension, and we collected evidence to build (2) the evidence profiles for the three most commonly reported threats to validity. We discovered that few mentions of validity threats in primary studies (31 of 409) included a reference to supporting evidence. For the three most commonly mentioned threats, namely the influence of programming experience, program length, and the selected comprehension measures, almost all cited studies (17 of 18) did not meet our criteria for evidence. We show that for many threats to validity that are currently assumed to be influential across all studies, their actual impact may depend on the design and context of each specific study. Researchers should discuss threats to validity within the context of their particular study and support their discussions with evidence. The present paper can be one resource for evidence, and we call for more meta-studies of this type to be conducted, which will then inform design decisions in primary studies. Further, although we have applied our methodology in the context of program comprehension, our approach can also be used in other SE research areas to enable evidence-based experiment design decisions and meaningful discussions of threats to validity.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{BaronWGW23,
title = {Evidence Profiles for Validity Threats in Program Comprehension Experiments},
author = {Marvin Muñoz Barón and Marvin Wyrich and Daniel Graziotin and Stefan Wagner},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00162}
}