一句话：通过映射研究分析 55 篇论文中的 194 个 DNN4SE 实验，揭示普遍存在的弱分析、变量控制不足与描述不具体等缺陷。
软件工程任务日益依赖深度神经网络，但评估此类技术的受控实验因架构、超参数、数据集与训练随机性等复杂变量空间而极具挑战。本文对 55 篇发表于软件工程顶级会议的论文中的 194 个 DNN 实验进行映射研究，刻画其实践现状并识别常见趋势与陷阱。研究发现，多数实验即使获得 ACM 工件徽章，仍存在根本性局限，令人质疑其结论可靠性：87% 的实验缺乏充分分析以确立自变量与因变量间的真实关系；100% 的实验对 DNN 相关变量空间控制有限，可能使处理与结果间的关系只是相关而非因果；86% 的实验未具体说明所用 DNN 变量及其取值，导致无法判断所评估的是否为所设计的技术，也难以厘清外部变异的控制方式。最后，作者针对这些局限提出若干实践建议。
Software engineering (SE) techniques are increasingly relying on deep learning approaches to support many SE tasks, from bug triaging to code generation. To assess the efficacy of such techniques researchers typically perform controlled experiments. Conducting these experiments, however, is particularly challenging given the complexity of the space of variables involved, from specialized and intricate architectures and algorithms to a large number of training hyper-parameters and choices of evolving datasets, all compounded by how rapidly the machine learning technology is advancing, and the inherent sources of randomness in the training process. In this work we conduct a mapping study, examining 194 experiments with techniques that rely on deep neural networks (DNNs) appearing in 55 papers published in premier SE venues to provide a characterization of the state of the practice, pinpointing experiments’ common trends and pitfalls. Our study reveals that most of the experiments, including those that have received ACM artifact badges, have fundamental limitations that raise doubts about the reliability of their findings. More specifically, we find: 1) weak analyses to determine that there is a true relationship between independent and dependent variables (87% of the experiments), 2) limited control over the space of DNN relevant variables, which can render a relationship between dependent variables and treatments that may not be causal but rather correlational (100% of the experiments), and 3) lack of specificity in terms of what are the DNN variables and their values utilized in the experiments (86% of the experiments) to define the treatments being applied, which makes it unclear whether the techniques designed are the ones being assessed, or how the sources of extraneous variation are controlled. We provide some practical recommendations to address these limitations.