一句话：实证揭示问题跟踪数据使用中因忽略时间顺序导致的数据泄漏，并量化其对三类预测任务的影响。
问题跟踪数据常被用于预测或推荐软件开发实践，但其属性随时间变化，研究者可能在独立于应用场景的时间点采集数据，从而因忽略数据产生的时间顺序而引入数据泄漏。本文考察已有文献以确认数据泄漏的存在，并复现三类典型研究——重复议题检测、议题定位和议题修复时间预测——使用适配时间的数据量化泄漏影响。结果显示，58 篇研究中 11 篇存在泄漏问题，44 篇疑似存在；数据泄漏会带来有偏结果，但程度并不惊人；summary、component 和 assignee 属性对结果影响最大。作者建议使用者在利用问题跟踪数据时充分理解应用场景、数据来源与变化以及关键属性，以管理数据泄漏风险。
Issue tracking data have been used extensively to aid in predicting or recommending software development practices. Issue attributes typically change over time, but users may use data from a separate time of data collection rather than the time of their application scenarios. We, therefore, investigate data leakage, which results from ignoring the chronological order in which the data were produced. Information leaked from the "future" makes prediction models misleadingly optimistic. We examine existing literature to confirm the existence of data leakage and reproduce three typical studies (detecting duplicate issues, localizing issues, and predicting issue-fix time) adjusted for appropriate data to quantify the impact of the data leakage. We confirm that 11 out of 58 studies have leakage problem, while 44 are suspected. We observe biased results caused by data leakage while the extent is not striking. Attributes of summary, component, and assignee have the largest impact on the results. Our findings suggest that data users are often unaware of the context of the data being produced. We recommend researchers and practitioners who attempt to utilize issue tracking data to address software development problems to have a full understanding of their application scenarios, the origin and change of the data, and the influential issue attributes to manage data leakage risks.
DOI 原文 ·
@inproceedings{paperbot4578,
title = {Be careful of when: an empirical study on time-related misuse of issue tracking data},
author = {Feifei Tu and Jiaxin Zhu and Qimu Zheng and Minghui Zhou},
booktitle = {FSE},
year = {2018},
doi = {10.1145/3236024.3236054}
}