一句话：首次实证研究深度学习库测试中 oracle 近似断言的普遍性、来源、演化模式与维护挑战。
深度学习库的测试与一般软件不同：一般软件常做确定性断言（如输出与 oracle 相等），而 DL 库测试往往需要 oracle 近似，即允许输出落在 oracle 的某个范围内。已有实证研究只关注传统测试实践，oracle 近似的普遍程度、常见做法及其维护与演化挑战尚不清楚。本文对四个流行深度学习库的测试用例中实现的 oracle 近似断言进行了实证研究。研究表明，DL 库测试中存在相当比例的断言采用 oracle 近似；作者通过全面的人工研究识别出进行近似所依据的常见 oracle 来源；并发现开发者频繁修改与 oracle 近似相关的代码，例如使用不同的近似 API、修改 oracle 或被测代码的输出、以及使用不同的近似阈值。最后，作者深入分析了 oracle 近似断言演化的原因，揭示了随着 DL 库代码演化，开发者在维护 oracle 近似实践时可能面临的重要维护挑战。
Due to the increasing popularity of deep learning (DL) applications, testing DL libraries is becoming more and more important. Different from testing general software, for which output is often asserted definitely (e.g., an output is compared with an oracle for equality), testing deep learning libraries often requires to perform oracle approximations, i.e., the output is allowed to be within a restricted range of the oracle. However, oracle approximation practices have not been studied in prior empirical work that focuses on traditional testing practices. The prevalence, common practices, maintenance and evolution challenges of oracle approximations remain unknown in literature. In this work, we study oracle approximation assertions implemented to test four popular DL libraries. Our study shows that there exists a non-negligible portion of assertions that leverage oracle approximation in the test cases of DL libraries. Also, we identify the common sources of oracles on which oracle approximations are being performed through a comprehensive manual study. Moreover, we find that developers frequently modify code related to oracle approximations, i.e., using a different approximation API, modifying the oracle or the output from the code under test, and using a different approximation threshold. Last, we performed an in-depth study to understand the reasons behind the evolution of oracle approximation assertions. Our findings reveal important maintenance challenges that developers may face when maintaining oracle approximation practices as code evolves in DL libraries.
DOI 原文 ·
@inproceedings{paperbot5338,
title = {A Study of Oracle Approximations in Testing Deep Learning Libraries},
author = {Mahdi Nejadgholi and Jinqiu Yang},
booktitle = {ASE},
year = {2019},
doi = {10.1109/ase.2019.00078}
}