一句话：实证评估EVOSUITE与Randoop在五个机器学习库上的单元测试生成效果，发现现有工具提升有限且未覆盖代码存在共性模式。
现有自动单元测试生成工具多面向通用软件评估，其在统计导向、构造方式迥异的机器学习库上的实用效果尚不明确。本文对五个广泛使用的机器学习库开展实证研究，采用EVOSUITE与Randoop两种流行工具生成单元测试，并以代码覆盖率和变异得分作为质量指标进行衡量。结果表明：第一，多数机器学习库自身维护的单元测试套件质量不高，平均代码覆盖率仅34.1%，平均变异得分仅21.3%；第二，两种生成工具虽能明显提升覆盖率与变异得分，但提升幅度有限；第三，五个库中未被覆盖的代码存在共性模式，这些模式可用于改进单元测试生成任务。该工作揭示了自动测试生成在机器学习库场景下的局限，并为后续针对性改进提供了实证依据。
Automatic unit test generation that explores the input space and produces effective test cases for given programs have been studied for decades. Many unit test generation tools that can help generate unit test cases with high structural coverage over a program have been examined. However, the fact that existing test generation tools are mainly evaluated on general software programs calls into question about its practical effectiveness and usefulness for machine learning libraries, which are statistically orientated and have fundamentally different nature and construction from general software projects. In this paper, we set out to investigate the effectiveness of existing unit test generation techniques on machine learning libraries. To investigate this issue, we conducted an empirical study on five widely used machine learning libraries with two popular unit testcase generation tools, i.e., EVOSUITE and Randoop. We find that (1) most of the machine learning libraries do not maintain a high-quality unit test suite regarding commonly applied quality metrics such as code coverage (on average is 34.1%) and mutation score (on average is 21.3%), (2) unit test case generation tools, i.e., EVOSUITE and Randoop, lead to clear improvements in code coverage and mutation score, however, the improvement is limited, and (3) there exist common patterns in the uncovered code across the five machine learning libraries that can be used to improve unit test case generation tasks.
@inproceedings{WangSSWWN21,
title = {Automatic Unit Test Generation for Machine Learning Libraries: How Far Are We?},
author = {Song Wang and Nishtha Shrestha and Abarna Kucheri Subburaman and Junjie Wang and Moshi Wei and Nachiappan Nagappan},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00138}
}