一句话：首个面向真实世界项目的覆盖率工具实证研究，用 Debian 包对 Gcov 与 LLVM-cov 做差分测试，发现 34 个新缺陷。
测量代码覆盖率是软件测试中的关键实践，覆盖率工具报告的错误或误导性信息会推高开发成本，对安全攸关软件尤为危险，因此保证其正确性十分重要。已有工作虽对 C/C++ 最常用的两款覆盖率工具 Gcov 与 LLVM-cov 做过缺陷挖掘，但存在两点局限：仅使用小规模、往往是合成的程序，可能漏掉真实场景中的缺陷；且只关注基础的行覆盖，忽略了实现更复杂、却在安全攸关软件中常被要求的高级覆盖度量。本文提出 DebCovDiff，一个以 Debian 软件包作为输入程序的测试框架，对 Gcov 与 LLVM-cov 的行覆盖及两种高级覆盖度量执行差分测试，并设计了鲁棒的差分判定器，用于过滤工具输出呈现细节差异导致的不一致、克服部分软件包固有的非确定性，以及支持高级覆盖度量。在 47 个 Debian 软件包上的实验共发现 34 个新缺陷，其中 2 个为崩溃缺陷，32 个为产生错误覆盖报告的更深层缺陷。
Measuring code coverage is a critical practice in software testing. Incorrect or misleading coverage information reported by automatic tools can increase the software development cost and lead to negative consequences especially for safety-critical software. Ensuring the correctness of coverage measurement tools is therefore important. Prior studies have applied various techniques to find bugs in Gcov and LLVM-cov, the two most widely used coverage tools for C/C++. However, those studies had two limiting factors. First, they used only small, often synthetic, programs, potentially missing bugs in real-world scenarios. Second, they focused only on basic line coverage, neglecting advanced metrics that are both more complex to implement and commonly required for safety-critical software.This paper presents the first empirical study of coverage measurement tools for real-world projects. We implement DebCovDiff, a testing framework that takes Debian packages as the input programs and performs differential testing of Gcov and LLVM-cov, for line coverage and two advanced coverage metrics. We design robust differential oracles to (1) filter out discrepancies arising from subtle differences in the tool output presentation, (2) overcome the nondeterministic nature of certain packages, and (3) support advanced coverage metrics. From results on 47 Debian packages, we identify 34 new bugs, including 2 crashing bugs and 32 deeper bugs that produce wrong coverage reports.
DOI 原文 ·
@inproceedings{paperbot9539,
title = {DebCovDiff: Differential Testing of Coverage Measurement Tools on Real-World Projects},
author = {Wentao Zhang and Jinghao Jia and Erkai Yu and Darko Marinov and Tianyin Xu},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00094}
}