一句话：系统实证比较 LLM 提示方法与基于约束的符号执行在定向测试输入生成上的效果与局限。
定向测试输入生成旨在构造能够到达特定未覆盖目标分支的输入，是自动测试中长期存在的难题。符号执行与 concolic 测试等基于约束的技术已被广泛研究并集成到现有工具中，而近期工作开始尝试设计定制提示，让大语言模型直接生成可抵达目标分支的输入。然而，LLM 在此任务上的实际有效性，以及其与已被验证的约束求解方案的相对表现，尚缺乏系统性的理解。该论文围绕这一问题展开实证研究，设计并考察基于 LLM 的定向输入生成提示策略，并将其与基于约束的方法在同一目标分支设定下进行比较，分析 LLM 在何种情形下有效、何处失效，以及两类方法的差距与潜在互补性，为后续定向测试输入生成技术的设计提供依据。
Automatic testing has garnered significant attention and success over the past few decades. Techniques such as unit testing and coverage-guided fuzzing have revealed numerous critical software bugs and vulnerabilities. However, a long-standing, formidable challenge for existing techniques is how to achieve higher testing coverage. Constraint-based techniques, such as symbolic execution and concolic testing, have been well-explored and integrated into the existing approaches. With the popularity of Large Language Models (LLMs), recent research efforts to design tailored prompts to generate inputs that can reach more uncovered target branches. However, the effectiveness of using LLMs for generating such directed inputs and the comparison with the proven constraint-based solutions has not been systematically explored.
DOI 原文 ·
@inproceedings{paperbot8383,
title = {Towards Understanding the Effectiveness of Large Language Models on Directed Test Input Generation},
author = {Zongze Jiang and Ming Wen and Jialun Cao and Xuanhua Shi and Hai Jin},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695513}
}