paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Retrieval-Augmented Test Generation: How Far Are We?

检索增强的测试生成：我们距离目标还有多远？
ICSE Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering2026匹配度 3/10 · 论文关注用 RAG 为 ML/DL API 生成单元测试，属于 LLM 辅助软件测试，而非你重点关注的静态分析、形式化验证、浮点程序分析或贝叶斯程序分析。虽然涉及机器学习系统与 LLM 工具，但与 PL↔AI 交叉的核心问题距离较远。引用 24 inbox
Jiho Shin, Nima Shiri Harzevili, Reem Aleithan, Hadi Hemmati, Song Wang
测试、调试与修复机器学习与程序语言
crossref: 2026-09-12dblp: 2026-09-12arxiv: 2026-09-12

AI 速览

一句话：系统评估 RAG 在 ML/DL API 单元测试生成中的效果：RAG 不改善正确性，但平均提升行覆盖率 6.5%，并发现 28 个新 bug。

RAG 已在多个软件工程任务中见效，但在单元测试生成中研究不足，ML/DL API 场景尤甚。本文以 TensorFlow、PyTorch、Scikit-learn、JAX、XGBoost 五个 Python 库的热门 API 为对象，考察 API 文档、GitHub issues、StackOverflow 问答三类知识源，并在 GPT-3.5-Turbo、GPT-4o、Mistral MoE 8x22B、Llama 3.1 405B 上比较基础指令提示、Basic RAG 与 API-level RAG 三种策略，从语法正确性、动态正确性与行覆盖率量化评估。结果显示 RAG 未提升正确性，但行覆盖率平均提高 6.5%，GitHub issues 因提供边界情形增益最大；生成测试还发现 28 个新 bug，24 个已报告，10 个获确认。

问题
RAG 在单元测试生成中尚缺乏系统研究，尤其针对 ML/DL API，且不同类型领域知识源对生成效果的影响不明。
方法
对五个 Python ML/DL 库的热门 API，分别以 API 文档、GitHub issues、StackOverflow 问答构建 RAG，在四个 LLM 上比较基础指令提示、Basic RAG 与 API-level RAG 三种策略。
评估
从语法正确性、动态正确性与行覆盖率进行量化评估，并统计生成测试发现并报告给开发者的新 bug 数量与确认情况。
原文摘要（Abstract）

Retrieval Augmented Generation (RAG) has advanced software engineering tasks but remains underexplored in unit test generation. To bridge this gap, we investigate the efficacy of RAG-based unit test generation for machine learning (ML/DL) APIs and analyze the impact of different knowledge sources on their effectiveness. We examine three domain-specific sources for RAG: (1) API documentation (official guidelines), (2) GitHub issues (developer-reported resolutions), and (3) StackOverflow Q&As (community-driven solutions). Our study focuses on five widely used Python-based ML/DL libraries, TensorFlow, PyTorch, Scikit-learn, Google JAX, and XGBoost, targeting the most-used APIs. We evaluate four state-of-the-art LLMs: GPT-3.5-Turbo, GPT-4o, Mistral MoE 8x22B, and Llama 3.1 405B, across three strategies: basic instruction prompting, Basic RAG, and API-level RAG. Quantitatively, we assess syntactical and dynamic correctness and line coverage. While RAG does not improve correctness, it increases line coverage by 6.5% on average. We found that GitHub issues yield the greatest improvement in line coverage by providing edge cases from various issues. We also found that these generated unit tests can help detect new bugs. Specifically, 28 bugs were detected, 24 unique bugs were reported to developers, ten were confirmed, four were rejected, and ten are awaiting developers’ confirmation. Our findings highlight RAG’s potential in unit test generation for improving test coverage with well-targeted knowledge sources. Future work should focus on retrieval techniques that identify documents with unique program states to optimize RAG-based unit test generation further.

链接与引用

DOI 原文 · arXiv · DBLP

BibTeX 
@inproceedings{abs-2409-12682,
  title = {Retrieval-Augmented Test Generation: How Far Are We?},
  author = {Jiho Shin and Nima Shiri Harzevili and Reem Aleithan and Hadi Hemmati and Song Wang},
  booktitle = {ICSE},
  year = {2026},
  doi = {10.1145/3744916.3773163}
}