paperbot · PL 论文追踪

全部论文 待读 收藏

P2 On the Evaluation of Large Language Models in Unit Test Generation

大型语言模型在单元测试生成中的评估
ASE Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering2024匹配度 2/10 · 论文聚焦于LLM在Java单元测试生成中的提示策略与开源模型能力评估，属于LLM4SE/测试生成方向，与用户关心的静态分析、形式化验证、浮点程序分析、贝叶斯程序分析以及Agentic程序分析没有直接交集。引用 138 inbox
Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, Junjie Chen
测试、调试与修复机器学习与程序语言
crossref: 2026-09-12dblp: 2026-09-12arxiv: 2026-09-12

AI 速览

一句话：首次实证评估多种开源LLM在不同提示设置下的Java单元测试生成能力，并与GPT-4和Evosuite对比。

单元测试对验证软件组件正确性至关重要，但人工编写耗时且困难。LLM 为自动化单元测试生成提供了新途径，然而已有研究多关注闭源模型（如 ChatGPT、CodeX）和固定提示策略，尚未系统探索先进开源 LLM 在不同提示设置下的能力。本文开展首个针对性实证研究，基于 17 个 Java 项目、五个结构和参数量各异的开源 LLM，并采用综合评估指标，系统考察提示因素对生成效果的影响。研究还比较了开源 LLM 与商用 GPT-4 及传统 Evosuite 的表现，识别了基于 LLM 的单元测试生成局限，并据此给出一系列对未来研究和实践应用的启示。

问题
已有 LLM 单元测试生成研究多聚焦闭源模型和固定提示策略，未充分探索先进开源 LLM 在多种提示设置下的能力。
方法
在 17 个 Java 项目上，对五个不同结构和参数规模的开源 LLM 使用综合评估指标开展实证研究，并分析多种提示因素的影响。
评估
比较开源 LLM 与 GPT-4、Evosuite 的表现，分析提示因素的影响，总结基于 LLM 的单元测试生成局限并提炼实践与未来研究启示。
原文摘要（Abstract）

Unit testing is an essential activity in software development for verifying the correctness of software components. However, manually writing unit tests is challenging and time-consuming. The emergence of Large Language Models (LLMs) offers a new direction for automating unit test generation. Existing research primarily focuses on closed-source LLMs (e.g., ChatGPT and CodeX) with fixed prompting strategies, leaving the capabilities of advanced open-source LLMs with various prompting settings unexplored. Particularly, open-source LLMs offer advantages in data privacy protection and have demonstrated superior performance in some tasks. Moreover, effective prompting is crucial for maximizing LLMs’ capabilities. In this paper, we conduct the first empirical study to fill this gap, based on 17 Java projects, five widely-used open-source LLMs with different structures and parameter sizes, and comprehensive evaluation metrics. Our findings highlight the significant influence of various prompt factors, show the performance of open-source LLMs compared to the commercial GPT-4 and the traditional Evosuite, and identify limitations in LLM-based unit test generation. We then derive a series of implications from our study to guide future research and practical use of LLM-based unit test generation.CCS CONCEPTS • Software and its engineering → Software testing and debugging.

链接与引用

DOI 原文 · arXiv · PDF（开放获取） · DBLP

BibTeX 
@inproceedings{YangYGW0ZCZLW024,
  title = {On the Evaluation of Large Language Models in Unit Test Generation},
  author = {Lin Yang and Chen Yang and Shutao Gao and Weijing Wang and Bo Wang and Qihao Zhu and Xiao Chu and Jianyi Zhou and Guangtai Liang and Qianxiang Wang and Junjie Chen},
  booktitle = {ASE},
  year = {2024},
  doi = {10.1145/3691620.3695529}
}