一句话：首个自动生成异常行为测试的框架，通过指令微调 CodeLlama 并嵌入异常路径推理，超越现有模型与工具。
异常行为测试（EBT）用于验证代码能否检测异常并抛出恰当异常，但开发者大多只关注无异常事件的“happy path”，导致这类测试严重缺失。本文提出 ExLoNG——首个自动生成 EBT 的框架，它是对 CodeLlama 进行指令微调得到的大语言模型，并在生成过程中嵌入三类推理信息：通向 throw 语句的执行轨迹、守卫 throw 语句的条件表达式，以及执行相似轨迹的非异常行为测试。评估方面，作者将 ExLoNG 与当前最优的测试生成模型 CAT-LM、最强基础模型之一 GPT-4o，以及基于分析的测试生成工具 Randoop 和 EvoSuite 进行对比，结果显示 ExLoNG 均优于这些模型与工具。此外，作者向多个开源项目提交了 pull request，其中由 ExLoNG 生成的 23 个 EBT 已被合并接受，验证了其实际可用性。
Many popular programming languages, including C#, Java, and Python, support exceptions. Exceptions are thrown during program execution if an unwanted event happens, e.g., a method is invoked with an illegal argument value. Software developers write exceptional behavior tests (EBTs) to check that their code detects unwanted events and throws appropriate exceptions. Prior research studies have shown the importance of EBTs, but those studies also highlighted that developers put most of their efforts on “happy paths”, e.g., paths without unwanted events. To help developers fill the gap, we present the first framework, dubbed ExLoNG, that automatically generates EBTs. ExLONG is a large language model instruction fine-tuned from CodeLlama and embeds reasoning about traces that lead to throw statements, conditional expressions that guard throw statements, and non-exceptional behavior tests that execute similar traces. We compare ExLONG with the state-of-the-art models for test generation (CAT-LM) and one of the strongest foundation models (GPT-4o), as well as with analysis-based tools for test generation (Randoop and EvoSuite). Our results show that ExLONG outperforms existing models and tools. Furthermore, we contributed several pull requests to open-source projects and 23 EBTs generated by ExLONG were already accepted.
DOI 原文 ·
@inproceedings{paperbot8820,
title = {exLong: Generating Exceptional Behavior Tests with Large Language Models},
author = {Jiyang Zhang and Yu Liu and Pengyu Nie and Junyi Jessy Li and Milos Gligoric},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00176}
}