一句话：提出 LLM4Fin，将微调 LLM 与算法化场景生成结合，全自动从自然语言业务规则生成高覆盖验收测试用例。
金融科技软件需依据监管业务规则进行验收测试，但自然语言规则的非结构化特性与全面用例设计高度依赖人工。现有 LLM 虽可辅助生成测试用例，却常需人工设计提示，效率受限。本文提出全自动方法，将 LLM 与算法方法结合：先微调预训练 LLM 以提升业务规则信息抽取准确率，再基于抽取结果算法化生成全面的可测试场景。原型系统 LLM4Fin 面向真实股票交易软件。实验表明，其优于 ChatGPT 等先进 LLM 和熟练测试工程师，业务场景覆盖率最高达 98.18%，平均提升 20%–110%，代码覆盖率最高达 93.72%，时间成本从 20 分钟降至 7 秒。
FinTech software, crucial for both safety and timely market deployment, presents a compelling case for automated acceptance testing against regulatory business rules. However, the inherent challenges of comprehending unstructured natural language descriptions of these rules and crafting comprehensive test cases demand human intelligence. The emergence of Large Language Models (LLMs) holds promise for automated test case generation, leveraging their natural language processing capabilities. Yet, their dependence on human intervention for effective prompting hampers efficiency. In response, we introduce a groundbreaking, fully automated approach for generating high-coverage test cases from natural language business rules. Our methodology seamlessly integrates the versatility of LLMs with the predictability of algorithmic methods. We fine-tune pre-trained LLMs for improved information extraction accuracy and algorithmically generate comprehensive testable scenarios for the extracted business rules. Our prototype, LLM4Fin, is designed for testing real-world stock-trading software. Experimental results demonstrate LLM4Fin’s superiority over both state-of-the-art LLM, such as ChatGPT, and skilled testing engineers. We achieve remarkable performance, with up to 98.18% and an average of 20%−110% improvement on business scenario coverage, and up to 93.72% on code coverage, while reducing the time cost from 20 minutes to a mere 7 seconds. These results provide robust evidence of the framework’s practical applicability and efficiency, marking a significant advancement in FinTech software testing.
DOI 原文 ·
@inproceedings{paperbot8566,
title = {LLM4Fin: Fully Automating LLM-Powered Test Case Generation for FinTech Software Acceptance Testing},
author = {Zhiyi Xue and Liangguo Li and Senyue Tian and Xiaohong Chen and Pingping Li and Liangyu Chen and Tingting Jiang and Min Zhang},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680388}
}