一句话：提出 ACH：用变异引导 LLM 为特定问题生成少量高价值测试，并在 Meta 工业规模验证。
论文介绍 Meta 的自动化合规加固（ACH）系统，用于变异引导的 LLM 测试生成。传统变异测试通常产生大量变异体，ACH 则仅针对特定关注问题生成当前未被检测到的故障（变异体），再由 LLM 生成能杀死这些变异体的测试，从而加固平台、防止回归。作者以隐私问题为例，但方法可推广到其他回归类型。ACH 已应用于 Meta 的 7 个软件平台、10,795 个 Android Kotlin 类，生成 9,095 个变异体和 571 个隐私加固测试用例；其 LLM 等价变异体检测代理精度 0.79、召回 0.47，简单预处理后达 0.95 和 0.96。Messenger 与 WhatsApp 测试活动中工程师接受 73% 的测试，其中 36% 被认为与隐私相关。
This paper1 describes Meta's Automated Compliance Hardening (ACH) system for mutation-guided LLM-based test generation. ACH generates relatively few mutants (aka simulated faults), compared to traditional mutation testing. Instead, it focuses on generating currently undetected faults that are specific to an issue of concern. From these currently uncaught faults, ACH generates tests that can catch them, thereby 'killing' the mutants and consequently hardening the platform against regressions. We use privacy concerns to illustrate our approach, but ACH can harden code against any type of regression. In total, ACH was applied to 10,795 Android Kotlin classes in 7 software platforms deployed by Meta, from which it generated 9,095 mutants and 571 privacy-hardening test cases. ACH also deploys an LLM-based equivalent mutant detection agent that achieves a precision of 0.79 and a recall of 0.47 (rising to 0.95 and 0.96 with simple pre-processing). ACH was used in Messenger and WhatsApp test-a-thons where engineers accepted 73% of its tests, judging 36% to privacy relevant. We conclude that ACH hardens code against specific concerns and that, even when its tests do not directly tackle the specific concern, engineers find them useful for their other benefits.
@inproceedings{HarmanRHSMGFR25,
title = {Mutation-Guided LLM-based Test Generation at Meta},
author = {Mark Harman and Jillian Ritchey and Inna Harper and Shubho Sengupta and Ke Mao and Abhishek Gulati and Christopher Foster and Hervé Robert},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728544}
}