一句话:提出 RepGen,利用 LLM 驱动的智能 Agent 自动复现深度学习 bug,复现率达 80.19%。
深度学习应用广泛,但其 bug 因模型非确定性和软硬件环境耦合而极难可靠复现,人工方法仅能复现约 3% 的深度学习 bug。本文提出 RepGen,一种自动化的智能复现方法。RepGen 从项目中构建学习增强上下文,制定全面的 bug 复现计划,并采用迭代的生成-验证-精化机制,利用 LLM 生成能够复现目标 bug 的代码。作者在 106 个真实深度学习 bug 上评估 RepGen,取得 80.19% 的复现率,相比当前最佳方法提升 19.81%。一项 27 人参与的开发者研究表明,RepGen 将深度学习 bug 复现成功率提升 23.35%,复现时间减少 56.8%,并降低参与者的认知负荷。
Despite their wide adoption in various domains (e.g., healthcare, finance, software engineering), Deep Learning (DL)-based applications suffer from many bugs, failures, and vulnerabilities. Reproducing these bugs is essential for their resolution, but it is extremely challenging due to the inherent nondeterminism of DL models and their tight coupling with hardware and software environments. According to recent studies, only about 3% of DL bugs can be reliably reproduced using manual approaches. To address these challenges, we present RepGen, a novel, automated, and intelligent approach for reproducing deep learning bugs. RepGen constructs a learning-enhanced context from a project, develops a comprehensive plan for bug reproduction, employs an iterative generate-validate-refine mechanism, and thus generates such code using an LLM that reproduces the bug at hand. We evaluate RepGen on 106 real-world deep learning bugs and achieve a reproduction rate of 80.19%, a 19.81% improvement over the state-of-the-art measure. A developer study involving 27 participants shows that RepGen improves the success rate of DL bug reproduction by 23.35%, reduces the time to reproduce by 56.8%, and lowers participants’ cognitive load.