一句话：首个针对 AI 编程工具的大规模 bug 实证研究，基于 3.8K+ GitHub 报告刻画其失败模式与工程挑战。
LLM 正快速融入软件开发流程，催生了 Claude Code、Codex CLI、Gemini CLI 等 AI 辅助编程工具。这些工具处于传统软件工程、AI 系统设计与人机交互的交叉点，其构建过程存在独特且缺乏理解的工程难题。本文首次对 AI 编程工具中的工程陷阱开展实证研究，系统人工分析了来自这三款工具的 3.8K 余条公开 GitHub bug 报告。作者采用开放编码，考察 issue 描述、用户讨论与开发者回复，并按 bug 类型、位置、根因与症状进行标注，以刻画常见失败模式与反复出现的工程挑战。结果显示：超过 67% 的 bug 与功能相关；根因方面，37.3% 源于 API、集成或配置错误；用户报告最多的症状是 API 错误（18.3%）、终端问题（14%）与命令失败（12.7%）；这些 bug 主要影响系统工作流中的工具调用（37.6%）与命令执行（25%）阶段。该研究为设计下一代可靠、鲁棒的 AI 编程工具提供了关键路线图。
The rapid integration of Large Language Models (LLMs) into software development workflows has given rise to a new class of AI-assisted coding tools, such as Claude Code, Codex, and Gemini CLIs. While promising significant productivity gains, the engineering process of building these tools, which sit at the complex intersection of traditional software engineering, AI system design, and human-computer interaction, is fraught with unique and poorly understood challenges. This paper presents the first empirical study of engineering pitfalls in AI coding tools, based on a systematic manual analysis of over 3.8K publicly reported GitHub bugs from Claude Code, Codex CLI, and Gemini CLI. Using open coding, we examine issue descriptions, user discussions, and developer responses, and annotate each bug by type, location, root cause, and symptom to characterize common failure patterns and recurring engineering challenges. Our results show that more than 67% of the bugs in these tools are related to functionality. In terms of root causes, 37.3% of the bugs stem from API, integration, or configuration errors. Consequently, the most commonly observed symptoms reported by users are API errors (18.3%), terminal problems (14%), and command failures (12.7%). These bugs predominantly affect the tool invocation (37.6%) and command execution (25%) stages of the system workflow. Collectively, our findings provide a critical roadmap for developers seeking to design the next generation of reliable and robust AI coding tools.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2603-20847,
title = {Engineering Pitfalls in AI Coding Tools: An Empirical Study of Bugs in Claude Code, Codex, and Gemini CLI},
author = {Ruixin Zhang and Wuyang Dai and Hung Viet Pham and Gias Uddin and Jinqiu Yang and Song Wang},
booktitle = {FSE},
year = {2026},
doi = {10.1145/3803437.3805213}
}