一句话：首个针对 IDE 原生 AI 编程助手的大规模实证研究，基于 11,579 个真实会话揭示开发者对话式编程的行为模式。
IDE 集成的 AI 编程助手可在开发者真实代码库中对话式工作，并能访问项目上下文、进行多文件编辑，正在改变软件开发方式。然而现有实证研究多基于小规模受控实验，或分析通用聊天机器人，而非具备代码库感知的 IDE 工作流。本文首次对 IDE 原生环境中的真实对话式编程开展大规模研究，分析了 Cursor 与 GitHub Copilot 中来自 1,300 个仓库、899 名开发者的 11,579 个会话、74,998 条开发者消息；这些对话作为日常开发的一部分被提交到公开仓库，因而反映真实行为。研究发现三方面转变：对话式编程表现为渐进式规格说明，开发者迭代细化输出而非一次性给出完整任务；开发者将诊断、理解与验证等认知工作重新分配给 AI；开发者主动管理协作，将计划外化为持久产物，并通过上下文注入与行为约束协商 AI 自主性。
IDE-integrated AI coding assistants, which operate conversationally within developers' working codebases with access to project context and multi-file editing, are rapidly reshaping software development. However, empirical investigation of this shift remains limited: existing studies largely rely on small-scale, controlled settings or analyze general-purpose chatbots rather than codebase-aware IDE workflows. We present, to the best of our knowledge, the first large-scale study of real-world conversational programming in IDE-native settings, analyzing 74,998 developer messages from 11,579 chat sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot. These chats were committed to public repositories as part of routine development, capturing in-the-wild behavior. Our findings reveal three shifts in how programming work is organized: conversational programming operates as progressive specification, with developers iteratively refining outputs rather than specifying complete tasks upfront; developers redistribute cognitive work to AI, delegating diagnosis, comprehension, and validation rather than engaging with code and outputs directly; and developers actively manage the collaboration, externalizing plans into persistent artifacts, and negotiating AI autonomy through context injection and behavioral constraints. These results provide foundational empirical insights into AI-assisted development and offer implications for the design of future programming environments.
@inproceedings{abs-2604-00436,
title = {Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions},
author = {Ningzhi Tang and Chaoran Chen and Zihan Fang and Gelei Xu and Maria Dhakal and Yiyu Shi and Collin McMillan and Yu Huang and Toby Jia-Jun Li},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834377}
}