一句话：提出 NITR 探针-预言机框架，诊断行为正确但破坏可维护性的 AI 仓库编辑。
现有 AI 编码 agent 评估多聚焦行为正确性，忽视模块化、可测试性等可维护性风险。本文提出 NITR，一个“探针—预言机”诊断框架：把软件工程经验蒸馏为小型多文件代码库中的受控探针，每个探针只针对一个可维护性维度，并配有隐藏 harness——功能测试保证行为，结构预言机编码可维护性约束并给出可解释诊断。作者评估 GPT、Claude、Gemini、Qwen 的 23 种直接推理与 agent 配置：平均仅解决 36.2%，最佳 57.1%；微案例 53.5% 降至多步 20.6%；依赖控制 4.3%、职责分解 15.2% 最难；13.3% 通过全部功能测试却违反结构预言机；agent 模式把平均从 28.2% 提到 45.0%，但未消除架构性失败。
AI coding agents can now complete complex programming tasks, but existing evaluations largely emphasize behavioral correctness and often overlook maintainability risks such as weak modularity or testability. Automation without accountability shifts invisible cost downstream because hidden maintainability failures incurred today are deferred to the developers who must extend, debug, and sustain these systems tomorrow, raising significant technical and organizational concerns for software engineering. We present Needle in the Repo (NITR), a diagnostic, probe-and-oracle framework for evaluating whether behaviorally correct repository edits preserve maintainable structure. The key idea is to distill recurring software engineering wisdom into controlled probes embedded in small, realistic multi-file codebases, with each probe designed so that success depends primarily on one targeted maintainability dimension. Each probe is paired with a hidden evaluation harness that combines (i) functional tests for required behavior, and (ii) structural oracles that encode the targeted maintainability constraint and return interpretable diagnoses. Using NITR, we extensively evaluate 23 coding configurations across GPT, Claude, Gemini, and Qwen families in both direct-inference and agent-based settings. We find that current AI coding systems remain far from robust. On average, all AI coding configurations solve only 36.2% of cases, the best reaches 57.1%, and performance drops from 53.5% on micro cases to 20.6% on multi-step cases. The hardest pressures are architectural rather than local edits, especially dependency control (4.3%) and responsibility decomposition (15.2%). Moreover, 64/483 outcomes (13.3%) pass all functional tests yet fail the structural oracle. Under our harness, agent-mode configurations improve average performance from 28.2% to 45.0%, but do not eliminate the architectural failures. These results show that progress in code generation is not yet progress in maintainable code evolution, and that NITR exposes a critical failure surface largely missed by conventional evaluation.
@inproceedings{paperbot12480,
title = {Needle in the Repo: Diagnosing Maintainability Failures in AI-Generated Repository Edits},
author = {Haichao Zhu and Qian Zhang and Jiyuan Wang and Zhaorui Yang and Yuxin Qiu},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834401}
}