paperbot · PL 论文追踪

全部论文 待读 收藏

Continuum: Automated Construction and Retrieval of Software Decision Knowledge Graphs from Developer-AI Conversations

Continuum:从开发者-AI 对话中自动构建与检索软件决策知识图谱
ASE Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering2026引用 0 inbox
Mohammad Ali Shehral, Karthik Ravi, Nikhil Trivedi, Akram Bayat
crossref: 2026-10-09

AI 速览

一句话:提出端到端系统 Continuum,从开发者与 AI 助手的对话日志中抽取架构决策并构建可检索的知识图谱。

开发者在与编码助手、聊天式编程工具的对话中会产生大量架构决策痕迹(触发因素、上下文、备选方案、决策与理由),但会话结束后这些信息即丢失:日志非结构化、未被索引、彼此孤立,后来加入的团队成员无法追溯某项技术选型的缘由。本文提出 Continuum,一条端到端流水线,包含五个工程化阶段:基于思维链提示的 LLM 决策抽取;结合缓存查询、534 条技术名规范词典、模糊匹配与嵌入相似度的七阶段级联实体消解;Neo4j 知识图谱配合 PostgreSQL/Redis 的多语言持久化;融合 BM25 全文与 NV-EmbedQA 向量检索(RRF 融合 + K 跳子图扩展)的 GraphRAG 检索层;以及基于 MCP 的智能体集成层,使编码智能体既能查询历史决策也能贡献新决策。在 9 个技术领域、200 段合成开发者-AI 对话上采用留出法评估:实体消解在 981 条测试提及上达到 97.2% 准确率(B-cubed F1 = 0.979),显著优于精确匹配(74.6%)与仅模糊匹配(76.7%);决策抽取精度为 75.0%,排除探索性非决策后为 97.3%;GraphRAG 在 40 条图派生查询上实现全召回(MRR = 0.775)。真实日志案例中对 Vibe Voyager 的 Claude Code 会话历史抽取 379 条决策与 290 个规范实体,验证了可行性。

问题
开发者与 AI 助手的对话中包含丰富的架构决策痕迹,但会话结束后这些知识即丢失,日志非结构化、不可检索且彼此孤立,团队后续成员无法恢复技术选型的上下文与备选方案。
方法
构建端到端系统 Continuum:用思维链提示让 LLM 抽取决策痕迹,经七阶段级联实体消解(缓存、规范词典、模糊匹配、嵌入相似度)归一化实体,存入 Neo4j 知识图谱并多语言持久化,再以融合 BM25 与向量检索的 GraphRAG 层提供检索,并通过 MCP 协议让编码智能体查询与贡献决策。
评估
在 9 个领域 200 段合成开发者-AI 对话上按留出法评估:实体消解准确率 97.2%(981 条提及,B-cubed F1 = 0.979),优于精确匹配与模糊匹配基线(McNemar 检验 p < 0.001);人工判定 40 条决策痕迹得到抽取精度 75.0%(排除探索性非决策后 97.3%);40 条图派生查询上 GraphRAG 全召回(MRR = 0.775);并以 Vibe Voyager 的完整 Claude Code 会话历史作为真实日志案例研究。
原文摘要(Abstract)

Developer-AI conversations, through coding assistants and chat-based programming aids, produce rich architectural decision traces (trigger, context, alternatives considered, decision, rationale) that are lost when the chat session ends. Conversation logs are unstructured, not indexed for retrieval, and disconnected from one another, so a team member joining weeks later cannot recover why a technology was chosen or what alternatives were considered. We present Continuum, an end-to-end system that captures and serves software decision knowledge from developer-AI conversation logs. The system integrates five engineered stages: (1) LLM-based decision extraction with chain-of-thought prompting, (2) a 7-stage cascading entity resolution pipeline combining cached lookups, a curated canonical dictionary (534 technology-name mappings), fuzzy matching, and embedding similarity, (3) a Neo4j knowledge graph with polyglot persistence across PostgreSQL and Redis, (4) a GraphRAG retrieval layer combining BM25 fulltext and NV-EmbedQA vector search via Reciprocal Rank Fusion with K-hop subgraph expansion, and (5) an agent integration layer via the Model Context Protocol (MCP) that enables coding agents to both query prior decisions and contribute new ones. The contribution is the end-to-end integration: prior work has explored individual pieces (decision extraction, knowledge graphs, GraphRAG), but not a deployable system that ties them together for developer-AI conversations. We report a preliminary evaluation on a synthetic corpus of 200 generated developer-AI conversations across 9 technical domains, using a held-out train/test methodology. On our synthetic dataset, the 7-stage entity resolver attains 97.2% accuracy on 981 test mentions (95% CI: [96.2%, 98.3%]), outperforming exact-match (74.6%) and fuzzy-only (76.7%) baselines under McNemar's test (p < 0.001), with B-cubed F1 = 0.979. Two of the authors independently judged 40 sampled decision traces, yielding extraction precision of 75.0% (all categories) and 97.3% when exploratory non-decisions are excluded; on the 10-item overlap, Cohen's kappa = 0.07 reflects genuine disagreement on what qualifies as a "decision" in open-ended AI chat. On 40 graph-derived queries, hybrid GraphRAG retrieval achieves full recall across decision-search, tech-comparison, and entity-lookup query types (MRR = 0.775); a manual relevance assessment by the two reviewers on 10 natural-language queries yielded a mean rating of 2.2/3.0. As a real-log case study, we apply the pipeline to the full Claude Code session history from the build of Vibe Voyager, a public agent-built browser game (github.com/shehral/vibe); from 77 segmented and scrubbed sub-conversations the system extracts 379 decision traces and 290 unique canonical entities, with the decisions splitting 65% agent-made vs. 35% human-made (Section 4.2). We are transparent about the limitations of this evaluation (synthetic corpus, small annotator pool, proxy ground truth for 150 conversations, graph-derived queries) and discuss them in Section 4.3. Overall, these results indicate that architectural decision knowledge can be recovered from developer-AI conversations with high entity-resolution accuracy and demonstrated feasibility on real logs, while robust measurement of extraction quality and generalization to multi-developer settings remain open.

链接与引用

DOI 原文 · PDF(开放获取) ·

BibTeX
@inproceedings{paperbot12414,
  title = {Continuum: Automated Construction and Retrieval of Software Decision Knowledge Graphs from Developer-AI Conversations},
  author = {Mohammad Ali Shehral and Karthik Ravi and Nikhil Trivedi and Akram Bayat},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3834535}
}