一句话：通过 Mozilla Firefox 真实查询，比较人类、GPT 与 RAG-GPT 回答技术问题的效果。
随着 LLM 在软件开发支持中的使用增长，本文与 Mozilla 基金会合作，评估检索增强生成（RAG）在 Firefox 项目中辅助开发者的效果。研究收集 Mozilla 开发者聊天室的真实查询，让人类开发者、标准 GPT 模型以及增强 RAG 的 GPT 模型分别作答，并由 Mozilla 专家从有用性、全面性和简洁性三个维度评估。结果显示，RAG 辅助回答在全面性上高于人类开发者（62.50% 对 54.17%），有用性接近人类（75.00% 对 79.17%），但不够简洁、常显冗长。这表明 RAG 工具有潜力减轻开源软件核心维护者负担，同时不显著降低回答质量。
The use of Large Language Models (LLMs) to support tasks in software development has steadily increased over recent years. From assisting developers in coding activities to providing conversational agents that answer newcomers’ questions. In collaboration with the Mozilla Foundation, this study evaluates the effectiveness of Retrieval-Augmented Generation (RAG) in assisting developers within the Mozilla Firefox project. We conducted an empirical analysis comparing responses from human developers, a standard GPT model, and a GPT model enhanced with RAG, using real queries from Mozilla’s developer chat rooms. To ensure a rigorous evaluation, Mozilla experts assessed the responses based on helpfulness, comprehensiveness, and conciseness. The results show that RAG-assisted responses were more comprehensive than human developers (62.50% to 54.17%) and almost as helpful (75.00% to 79.17%), suggesting RAG’s potential to enhance developer assistance. However, the RAG responses were not as concise and often verbose. The results show the potential to apply RAG-based tools to Open Source Software (OSS) to minimize the load to core maintainers without losing answer quality. Toning down retrieval mechanisms and making responses even shorter in the future would enhance developer assistance in massive projects like Mozilla Firefox.
@inproceedings{abs-2510-21933,
title = {A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case},
author = {João Correia and Daniel Coutinho and Marco Castelluccio and Caio Barbosa and Rafael de Mello and Anita Sarma and Alessandro Garcia and Marco Gerosa and Igor Steinmacher},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773170}
}