一句话：构建航空航天需求问答基准，评估五种开源LLM在零样本、少样本及RAG下的表现，分析幻觉并开展用户研究。
航空航天软件需求工程因设计复杂、安全标准严格而困难，人工撰写需求文档需兼顾领域知识与异构数据，易出错且低效。本文评估LLM理解航空航天需求并辅助需求问答（QA）的能力，基于工业资产、书籍与研究资料构建含6,696个QA对、覆盖十项任务及文本/表格/公式三种格式的基准；采用零样本、少样本和RAG评估五种开源LLM，分类幻觉并分析误差，同时开展用户研究。结果表明：LLM在该领域表现有限；RAG显著提升文本任务，少样本对多数模型有效；识别出四类QA幻觉；LLM问答对初级工程师尤其有益。
Aerospace software presents significant challenges to requirements engineering due to its design complexity and stringent safety standards. When manually drafting requirement documents, engineers need strong domain knowledge while also navigating heterogeneous data, which leads to errors and inefficiencies. This paper evaluates the capabilities of large language models (LLMs) in understanding aerospace software requirements and their potential to assist in requirements question answering (QA). We develop an aerospace requirements QA benchmark based on industrial software assets, books, and research materials, creating a total of 6, 696 QA pairs across ten tasks and three heterogeneous data formats: text, tables, and formulas. We then evaluate the domain-specific performance of five mainstream open-source LLMs using zero-shot learning, few-shot learning, and retrieval-augmented generation (RAG) techniques. We further categorize hallucinations from LLMs and quantitatively analyze error distributions. Moreover, we conduct a user study to assess the LLM's practical usefulness when applying to requirements QA. The evaluation results show that (1) LLMs demonstrate limited performance in the aerospace software domain, (2) RAG techniques significantly enhance the capabilities of LLMs for text-based tasks, while few-shot learning improves the performance of most LLMs, (3) four distinct types of QA hallucinations are identified, and (4) LLM QA is particularly beneficial for junior engineers. This research provides valuable perspectives for the future application of LLMs in aerospace software.
@inproceedings{paperbot9062,
title = {Evaluating Large Language Models for Requirements Question Answering in Industrial Aerospace Software},
author = {Longxing Yang and Yixing Luo and Hao Gao and Yingshuang Fan and Jingru Zhang and Xiaofeng Li and Xiaogang Dong and Bin Gu and Zhi Jin and Mengfei Yang},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728560}
}