一句话：提出 LLPut，实证评估三种开源生成式 LLM 从缺陷报告中提取失败触发输入的效果。
失败触发输入对诊断和分析软件缺陷至关重要，而这类输入通常隐含在自然语言书写的缺陷报告里，需要开发者手工提取。已有工作借助各类自然语言处理技术实现自动化输入抽取。随着生成式大语言模型的出现，一个关键问题随之而来：生成式 LLM 究竟能在多大程度上从缺陷报告中准确抽取失败触发输入？本文提出 LLPut，一个用于实证评估该问题的技术方案，选取 LLaMA、Qwen 与 Qwen-Coder 三个开源生成式 LLM，在包含 206 份缺陷报告的数据集上开展实验，从准确性与有效性两方面考察模型抽取相关输入的表现，并总结了生成式 LLM 在自动化缺陷诊断中的能力与局限。
Failure-inducing inputs play a crucial role in diagnosing and analyzing software bugs. Bug reports typically contain these inputs, which developers extract to facilitate debugging. Since bug reports are written in natural language, prior research has leveraged various Natural Language Processing (NLP) techniques for automated input extraction. With the advent of Large Language Models (LLMs), an important research question arises: how effectively can generative LLMs extract failure-inducing inputs from bug reports? In this paper, we propose LLPut, a technique to empirically evaluate the performance of three open-source generative LLMs—LLaMA, Qwen, and Qwen-Coder—in extracting relevant inputs from bug reports. We conduct an experimental evaluation on a dataset of 206 bug reports to assess the accuracy and effectiveness of these models. Our findings provide insights into the capabilities and limitations of generative LLMs in automated bug diagnosis.
@inproceedings{paperbot9111,
title = {LLPut: Investigating Large Language Models for Bug Report-Based Input Generation},
author = {Alif Al Hasan and Subarna Saha and Mia Mohammad Imran and Tarannum Shaila Zaman},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728701}
}