paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Tone Perturbations as Trust Probes: Evaluating Behavioral Stability of LLMs in Software Engineering Tasks

语气扰动作为信任探针:评估 LLM 在软件工程任务中的行为稳定性
FSE Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering2026匹配度 2/10 · 论文关注LLM在软件工程任务中对礼貌/强制语气扰动的行为稳定性,属于LLM自身评估与提示鲁棒性研究,不涉及静态分析、程序分析、形式化验证、浮点程序分析或贝叶斯程序分析,也未将LLM/Agent作为改进程序分析的工具,与用户核心方向不符。引用 0 inbox
Mohamad Kassab, Youssab Ibrahim
crossref: 2026-09-12dblp: 2026-09-12

AI 速览

一句话:通过任务内礼貌/强制语气配对扰动,测量四个 LLM 在软件工程任务中的行为稳定性与安全信号差异。

开发者日益依赖大语言模型完成编程、推理与文档撰写,但其行为在社会语用变化下是否保持稳定尚不清楚。本文开展受控的任务内扰动研究:对 60 个软件工程任务分别构造礼貌与强制语气变体,在四个已部署的 LLM 助手上共获得 480 条响应,形成 240 组任务内配对比较。作者采用经过验证的计算指标并结合针对性人工审阅,测量交际投入、情感立场、任务相关指标与安全信号(拒答、免责声明、毒性)。结果显示:礼貌提示稳定地引出更长、更积极表述的回答(+222 token,约 18%),而强制语气压缩了阐述篇幅,但审计的任务相关指标与安全信号并未出现可观察的退化。语气敏感性因领域与模型而异:伦理推理任务的情感偏移更大,GPT-4o 表现出最高的行为稳定性。作者将语气扰动解释为轻量级鲁棒性探针,而非风格噪声,主张把社会语用变异作为 LLM 辅助软件工程的一等评估维度。

问题
开发者依赖 LLM 完成软件工程任务,但其在真实社会语用变异(如礼貌与强制语气)下是否保持行为稳定缺乏系统证据。
方法
对 60 个软件工程任务构造礼貌与强制语气变体,在四个已部署 LLM 助手上做任务内配对扰动实验(480 条响应、240 组配对),用验证过的计算指标加人工审阅测量交际投入、情感立场、任务相关指标与安全信号。
评估
通过任务内配对对比与人工审阅,量化语气变化对响应长度、情感框架、任务相关指标及拒答/免责声明/毒性等安全信号的影响,并比较不同领域与模型间的稳定性差异。
原文摘要(Abstract)

Developers increasingly rely on large language models (LLMs) for programming, reasoning, and documentation, yet little is known about whether their behavior remains stable under realistic sociopragmatic variation. We conduct a controlled, within-task perturbation study pairing polite and coercive variants of 60 softwareengineering tasks across four deployed LLM assistants, yielding 480 responses (240 within-task paired comparisons). Using validated computational metrics and targeted human review, we measure communicative effort, affective stance, task-relevant indicators, and safety signals (refusals, disclaimers, toxicity). Polite prompts consistently elicit longer and more positively framed responses (+222 tokens; ≈18%), while coercive tone compresses elaboration without observable degradation in audited task-relevant indicators or safety signals. Tone sensitivity varies by domain and across models, with ethical reasoning exhibiting larger affective shifts and GPT-4o demonstrating the highest behavioral stability. We interpret tone perturbation not as stylistic noise but as a lightweight robustness probe, supporting the inclusion of socio-pragmatic variation as a first-class evaluation dimension for LLM-assisted software engineering.

链接与引用

DOI 原文 · PDF(开放获取) · DBLP

BibTeX
@inproceedings{KassabI26,
  title = {Tone Perturbations as Trust Probes: Evaluating Behavioral Stability of LLMs in Software Engineering Tasks},
  author = {Mohamad Kassab and Youssab Ibrahim},
  booktitle = {FSE},
  year = {2026},
  doi = {10.1145/3803437.3806716}
}