一句话：通过设计探针研究 Azure 团队差异测试实践，发现 LLM 可辅助差异标注与摘要，但工程师要求预测附带显式理由。
差异测试被许多团队用作发布质量门禁，但将生产与测试系统的行为差异人工标注为回归、预期变更或偶然变更（如非确定性、时序导致）十分费力，需要审阅日志、拉取请求与团队讨论等大量文本资料。本文与 Azure 中央服务开发团队合作，通过设计探针方法了解其差异测试实践，并让工程师体验多种 LLM 辅助方案：自动预测差异标签、为各类工件与讨论生成摘要。发布工程师反馈，预测标签可显著节省精力，但他们希望模型给出显式理由以建立信任；生成的摘要信息丰富，但略显冗长。
Many software development teams use differential testing as a quality gate in their release process. Differential testing—namely, comparing behavioral differences between a system in production and a system in test—is a laborious process to label changes as regressions, expected changes, or incidental changes (e.g. those due to nondeterminism or timing). This manual process involves inspecting large textual artifacts, like logs, pull requests, and team discussions, which suggests that Large Language Models (LLMs) could be helpful. In this paper, we engage with the team developing a central Azure service to understand their work practice for differential testing. We used a design probe method to elicit feedback about several ways to use LLMs to improve their work practice, including automatically labeling behavior differences and providing summaries of various artifacts and discussions. Release engineers on the team report that predicting a difference's label would save them effort, but they want an explicit rationale to improve their trust in the prediction; they found the generated summaries to be informative, if a bit wordy.
@inproceedings{paperbot9099,
title = {Using Large Language Models to Support the Workflow of Differential Testing},
author = {Arun Krishna Vajjala and Ajay Krishna Vajjala and Carmen Badea and Christian Bird and Jade D'Souza and Robert DeLine and Mikhail O Demyanyuk and Jason Entenmann and Nicole Forsgren and Aliaksandr Hramadski and Haris Mohammad and Sandeepan Sanyal and Oleg Surmachev and Thomas Zimmermann},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728559}
}