paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Technical Briefing on Beyond A/B Testing - Orthogonal and Experimental Designs

超越 A/B 测试：正交与实验设计的技术简报
FSE Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering2026匹配度 1/10 · 该论文关注软件工程中的组合实验设计（正交/交叉设计）与受试者稀缺问题，属于实证研究方法与实验设计，与用户核心方向（程序分析、LLM/Agent 用于静态分析与形式化验证、浮点程序分析、贝叶斯程序分析）没有直接交集；摘要中仅将 LLM 作为实验对象/算力场景提及，并非研究 LLM 与程序分析的交叉。引用 0 inbox
Fabio Massacci, Winnie Bahati Mbaka, Katja Tuma
编程环境与开发者工具
crossref: 2026-09-12crossref: 2026-09-12dblp: 2026-09-12

AI 速览

一句话：介绍多因素与受试者稀缺场景下，软件工程组合实验的正交与交叉设计方法。

该技术简报面向 FSE 参会者，讨论当实验因素远多于 A/B 两种条件，或可用的开发者、用户或 LLM 受试者数量有限时，如何超越传统 A/B 测试。简报基于研究团队在软件工程组合实验方面的经验，结合大量实例和交互式练习，帮助参会者掌握设计、执行和分析自身实验研究的方法。其科学基础部分来自 JSS'24 论文《Addressing Combinatorial Experiments and Scarcity of Subjects by Provably Orthogonal and Crossover Experimental Designs》，核心思想是利用可证明正交设计和交叉设计来应对组合实验与受试者稀缺问题。简报以教学和动手实践为主，不报告新的实证评估结果。

问题
当实验设计旋钮远多于 A/B 两个条件，或只能使用少量开发者、用户或 LLM 作为受试者时，如何有效设计软件工程实验。
方法
以技术简报和交互式动手练习介绍组合实验设计，重点基于可证明正交与交叉实验设计，并结合研究团队经验与大量实例进行教学。
评估
摘要未报告新的实证评估；该简报以设计、执行和分析实验研究的教学效果为目标，科学依据部分来自 JSS'24 论文。
原文摘要（Abstract）

Designing A/B testing experiments is relatively easy if you have plenty of subjects or hardware to run LLMs. What if you have many more design knobs than just two conditions, or can only ask relatively few developers/users/LLMs to test your solutions? This technical briefing will help FSE attendees gain a hands-on understanding of designing combinatorial experiments in Software Engineering. The activity is grounded in the research group's significant experience and will draw on many examples. It includes interactive hands-on exercises with the learning objectives of designing, conducting, and analyzing one's own experimental study. The scientific underpinnings are partly available in the paper "Addressing Combinatorial Experiments and Scarcity of Subjects by Provably Orthogonal and Crossover Experimental Designs"(JSS'24)

链接与引用

DOI 原文 · PDF（开放获取） · DBLP

BibTeX 
@inproceedings{MassacciMT26,
  title = {Technical Briefing on Beyond A/B Testing - Orthogonal and Experimental Designs},
  author = {Fabio Massacci and Winnie Bahati Mbaka and Katja Tuma},
  booktitle = {FSE},
  year = {2026},
  doi = {10.1145/3803437.3804905}
}