一句话：MERA 用三个轻量专家分别建模用户意图、系统上下文与历史先例，融合证据实现可审计的 Beta 工单严重性自动分级。
移动 OS Beta 测试产生海量用户工单，严重性分级成为分诊瓶颈。与遥测丰富的服务端日志不同，这些工单只有模糊的症状式描述，且频繁特性更新引发概念漂移，使深度学习与预训练模型分类器难以适应，黑盒特性也缺乏推理透明度。受资深工程师“用户意图—系统上下文—历史先例”三角验证实践启发，MERA 将严重性评估解耦为三个专家视角，把大教师模型推理能力蒸馏到三个轻量专家，并由动态融合层加权证据，输出可审计、可人工审阅理由的严重性决策。系统已在某全球领先 ICT 厂商的移动 OS Beta 流水线中作为默认评分器部署，五个月处理超 37 万条工单，准确率 85%，概念漂移下显著优于基线，人工分诊工作量减少 60%。
Large-scale mobile OS beta testing generates massive ticket vol- umes, creating a triage bottleneck in which assigning severity dic- tates engineering priority. Automating this is difficult: unlike server logs with rich telemetry, beta tickets rely on vague, symptom-driven user descriptions, which confound standard AIOps tools. Tradi- tional classifiers based on deep learning or pre-trained language models (PLMs) further struggle to adapt to the rapid concept drift triggered by frequent feature updates, and their black-box nature fails to provide the reasoning transparency that engineers require for trust. We observe that experienced triage engineers resolve these challenges by triangulating across user intent, system context, and historical precedent—a discipline that no single monolithic model captures. We present MERA (Multi-expert Evidence Reasoning and Arbitration), the first production-deployed multi-expert LLM framework for severity assessment in telemetry-void mobile OS beta testing. MERA operationalizes this discipline by decoupling severity assessment into three expert perspectives and fusing their evidence into an auditable decision. To meet strict online latency budgets, MERA distills the reasoning capability of a large teacher model into three lightweight experts, and a dynamic fusion layer weighs their evidence to produce a severity decision with human- reviewable rationales. In the mobile OS beta-testing pipeline of a leading global information and communications technology (ICT) provider, MERA serves as the default severity grader for all incom- ing beta tickets, with engineers performing spot-checks. Over five months it processed more than 370,000 tickets at 85% accuracy, sub- stantially outperforming baselines under concept drift and reducing manual triage effort by 60%.
@inproceedings{paperbot12216,
title = {MERA: A Multi-expert LLM Framework for Severity Assessment of User Tickets in Mobile OS Beta Testing},
author = {Yongqian Sun and Yimin Zuo and Qingliang Zhang and Bowen Hao and Wenwei Gu and Jiaqi Luan and Sixu Zhou and Shenglin Zhang and Dan Pei},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834518}
}