一句话：提出首个 ArkTS/OpenHarmony 仓库级自动修复基准，并评测八个模型，揭示编译通过与行为修复的差距。
现有自动程序修复基准主要覆盖 Python、Java 等成熟语言，对 ArkTS 与 OpenHarmony 支持有限。本文提出 ArkEval，一个面向真实 ArkTS/OpenHarmony issue 解决任务的仓库级可执行修复基准，据称是首个此类基准。作者从九个公开仓库挖掘截至 2025 年 7 月 3 日的 6000 多条 PR/MR 记录，经自动画像与三名 ArkTS/OpenHarmony 开发者一致筛选得到 502 个实例；复现测试通过六轮基准级精化构建，并对剩余六个实例做定向闭环。评估采用相同的定位引导、关闭 RAG 的修复协议测试八个模型。最佳模型的补丁编译率为 54.38%，而最高行为 Pass@1 仅 19.32%，表明 ArkEval 能区分构建有效性与行为修复，并提供可追溯测试。
Automated program repair benchmarks largely cover mature languages such as Python and Java, leaving limited support for ArkTS and OpenHarmony. We present ArkEval, an executable repository-level benchmark for evaluating repair on real ArkTS/OpenHarmony issue-resolution tasks. To the best of our knowledge, it is the first benchmark with this scope. We mined more than 6,000 pull-request and merge-request records available through July 3, 2025, from nine public repositories. Automated profiling and unanimous screening by three ArkTS/OpenHarmony developers produced 502 instances. Their reproduction tests were constructed through a six-round benchmark-wide refinement process, followed by targeted closure for the remaining six instances. We evaluated eight current models with the same localization-guided, RAG-off repair protocol. The best model compiled 54.38% of its patches, whereas the highest behavioral Pass@1 was 19.32%. ArkEval therefore separates build validity from behavioral repair and supplies traceable tests for studying both.
@inproceedings{abs-2602-08866,
title = {ArkEval: Benchmarking and Evaluating Automated Code Repair for ArkTS},
author = {Bang Xie and Senjian Zhang and Zhiyuan Peng and Wei Chen and Xin Yin and Chenhao Ying and Yuan Luo},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837514}
}