一句话：首次系统研究 LLM-as-a-Judge 用于人向二进制逆向工程评估，提出专家标注基准 BinJudgeBench 与自适应路由评测器 BinJudge。
人向二进制逆向工程（HOBRE）旨在将反编译伪代码转化为更友好的表示，但可靠评估其输出仍是难题：人工评估昂贵且难扩展，已有自动指标或依赖可执行测试与运行环境，或依赖难以获取的源参考，且无法捕捉语义等价但表述多样的输出。本文首次系统研究将 LLM-as-a-Judge 范式用于 HOBRE，覆盖函数名恢复、二进制代码摘要和反编译优化三个代表性任务，并构建首个基于多维度人工判断的专家标注、无参考评估基准 BinJudgeBench。实验表明，LLM-as-a-Judge 与人工判断的平均相关性达 63.20%，显著优于传统自动指标的 35.04%。通过分析不同评判配置（基座 LLM、提示策略、解码温度）对相关性与成本的影响，作者发现不存在通用最优配置。为此提出 BinJudge，以轻量路由机制为每个任务和样本自适应选择最优评判配置，将相关性提升 4.5%–24.7%，并将 API 成本降至静态最优配置的 0.06×–0.84×，为 HOBRE 提供可扩展、低成本且高保真的自动评估方案。
Human-Oriented Binary Reverse Engineering (HOBRE) aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. However, reliably evaluating HOBRE outputs remains a fundamental challenge. On one hand, human evaluation is costly, time-consuming, as well as difficult to scale. On the other hand, existing automated metrics either rely on executable test cases and runtime environments, which are often unavailable for real-world binaries, or depend on high-quality source code references that are typically inaccessible and fail to capture semantically equivalent but lexically diverse outputs. Although the emerging LLM-as-a-Judge paradigm is naturally well-suited to HOBRE evaluation, its effectiveness has not yet been fully studied. This paper presents the first systematic investigation of the LLM-as-a-Judge paradigm for HOBRE, covering three representative tasks: function name recovery, binary code summarization, and decompilation optimization. We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based on multi-dimensional human judgment. Our empirical study reveals that LLM-as-a-Judge achieves an average correlation of 63.20% with human judgment, significantly outperforming traditional automated metrics at 35.04%. By analyzing the impact of various judge configurations, including backbone LLMs, prompting strategies, and decoding temperatures, on both correlation and cost, we find that no “one-size-fits-all” configuration exists, as the optimal setup varies across tasks and individual samples. To address this, we propose BinJudge, which employs a lightweight routing mechanism to adaptively select the optimal judge configuration for each specific task and sample. BinJudge improves correlation with human experts by 4.5%–24.7% and reduces API cost to 0.06×–0.84× of that of static best configurations, providing a scalable, cost-effective, and high-fidelity automated evaluation scheme for HOBRE.
@inproceedings{paperbot12570,
title = {Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering},
author = {Xiuwei Shang and Li Hu and Xiao Jiang and Jieke Shi and Junda He and Zhou Yang and Shaoyin Cheng and Guoqiang Chen and Weiming Zhang and David Lo},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834339}
}