一句话：提出 MCTS-REFINE，用带反思与严格拒绝采样的 MCTS 生成高质量 CoT 数据，提升 LLM 仓库 issue 解决能力。
开源大模型在仓库 issue 解决上仍落后于闭源模型，现有 CoT 数据生成方法因拒绝采样弱、步骤验证不足而累积错误。本文提出 MCTS-REFINE：在蒙特卡洛树搜索中引入反思机制，以严格拒绝采样与精炼校验中间步骤；并把任务分解为文件定位、故障定位与补丁生成，要求中间输出与开发者补丁精确匹配，生成高质量 CoT 微调数据。在 SWE-bench Lite/Verified 上，微调后 Qwen2.5-72B-Instruct 解决率达 28.3%/35.0%，超过同规模 SOTA SWE-Fixer-Qwen-72B（24.7%/32.8%）；给定精确定位时 Verified 达 43.8%。
LLMs demonstrate strong performance in automated software engineering, particularly for code generation and issue resolution. While proprietary models like GPT-4o achieve high benchmarks scores on SWE-bench, their API dependence, cost, and privacy concerns limit adoption. Open-source alternatives offer transparency but underperform in complex tasks, especially sub-100B parameter models. Although quality Chain-of-Thought (CoT) data can enhance reasoning, current methods face two critical flaws: (1) weak rejection sampling reduces data quality, and (2) inadequate step validation causes error accumulation. These limitations lead to flawed reasoning chains that impair LLMs’ ability to learn reliable issue resolution.The paper proposes MCTS-REFINE, an enhanced Monte Carlo Tree Search (MCTS)-based algorithm that dynamically validates and optimizes intermediate reasoning steps through a rigorous rejection sampling strategy, generating high-quality CoT data to improve LLM performance in issue resolution tasks. Key innovations include: (1) augmenting MCTS with a reflection mechanism that corrects errors via rejection sampling and refinement, (2) decomposing issue resolution into three subtasks—File Localization, Fault Localization, and Patch Generation—each with clear ground-truth criteria, and (3) enforcing a strict sampling protocol where intermediate outputs must exactly match verified developer patches, ensuring correctness across reasoning paths.Experiments on SWE-bench Lite and SWE-bench Verified demonstrate that LLMs fine-tuned with our CoT dataset achieve substantial improvements over baselines. Notably, Qwen2.5-72B-Instruct achieves 28.3%(Lite) and 35.0%(Verified) resolution rates, surpassing SOTA baseline SWE-Fixer-Qwen-72B with the same parameter scale, which only reached 24.7%(Lite) and 32.8%(Verified). Given precise issue locations as input, our fine-tuned Qwen2.5-72B-Instruct model achieves an impressive issue resolution rate of 43.8%(Verified), comparable to the performance of Deepseek-v3. We open-source our MCTS-REFINE framework, CoT dataset, and fine-tuned models to advance research in AI-driven software engineering.
DOI 原文 ·
@inproceedings{paperbot9202,
title = {MCTS-Refined CoT: High-Quality Fine-Tuning Data for LLM-Based Repository Issue Resolution},
author = {Yibo Wang and Zhihao Peng and Ying Wang and Zhao Wei and Hai Yu and Zhiliang Zhu},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00154}
}