一句话：提出 GUIRepair，用 Image2Code 与 Code2Image 进行跨模态推理，提升多模态 GUI 软件问题修复。
现有基于 LLM 的自动程序修复系统主要在单模态设置下评估，仅依赖文本问题描述与源码，难以理解和利用 GUI 中的视觉信息，因而在多模态问题场景（如 SWE-bench M）中表现受限。本文提出 GUIRepair，一种面向多模态问题场景的跨模态推理修复方法，包含 Image2Code 与 Code2Image 两个组件。Image2Code 根据 issue 报告提取相关项目文档，并利用领域知识生成可复现视觉症状的代码，将 GUI 图像转化为可执行上下文以增强故障理解；Code2Image 使用复现代码重放视觉问题，并捕获补丁程序的 GUI 渲染，判断修复是否在视觉上解决问题，从而为补丁验证提供反馈。在 SWE-bench M 上，GUIRepair 以 GPT-4o 为基座解决 157 个实例，超过最佳开源基线 26 个；以 o4-mini 为基座解决 175 个实例，超过最强商业系统 22 个，验证了利用视觉信息进行跨模态推理的有效性。
Large language model (LLM)-based automated program repair (APR) techniques have shown promising results in resolving real-world github issue tasks. Existing APR systems are primarily evaluated in unimodal settings (e.g., SWE-bench), relying solely on textual issue descriptions and source code. However, these autonomous systems struggle to resolve multimodal problem scenarios (e.g., SWE-bench M) due to limitations in interpreting and leveraging visual information. In multimodal scenarios, LLMs need to rely on visual information in the graphical user interface (GUI) to understand bugs and generate fixes. To bridge this gap, we propose GUIRepair, a cross-modal reasoning approach for resolving multimodal issue scenarios by understanding and capturing visual information. Specifically, GUIRepair integrates two key components including Image2Code and Code2Image to enhance fault comprehension and patch validation. Image2Code extracts relevant project documents based on the issue report, then applies this domain knowledge to generate the reproduced code responsible for the visual symptoms, effectively translating GUI images into executable context for better fault comprehension. Code2Image replays the visual issue scenario using the reproduced code and captures GUI renderings of the patched program to assess whether the fix visually resolves the issue, providing feedback for patch validation. We evaluate GUIRepair on SWE-bench M, and the approach demonstrates significant effectiveness. When utilizing GPT-4o as the base model, GUIRepair solves 157 instances, outperforming the best open-source baseline by 26 instances. Furthermore, when using o4-mini as the base model, GUIRepair can achieve even better results and solve 175 instances, outperforming the top commercial system by 22 instances. This emphasizes the success of our new perspective on incorporating cross-modal reasoning by understanding and capturing visual information to resolve multimodal issues.
@inproceedings{paperbot9332,
title = {Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair},
author = {Kai Huang and Jian Zhang and Xiaofei Xie and Chunyang Chen},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00100}
}