一句话：提出 GUI-ReRank 框架，结合嵌入检索与多模态 LLM 重排序，显著提升 GUI 检索准确率与泛化性。
GUI 原型构建在现代交互系统开发中至关重要，但过程耗时且依赖大量专业知识。已有的自然语言 GUI 检索方法多采用嵌入检索或面向特定仓库的排序模型，存在检索性能有限、跨数据集泛化能力差的问题。本文提出 GUI-ReRank，将基于嵌入的快速约束检索与高效的多模态大语言模型（M）LLM 重排序相结合，并提供可完全定制的 GUI 仓库标注与嵌入流水线，使用户能便捷地将自有仓库变为可检索资源，以支持灵感发现或集成到基于 LLM 的检索增强生成（RAG）工作流。在既有自然语言 GUI 检索基准上的评估表明，GUI-ReRank 在检索准确率和泛化能力上均显著优于当前最优的定制化 Learning-to-Rank 模型。此外，作者还对 MLLM 重排序进行了全面的成本与效率分析，揭示了检索效果与计算资源之间的权衡。
Graphical User Interface (GUI) prototyping is a fundamental component in the development of modern interactive systems, which are now ubiquitous across diverse application domains. GUI prototypes play a critical role in requirements elicitation by enabling stakeholders to visualize, assess, and refine system concepts collaboratively. Moreover, prototypes serve as effective tools for early testing, iterative evaluation, and validation of design ideas with both end users and development teams. Despite these advantages, the process of constructing GUI prototypes remains resource-intensive and time-consuming, frequently demanding substantial effort and expertise. Recent research has sought to alleviate this burden through natural language (NL)-based GUI retrieval approaches, which typically rely on embedding-based retrieval or tailored ranking models for specific GUI repositories. However, these methods often suffer from limited retrieval performance and struggle to generalize across arbitrary GUI datasets. In this work, we present GUI-ReRank, a novel framework that integrates rapid embedding-based constrained retrieval models with highly effective multi-modal (M)LLM-based reranking techniques. GUI-ReRank further introduces a fully customizable GUI repository annotation and embedding pipeline, enabling users to effortlessly make their own GUI repositories searchable, which allows for rapid discovery of relevant GUIs for inspiration or seamless integration into customized LLM-based retrieval-augmented generation (RAG) workflows. We evaluated our approach on an established NL-based GUI retrieval benchmark, demonstrating that GUI-ReRank significantly outperforms state-of-the-art (SOTA) tailored Learning-to-Rank (LTR) models in both retrieval accuracy and generalizability. Additionally, we conducted a comprehensive cost and efficiency analysis of employing MLLMs for reranking, providing valuable insights regarding the trade-offs between retrieval effectiveness and computational resources. Video presentation of GUI-ReRank available at: https://youtu.be/7x9UCh82ug
DOI 原文 ·
@inproceedings{paperbot9488,
title = {GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking},
author = {Kristian Kolthoff and Felix Kretzer and Alexander Maedche and Simone Paolo Ponzetto and Christian Bartelt},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00373}
}