一句话：提出首个交互式网页代码生成基准 Interaction2Code，评估 MLLM 并给出四类增强策略。
多模态大语言模型在 design-to-code 任务上表现突出，但现有基准仅含静态网页，忽略动态交互，限制了生成网页的实用性。本文首次系统研究 MLLM 生成交互式网页，形式化 Interaction-to-Code 任务并构建 Interaction2Code 基准，覆盖 127 个网页、374 个交互、15 种网页类型和 31 类交互。作者用自动指标与人工评估测试 SOTA MLLM，发现四类关键局限：交互生成不足、易出现十种失败、视觉细微交互表现差、单模态视觉描述下理解不足；进而提出交互元素高亮、失败感知提示、视觉显著性增强、视觉-文本描述组合四种增强策略。
Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance on the design-to-code task, i.e., generating UI code from UI mock-ups. However, existing benchmarks only contain static web pages for evaluation and ignore the dynamic interaction, limiting the practicality, usability and user engagement of the generated webpages.To bridge these gaps, we present the first systematic investigation of MLLMs in generating interactive webpages. Specifically, we formulate the Interaction-to-Code task and establish the Interaction2Code benchmark, encompassing 127 unique webpages and 374 distinct interactions across 15 webpage types and 31 interaction categories. Through comprehensive experiments utilizing state-of-the-art (SOTA) MLLMs, evaluated via both automatic metrics and human assessments, we identify four critical limitations of MLLM on Interaction-to-Code task: (1) inadequate generation of interaction compared with full page, (2) prone to ten types of failure, (3) poor performance on visually subtle interactions, and (4) insufficient undestanding on interaction when limited to single-modality visual descriptions. To address these limitations, we propose four enhancement strategies: Interactive Element Highlighting, Failure-aware Prompting (FAP), Visual Saliency Enhancement, and Visual-Textual Descriptions Combination, all aiming at improving MLLMs’ performance on the Interaction-to-Code task. Our data and code are available in https://github.com/WebPAI/Interaction2Code.
@inproceedings{paperbot9463,
title = {Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping},
author = {Jingyu Xiao and Yuxuan Wan and Yintong Huo and Zixin Wang and Xinyi Xu and Wenxuan Wang and Zhiyao Xu and Yuhang Wang and Michael R. Lyu},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00028}
}