paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

模式凌驾于像素:测量多模态代码生成中的模式补全偏差
ASE Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering2026匹配度 2/10 · 该论文关注多模态代码生成中的视觉模式补全偏差,本质上是对 MLLM 能力的基准评测,未涉及静态分析、形式化验证、浮点程序分析或贝叶斯程序分析,也不属于把 LLM/Agent 作为工具改进程序分析的方向。仅因涉及代码生成与基准构建,与 PL/AI 交叉有边缘联系,匹配度较低。引用 1 inbox
Khai-Nguyen Nguyen, Oscar Chaparro, Antonio Mastropaolo
crossref: 2026-10-09

AI 速览

一句话:首次构建视觉模式补全偏差基准,揭示多模态代码生成模型强烈偏向重复 UI 模式而忽略局部异常。

多模态大语言模型常用于从网页截图生成前端代码,但重复 UI 模式可能使其输出视觉错误却模式一致的结果。本文在 screenshot-to-code 填空任务上测量该偏差,并构建首个视觉模式补全偏差基准:扰动重复模式中的一个局部元素,要求模型从截图和 HTML 上下文恢复被掩码的 width 或 font-size。基于 Design2Code 的 30 个网页,生成 1,440 个截图,覆盖卡片与文本样式及噪声条件。评估五个前沿 MLLM 显示,所有模型均强烈偏向重复基线:卡片宽度与文本字号的偏差率分别达 69.78% 和 80.22%,平均准确率仅 21.17% 和 7.89%。噪声、细微扰动和边界位置加剧偏差;推理努力越高偏差越低,但模型仍可能识别异常后选择模式一致答案。该工作揭示了多模态代码生成的具体失败模式,其严重程度与视觉显著性密切相关。

问题
多模态大语言模型在将网页截图转为前端代码时,是否会因重复 UI 模式而偏向视觉错误但模式一致的输出。
方法
构建首个视觉模式补全偏差基准:在重复 UI 模式中扰动一个局部元素,要求模型从截图与 HTML 上下文恢复被掩码的 width 或 font-size;基于 Design2Code 的 30 个网页生成 1,440 个截图,并评估五个前沿 MLLM。
评估
测量标准与噪声条件下的偏差率和准确率,并分析扰动强度、边界位置与推理努力;发现所有模型强烈偏向重复基线,且偏差严重程度与视觉显著性相关。
原文摘要(Abstract)

Multimodal large language models (MLLMs) are increasingly used to translate webpage screenshots into front-end code, but repeated UI patterns may sway them toward visually incorrect yet pattern-consistent outputs. In this work, we test how repeated webpage patterns hurt MLLM accuracy on an objective screenshot-to-code fill-in-the-blank task. We introduce the first benchmark for visual pattern-completion bias, where one localized element in a repeated UI pattern is perturbed and the model must recover the masked width or font-size value from the screenshot and HTML context. Starting from 30 webpages curated from the Design2Code dataset, we build 1,440 evaluated screenshots spanning structural card and text-style patterns under standard and noise-overlaid conditions. We evaluate five frontier MLLMs and find that all are strongly biased toward the repeated baseline. Mean bias rate reaches 69.78% on card-width perturbations and 80.22% on text font-size perturbations, while mean accuracy is only 21.17% and 7.89%, respectively. performs best but still drops from 68.61% accuracy on cards to 13.89% on text, while reaches 96.11% bias on text. Noise, subtler perturbations, and boundary positions further increase bias rate. Reasoning analysis further shows that greater reasoning effort correlates with lower bias, yet qualitative evidence reveals that models can identify the anomalous element and still override it with the pattern-consistent answer. Our results identify a concrete failure mode in multimodal code generation and show that its severity is strongly associated with visual saliency.

链接与引用

DOI 原文 · arXiv ·

BibTeX
@inproceedings{paperbot12308,
  title = {Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation},
  author = {Khai-Nguyen Nguyen and Oscar Chaparro and Antonio Mastropaolo},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3834443}
}