一句话：提出角色化范式 EmbedAgent 与基准 EmbedBench，系统评测 LLM 在嵌入式编程、电路设计与跨平台迁移中的能力并提出改进策略。
针对大语言模型在嵌入式系统开发中缺乏系统评估的问题，本文提出 EmbedAgent 范式，模拟嵌入式系统程序员、架构师与集成者等真实角色，使 LLM 能在连接数字与物理系统的任务上接受测试；并构建首个覆盖嵌入式编程、电路设计与跨平台迁移的综合基准 EmbedBench，含 126 个用例、9 类电子元件与 3 个硬件平台。作者对 10 个主流 LLM 进行实验，发现即使任务简单，DeepSeek-R1 在给定原理图时 pass@1 仅为 55.6%，自行生成原理图时为 50.0%；跨平台迁移中 MicroPython/Raspberry Pi Pico 表现较好（最高 73.8%），ESP-IDF 上最佳模型仅 29.4%。分析表明通用聊天模型难以调用领域预训练知识，而推理模型易过度思考。基于此提出检索增强生成与编译器反馈两种策略，使 DeepSeek-R1 分别提升至 65.1% 与 53.1%，Arduino 到 ESP32 迁移准确率从 21.4% 提升至 27.8%。
Large Language Models (LLMs) have shown promise in various tasks, yet few benchmarks assess their capabilities in embedded system development. In this paper, we introduce EmbedAgent, a paradigm designed to simulate real-world roles in embedded system development, such as Embedded System Programmer, Architect, and Integrator. This paradigm enables LLMs to be tested in tasks that bridge the gap between digital and physical systems, allowing for a more comprehensive assessment of their capabilities. To evaluate LLMs on these tasks, we propose EmbedBench, the first comprehensive benchmark for embedded system programming, circuit design, and cross-platform migration. EmbedBench consists of 126 cases, covering 9 electronic components across 3 hardware platforms. Through extensive experiments on 10 mainstream LLMs, we uncover several key findings. Surprisingly, despite the simplicity of the cases, DeepSeek-R1 achieves only a 55.6% pass@1 rate when provided with schematic information, and 50.0% when tasked with generating the schematics itself. In the cross-platform migration tasks, LLMs show relatively strong performance with MicroPython on the Raspberry Pi Pico (with the top model achieving 73.8% pass@1), but perform poorly on ESP-IDF, where the best model reaches only 29.4% pass@1. Interestingly, we observe that general-purpose chat LLMs like DeepSeek-V3 often fail to utilize relevant pre-trained knowledge in this domain, while reasoning LLMs tend to overthink and overlook efficient knowledge during pretraining. Based on these insights, we propose two strategies—retrieval augmented generation and compiler feedback-to enhance LLM performance. These strategies result in significant improvements, with Deepseek-R1 reaching a 65.1% pass@1 with correct schematics, and 53.1% without. Additionally, the accuracy of the Arduino to ESP32 migration task improves from 21.4% to 27.8%.
@inproceedings{abs-2506-11003,
title = {EmbedAgent: Benchmarking Large Language Models in Embedded System Development},
author = {Ruiyang Xu and Jialun Cao and Mingyuan Wu and Wenliang Zhong and Yaojie Lu and Ben He and Xianpei Han and Shing-Chi Cheung and Le Sun},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3787788}
}