一句话：提出一项被试间研究，考察学生在真实软件工程任务中使用多模态生成式AI的交互模式、过程与成效。
多模态生成式AI将学生支持从文本提示扩展到音频对话、媒体上传和实时屏幕共享，但其在软件开发教育中的影响尚不清楚。本文提出一项被试间研究，计划招募至少25名参与者进行现场出声思维实验，使用Google AI Studio在不同交互模式下完成开源工作流中的缺陷修复、带拉取请求的功能实现和测试编写任务。研究将三角互证屏幕与音频记录、交互轨迹以及前后测问卷，刻画模态选择与提示策略、使用模式，并评估其对任务效率和解法准确性的影响。目标是帮助计算机/软件工程教育者和工具设计者判断多模态GenAI支持何时有益、何时增加负担，以及如何搭建支架以促进稳健学习和实践技能发展。
Recent advances in multimodal generative AI (GenAI) extend student support beyond text-focused prompting to include audio dialogue, media upload, and real-time stream (i.e., live screen-sharing). Yet, the educational implications of these multimodal capabilities in software engineering (SE) development settings remain under-explored. This extended abstract presents a proposed between-subjects study to investigate how students leverage multimodal GenAI while completing realistic SE tasks in an open-source software (OSS) workflow and how multimodal interaction shapes task-solving processes and outcomes. We will conduct in-person think-aloud sessions (min. n = 25) in which participants use a multi-modal GenAI tool (Google AI Studio) across interaction modes while working through bug fixing, feature implementation with pull request contribution, and test writing tasks. We will triangulate screen/audio recordings, interaction traces, and pre/post surveys to characterize modality selection and prompting strategies, assess participants' usage patterns, and explore impacts on efficiency and solution accuracy. Essentially, the study aims to provide insights for CS/SE educators and tool designers on when multimodal GenAI support is beneficial, when it adds overhead, and how to scaffold its use to promote robust learning and practical skill development.
@inproceedings{LiangB26,
title = {Beyond Text: Understanding How Multimodal Generative AI Impacts Students Learning Software Development (Extended Abstract)},
author = {Huayu Liang and Chris Brown},
booktitle = {FSE},
year = {2026},
doi = {10.1145/3803437.3805792}
}