一句话：评估九种 LLM 在 62 个教学视频转录摘要上的表现，揭示提示设计与模型选择的影响。
本研究探讨大语言模型（LLM）在总结教学视频转录上的有效性。作者使用简单基础提示与增强结构化提示，对九个 LLM 在 62 个教学视频上生成摘要，并由 gpt-4o-mini 和 gemini-1.5-flash 两个评估模型，按结构、示例、来源、相关性、连贯性、叙述和准确性七项标准评分。结果显示模型间差异明显，Mistral Large 与 Claude 3.5 Sonnet 表现最佳，尤其在增强提示下；但增强提示有时会提升叙述质量却牺牲结构清晰度。研究还观察到评估者偏差，gpt-4o-mini 比 gemini-1.5-flash 给分更高，说明多评估者必要。结论强调提示设计、模型选择与评估标准化在教育 LLM 应用中的重要性。
This study explores the effectiveness of large language models (LLMs) in summarizing instructional video transcriptions, a key application in educational technology. We assessed nine LLMs using two prompts—a simple base prompt and an enhanced, structured prompt—across 62 instructional videos. Two evaluating models, gpt-4o-mini and gemini-1.5-flash, scored the summaries based on seven criteria tailored to instructional content: overall structure, presence of examples, availability of sources, relevance, coherence, narration, and ACCURACY. Results showed notable performance differences, with models like Mistral Large and Claude 3.5 Sonnet performing best, especially with the enhanced prompt. However, the enhanced prompt improved narrative quality at the expense of structural clarity in some cases. Evaluator bias was also observed, with gpt-4o-mini assigning higher scores than gemini-1.5-flash, highlighting the need for multiple evaluators. These findings underscore the role of prompt design and model choice in educational LLM applications and suggest future research into optimizing prompts and standardizing evaluation methods.