一句话：在工业项目中评估16个LLM生成教育多选题，发现近期LLM优于微调专用模型，并指出高级提示策略的开放挑战。
本文针对LLM集成到教育产品中生成考试题的风险，在一个工业项目中研究LLM生成教育多选题的有效性。实验涵盖16个商用与开源LLM，采用F1和BLEU评估准确性，用困惑度和多样性评估语言质量，并与5个专用微调模型比较。结果表明，近期LLM在问题生成上可超越微调模型，开源LLM与商用LLM竞争力相当，其中Meta Llama表现最佳，DeepSeek与近期GPT-4模型相当。基于这些实证证据，作者建议进一步探索高级提示策略，并报告了短期内计划解决的开放挑战。
Large Language Models (LLMs) have revolutionized the way natural language tasks are handled, with big potential applications in the context of education. LLMs can save educators time and effort, for instance, in content creation and exam generation. Although promising, LLMs’ integration into educational products brings some risks that companies must mitigate.In the context of an industrial project, we investigate the effectiveness of LLMs to generate educational multiple-choice questions. The experiments include 16 commercial and open-source LLMs, rely on standard metrics to assess the accuracy (F1 and BLEU) and linguistic quality (perplexity and diversity) of the generated questions, and compare with five specialized models. The results suggest that recent LLMs can outperform the fine-tuned models for question generation, open-source LLMs are very competitive with the commercial ones, with Meta Llama models being the best performing, and DeepSeek as performing as recent GPT4 models.This promising empirical evidence encourages us to focus on advanced prompting strategies, for which we report relevant open challenges we aim to address in the short term.
DOI 原文 ·
@inproceedings{paperbot9337,
title = {Towards Reliable LLM-based Exam Generation Lessons Learned and Open Challenges in an Industrial Project},
author = {Renzo Degiovanni and Jordi Cabot},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00313}
}