一句话：提出全自动框架，零样本评估 16 个 LLM 在 30 个真实伦理场景中的伦理推理能力。
本文评估大语言模型能否在软件工程自动化伦理画像中充当可审计的伦理推理引擎。作者提出全自动零样本框架，用 30 个真实伦理争议场景测试 16 个 LLM：要求模型识别最适用的伦理理论、判断行为道德可接受性并解释理由，并与专家伦理学者选择进行一致性比较。结果显示，模型平均理论一致性率（TCR）为 73.3%，道德可接受性二元一致率（BAR）为 86.7%，分歧集中于伦理模糊案例；自由文本解释虽词汇多样，但概念趋同明显。研究聚焦画像流程的伦理解释器组件，表明当前 LLM 具备一定解释稳定性与理论一致性推理能力。
Large Language Models (LLMs) are increasingly integrated into software engineering (SE) tools for tasks that extend beyond code synthesis, including judgment under uncertainty and reasoning in ethically significant contexts. We present a fully automated framework for assessing ethical reasoning capabilities across 16 LLMs in a zero-shot setting, using 30 real-world ethically charged scenarios. Each model is prompted to identify the most applicable ethical theory to an action, assess its moral acceptability, and explain the reasoning behind their choice. Responses are compared against expert ethicists’ choices using inter-model agreement metrics. Our results show that LLMs achieve an average Theory Consistency Rate (TCR) of 73.3% and Binary Agreement Rate (BAR) on moral acceptability of 86.7%, with interpretable divergences concentrated in ethically ambiguous cases. A qualitative analysis of free-text explanations reveals strong conceptual convergence across models despite surface-level lexical diversity. These findings support the potential viability of LLMs as ethical inference engines within SE pipelines, enabling scalable, auditable, and adaptive integration of user-aligned ethical reasoning. Our focus is the Ethical Interpreter component of a broader profiling pipeline: we evaluate whether current LLMs exhibit sufficient interpretive stability and theory-consistent reasoning to support automated profiling.
DOI 原文 ·
@inproceedings{paperbot9494,
title = {Advancing Automated Ethical Profiling in SE: a Zero-Shot Evaluation of LLM Reasoning},
author = {Patrizio Migliarini and Mashal Afzal Memon and Marco Autili and Paola Inverardi},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00202}
}