一句话：提出 LLMorph，基于蜕变测试自动检测 LLM 在 NLP 任务中的不一致与错误行为，无需人工标注。
大语言模型（LLM）在 NLP 任务中表现优异，但其输出正确性缺乏自动化预言，导致可靠性评估依赖昂贵的人工标注。本文提出 LLMorph，一个面向 LLM NLP 任务的自动化蜕变测试工具。它利用蜕变测试（MT）和蜕变关系（MR），从源测试输入生成后续输入，并通过检查模型在相关输入上的输出一致性来发现错误，无需人工标注数据。LLMorph 设计上易于扩展到任意 LLM、NLP 任务与 MR 集合。作者详细介绍了其设计、实现和使用方式。评估中，LLMorph 在四个 NLP 基准上应用 36 条 MR，测试 GPT-4、LLAMA3 和 HERMES 2 三个先进模型，产生超过 561,000 次测试执行，结果表明其能大规模自动暴露不正确模型行为。工具源码与演示已公开。
Automated testing is essential for evaluating and improving the reliability of Large Language Models (LLMs), yet the lack of automated oracles for verifying output correctness remains a key challenge. We present LLMorph, an automated testing tool specifically designed for LLMs performing NLP tasks, which leverages Metamorphic Testing (MT) to uncover faulty behaviors without relying on human-labeled data. MT uses Metamorphic Relations (MRs) to generate follow-up inputs from source test input, enabling detection of inconsistencies in model outputs without the need of expensive labelled data. LLMorph is aimed at researchers and developers who want to evaluate the robustness of LLM-based NLP systems. In this paper, we detail the design, implementation, and practical usage of LLMorph, demonstrating how it can be easily extended to any LLM, NLP task, and set of MRs. In our evaluation, we applied 36 MRs across four NLP benchmarks, testing three state-of-the-art LLMs: GPT-4, LLAMA3, and HERMES 2. This produced over 561,000 test executions. The results demonstrate LLMorph’s effectiveness in automatically exposing incorrect model behaviors at scale.The tool source code is available at https://github.com/steven-b-cho/llmorph. A screencast demo is available at https://youtu.be/sHmqdieCfw4.