一句话：提出 ASTRAL，一个自动生成并执行测试提示、以评估大语言模型安全性的工具。
本文介绍 ASTRAL，一个用于自动化生成与执行测试输入（即提示）以评估大语言模型（LLM）安全性的工具。ASTRAL 由三个微服务模块构成：其一是测试生成器，采用一种新颖的黑盒覆盖准则，在广泛的安全类别与语言特征（如不同写作风格与说服技巧）上生成均衡且多样的不安全测试输入，并引入基于 LLM 的方法，结合检索增强生成（RAG）、few-shot 提示策略与网页浏览来产出最新的测试输入；其二是测试执行器，负责在待测 LLM 上运行生成的提示；其三是测试评估器，充当 oracle 判定执行输出中是否存在不安全回复，从而支持端到端的全自动 LLM 测试流程。该工作以工具形式呈现，重点在于三模块的架构设计与安全测试流程的自动化，摘要未报告定量实验结果。
In this paper, we present ASTRAL, a tool that automates the generation and execution of test inputs (i.e., prompts) to evaluate the safety of Large Language Models (LLMs). ASTRAL consists of three microservice modules. The first is a test generator, which employs a novel black-box coverage criterion to create balanced and diverse unsafe test inputs across a wide range of safety categories and linguistic characteristics (e.g., different writing styles and persuasion techniques). Additionally, the test generator incorporates an LLM-based approach that leverages Retrieval-Augmented Generation (RAG), few-shot prompting strategies, and web browsing to produce up-to-date test inputs. The second module is the test executor, which runs the generated test inputs on the LLM under test. Finally, the test evaluator acts an oracle to assess the execution outputs to identify unsafe responses, enabling a fully automated LLM testing process.
DOI 原文 ·
@inproceedings{paperbot9608,
title = {ASTRAL: A Tool for the Automated Safety Testing of Large Language Models},
author = {Miriam Ugarte and Pablo Valle and José Antonio Parejo and Sergio Segura and Aitor Arrieta},
booktitle = {ISSTA},
year = {2025},
doi = {10.1145/3713081.3731733}
}