一句话：提出轻量级灰盒方法 ToxicDetector，利用毒性概念提示的嵌入向量与 MLP 分类器高效检测 LLM 有害提示。
大语言模型（如 ChatGPT、Gemini）在聊天机器人与自动内容生成等应用中表现突出，但恶意用户可通过精心构造的有害提示并借助越狱技术绕过安全机制，诱导模型输出有害或不道德内容。现有黑盒与白盒检测方法在提示多样性、可扩展性和计算效率方面均面临挑战。本文提出 ToxicDetector，一种轻量级灰盒检测方法：它借助 LLM 生成“毒性概念提示”，将提示表示为嵌入向量构成的特性向量，并用多层感知机（MLP）分类器完成提示分类。作者在多个 LLama 版本、Gemma-2 以及多个数据集上进行了评估，结果显示 ToxicDetector 达到 96.39% 的准确率与 2.00% 的低假阳性率，优于当前最先进方法；单条提示处理时间仅 0.0780 秒，适合实时应用，兼具高准确率、高效率与可扩展性。
Large language models (LLMs) like ChatGPT and Gemini have significantly advanced natural language processing, enabling various applications such as chatbots and automated content generation. However, these models can be exploited by malicious individuals who craft toxic prompts to elicit harmful or unethical responses. These individuals often employ jailbreaking techniques to bypass safety mechanisms, highlighting the need for robust toxic prompt detection methods. Existing detection techniques, both blackbox and whitebox, face challenges related to the diversity of toxic prompts, scalability, and computational efficiency. In response, we propose ToxicDetector, a lightweight greybox method designed to efficiently detect toxic prompts in LLMs. ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP) classifier for prompt classification. Our evaluation on various versions of the LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector achieves a high accuracy of 96.39% and a low false positive rate of 2.00%, outperforming state-of-the-art methods. Additionally, ToxicDetector's processing time of 0.0780 seconds per prompt makes it highly suitable for real-time applications. ToxicDetector achieves high accuracy, efficiency, and scalability, making it a practical method for toxic prompt detection in LLMs.
@inproceedings{paperbot8357,
title = {Efficient Detection of Toxic Prompts in Large Language Models},
author = {Yi Liu and Junzhe Yu and Huijia Sun and Ling Shi and Gelei Deng and Yuqi Chen and Yang Liu},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695018}
}