一句话：通过问题变形与迭代提示检测 ChatGPT 回答中的错误，在库选择基准上 F1 达 0.74—0.75。
本文关注软件工程从业者使用 ChatGPT 等生成式 AI 聊天机器人时对其回答真实性的担忧。作者先对 135 名从业者进行调研，发现他们希望用 ChatGPT 完成软件库选择等任务，但常担心答案不正确。为此提出 CID（ChatGPT Incorrectness Detector）及配套技术：通过迭代提示，向 ChatGPT 提出上下文相似但文本表述不同的问题，并利用文本中的蜕变关系生成问题变体；其核心假设是，若某回答与同一问题多个变体所得回答差异显著，则该回答很可能不正确。在软件库选择基准评测中，CID 检测 ChatGPT 错误回答的 F1 分数为 0.74—0.75。
We conducted a survey of 135 software engineering (SE) practitioners to understand how they use Generative AI-based chatbots like ChatGPT for SE tasks. We find that they want to use ChatGPT for SE tasks like software library selection but often worry about the truthfulness of ChatGPT responses. We developed a suite of techniques and a tool called CID (ChatGPT Incorrectness Detector) to automatically test and detect the incorrectness in ChatGPT responses. CID is based on the iterative prompting to ChatGPT by asking it contextually similar but textually divergent questions (using an approach that utilizes metamorphic relationships in texts). The underlying principle in CID is that for a given question, a response that is different from other responses (across multiple incarnations of the question) is likely an incorrect response. In a benchmark study of library selection, we show that CID can detect incorrect responses from ChatGPT with an F1-score of 0.74 -- 0.75.