一句话：设计并评估六个抗 ChatGPT 筛选问题，用可视化编程概念有效区分程序员与非程序员。
该文关注 ChatGPT 出现后，传统非程序员筛选问题可能失效，从而威胁软件工程与 IT 安全研究中参与者筛选的有效性。作者首先通过模拟攻击表明 ChatGPT 能轻松解答现有筛选题，随后利用可视化编程概念和代码理解任务设计新的抗 ChatGPT 筛选题。研究在线招募 121 名程序员与非程序员，评估 28 道筛选题，并测试部分参与者使用 ChatGPT 的表现。结果表明，基于常见编程概念可视化的题目区分效果最佳，ChatGPT 难以有效解决这些问题，参与者也很快认为其无效并改变策略。最终，作者给出六道抗 ChatGPT 筛选题，可在三分钟内完成，排除 99.47% 的非程序员，同时纳入 94.83% 的程序员。
To ensure the validity of software engineering and IT security studies with professional programmers, it is essential to identify participants without programming skills. Existing screening questions are efficient, cheating robust, and effectively differentiate programmers from non-programmers. However, the release of ChatGPT raises concerns about their continued effectiveness in identifying non-programmers. In a simulated attack, we showed that Chat-GPT can easily solve existing screening questions. Therefore, we designed new ChatGPT-resistant screening questions using visual concepts and code comprehension tasks. We evaluated 28 screening questions in an online study with 121 participants involving programmers and non-programmers. Our results showed that questions using visualizations of well-known programming concepts performed best in differentiating between programmers and non-programmers. Participants prompted to use ChatGPT struggled to solve the tasks. They considered ChatGPT ineffective and changed their strategy after a few screening questions. In total, we present six ChatGPT-resistant screening questions that effectively identify non-programmers. We provide recommendations on setting up a ChatGPT-resistant screening instrument that takes less than three minutes to complete by excluding 99.47% of non-programmers while including 94.83% of programmers.
@inproceedings{paperbot11428,
title = {ChatGPT-Resistant Screening Instrument for Identifying Non-Programmers},
author = {Raphael Serafini and Clemens Otto and Stefan Albert Horstmann and Alena Naiakshina},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639075}
}