一句话：通过分析被筛除的开发者问卷并测试五种LLM，揭示传统编程能力筛选易误伤真实开发者且难防LLM作弊，建议改进招募策略。
软件工程研究常依赖匿名在线开发者调查，但付费激励会吸引机器人、问卷农场与作弊者，难以验证受访者是否为真实开发者；编程能力筛选题还可能误伤认真作答者，LLM 更削弱其有效性。作者分析一项大型无偿调查（n=1,048）中因筛选被排除的86份回答，估计最多86%来自真实开发者；并测试五个LLM回答常用筛选题，调研现有筛选技术。结果显示，LLM能答对大多数筛选题，已有强大问卷作答工具，现有筛选易受LLM、机器人和问卷农场欺诈。作者建议：有偿调查只邀请已知开发者，广泛分发不补偿，并重视问卷设计与持续有创意的招募策略。
Quantitative research on software development practice often depends on the results of anonymous online surveys. However, it is difficult to verify that the respondents to such surveys are genuine developers, especially if survey responses are paid for, incentivising respondents to game the system and attracting the attention of bots and professional survey farms. One solution is to include screening questions to assess respondents' coding skills, but this introduces the risk of unfairly excluding considered responses from developers who spent time and energy on the survey. We reviewed the responses of 86 developers who were excluded via screening from analysis of a large unpaid developer survey (n=1,048). We estimated that up to 86% of the developers who were excluded were genuine developers. The advent of large language models (LLMs) casts further doubt on the use of screening questions. Investigating LLM ability, we found that five sample LLMs could answer most widely-used screening questions. Powerful LLM-based survey-taking tools now exist. We researched current survey screening techniques and found that they are susceptible to fraud via LLM use, bots and survey farms. Recruitment strategy may be a better screening technique. We recommend that if survey respondents are compensated only known developers should be invited. Surveys that are widely distributed, e.g. on social media, should not compensate respondents. Instead, researchers should focus on good survey design and sustained, imaginative recruitment strategies.