一句话：设计并评估面向在线调查的程序员筛选问题，以识别并过滤不具备编程技能的参与者。
在线招募足够数量的专业程序员参与研究调查常受时间、地域与成本限制；Clickworker、Qualtrics 等平台虽可招募具备编程技能的参与者，但可能存在误解与欺诈，使无编程技能者进入研究并污染数据。本文设计了一套筛选问题：对具备编程技能者易于快速回答，对不具备者难以正确回答。作者招募多批有/无编程技能参与者，并在对抗性场景中评估问题的有效性与效率。结果显示，某批 Clickworker 中自称有编程技能者 42% 未达标准，应被过滤。最后给出可供研究者使用的推荐筛选问题。
Recruiting professional programmers in sufficient numbers for research studies can be challenging because they often cannot spare the time, or due to their geographical distribution and potentially the cost involved. Online platforms such as Clickworker or Qualtrics do provide options to recruit participants with programming skill; however, misunderstandings and fraud can be an issue. This can result in participants without programming skill taking part in studies and surveys. If these participants are not detected, they can cause detrimental noise in the survey data. In this paper, we develop screener questions that are easy and quick to answer for people with programming skill but difficult to answer correctly for those without. In order to evaluate our questionnaire for efficacy and efficiency, we recruited several batches of participants with and without programming skill and tested the questions. In our batch 42% of Clickworkers stating that they have programming skill did not meet our criteria and we would recommend filtering these from studies. We also evaluated the questions in an adversarial setting. We conclude with a set of recommended questions which researchers can use to recruit participants with programming skill from online platforms.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2103-04429,
title = {Do you Really Code? Designing and Evaluating Screening Questions for Online Surveys with Programmers},
author = {Anastasia Danilova and Alena Naiakshina and Stefan Horstmann and Matthew Smith},
booktitle = {ICSE},
year = {2021},
doi = {10.1109/icse43902.2021.00057}
}