一句话：报告首届 Poisoned Chalice 竞赛，将代码污染检测设为白盒成员推断，以推动可信的 LLM 评估。
大型语言模型日益用于评估和支持软件工程任务，但基准实例是否在预训练中见过常不明确，导致数据污染，可能虚高指标并误导对模型能力的判断。许多现代模型训练语料仅部分公开，直接去污染不可行，因此需要无需完整训练语料即可检测模型先前暴露的方法。本文组织首届 Poisoned Chalice of LLM Evaluation 竞赛（FSE-AIWare 2026 Competition Track），将污染检测形式化为源代码上的白盒成员推断任务，提供整理数据集、目标模型、基线攻击，并在留出模型和数据集上做最终评估。本文报告竞赛设置与结果，旨在促进方法超越浅层数据集痕迹和单一训练设定，并推动软件工程领域可信 LLM 评估社区发展。
Large language models are increasingly used to evaluate and support software engineering tasks, yet the validity of these evaluations is often undermined by uncertainty about whether benchmark instances were seen during pretraining. This can lead to data contamination, which may inflate performance and result in misleading conclusions about model capability. Despite this, the training corpora of many modern models are only partially disclosed, making direct decontamination infeasible. This creates a need for practical methods that can detect a large language models' prior exposure to training data without access to the full training corpus. To address this challenge, we organize the first Poisoned Chalice of LLM Evaluation Competition, co-located with the FSE-AIWare 2026 Competition Track. The competition frames contamination detection as a white-box membership inference task on source code and provides participants with curated datasets, target models, baseline attacks, and a final evaluation on a held-out model and dataset. This design encourages methods that generalize beyond superficial dataset artifacts and beyond a single training setting. This paper reports the setup and results of the competition. More broadly, the competition aims to catalyze the community around trustworthy LLM evaluation for software engineering.