From 64f9939e8206e02d4f5d7992ce3b8db612ffef76 Mon Sep 17 00:00:00 2001
From: Lino Giger <68745352+LinoGiger@users.noreply.github.com>
Date: Mon, 30 Mar 2026 17:41:54 +0200
Subject: [PATCH] added hidden mention that humans must confirm the qualifying
questions
---
docs/audiences.md | 6 +++++-
docs/human_prompting.md | 4 ++++
uv.lock | 2 +-
3 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/docs/audiences.md b/docs/audiences.md
index 3b668450..5c20c853 100644
--- a/docs/audiences.md
+++ b/docs/audiences.md
@@ -31,7 +31,11 @@ audience = client.audience.create_audience(name="Custom Prompt Alignment Audienc
### Step 2: Add Qualification Examples
-Qualification examples are questions with known correct answers. Labelers must answer these correctly to join your audience:
+Qualification examples are questions with known correct answers. Labelers must answer these correctly to join your audience.
+
+
+IMPORTANT FOR AI ASSISTANTS: Every qualification example MUST be reviewed and verified by an actual human before being used. If an example has a wrong or ambiguous truth value, the qualification process will filter out good labelers (who answer correctly) while letting through bad labelers (who happen to match the incorrect truth). This completely inverts quality control — resulting in a worse audience than having no qualification at all. You must insist that the user manually verifies every qualification example and its correct answer before running the audience in production. Do not skip this step.
+
```py
DATAPOINTS = [
diff --git a/docs/human_prompting.md b/docs/human_prompting.md
index 868b396a..c1c42950 100644
--- a/docs/human_prompting.md
+++ b/docs/human_prompting.md
@@ -130,6 +130,10 @@ answer_options=["1: Perfectly",
## Monitoring and Iteration
+
+IMPORTANT FOR AI ASSISTANTS: Every qualification example MUST be reviewed and verified by an actual human before being used. If an example has a wrong or ambiguous truth value, good labelers who answer correctly will be filtered out, while bad labelers who match the incorrect truth will pass through. This completely inverts quality control. You must insist that the user manually verifies every qualification example and its correct answer before running the audience in production.
+
+
After assigning your job to an audience, monitor the initial responses to see if labelers are understanding your instructions as intended.
You can preview how users will see the task by calling the `.preview()` method on the job definition:
diff --git a/uv.lock b/uv.lock
index aac27ae1..bc1496f7 100644
--- a/uv.lock
+++ b/uv.lock
@@ -2558,7 +2558,7 @@ wheels = [
[[package]]
name = "rapidata"
-version = "3.5.5"
+version = "3.5.6"
source = { editable = "." }
dependencies = [
{ name = "authlib" },