This is the exact QA pack that ships with every Goldset batch, shown on a representative code-review eval pilot. Client identity, rubric text and batch ID are redacted; the data below is illustrative.
Thresholds agreed before kickoff: aggregate gold accuracy ≥ 90% and Cohen's κ ≥ 0.70. Delivered: 95.0% gold accuracy and κ = 0.80. Redo guarantee not triggered.
| Annotator | Labels | Gold checks | Gold accuracy | Status |
|---|---|---|---|---|
| A-01 | 100 | 19 / 20 | 95% | CLEAR |
| A-02 | 100 | 20 / 20 | 100% | CLEAR |
| A-03 | 100 | 18 / 20 | 90% | RECALIBRATED D4 |
| A-04 | 100 | 19 / 20 | 95% | CLEAR |
| A-05 | 100 | 19 / 20 | 95% | CLEAR |
Aggregate: 95 / 100 gold checks correct (95.0%) against the 90% acceptance threshold. Gold items are seeded blind — annotators cannot distinguish them from production items.
Most common disagreement: Correct vs Correct, minor issues (17 of 29) — an expected boundary; adjudications applied rubric §3.1 consistently and are marked in the dataset. One drift alert (day 4, A-03, minor/incorrect boundary) → 30-minute recalibration against the rubric; A-03's subsequent gold checks: 9/9.
R-01 · The rubric's minor issues boundary needed one clarification (over-broad exception handling, §3.1) — pressure-tested with the client on day 2, applied consistently from day 3. Recommend folding the clarified wording into rubric v1.4 before any production batch.
R-01 · Security verdicts (34 items) were the strongest dimension: every Unsafe call names the concrete attack vector in its rationale, per the rubric's evidence requirement.
R-01 · All 29 adjudications are flagged in labelled_data.json with the deciding rationale, so downstream filtering by agreement level is possible.