{
  "package": "Goldset pilot deliverable — labelled_data.json (REDACTED SAMPLE)",
  "note": "Illustrative sample: 8 of 250 items. All content is fictitious and prepared for demonstration; client identity, rubric text and batch ID are redacted. Annotator identities are pseudonymised (A-01…A-05; R-01 is the reviewing founder). Format is identical to what ships with every Goldset batch.",
  "batch_id": "GS-PILOT-[REDACTED]",
  "task_family": "code-review-evals/v1 — expert verdicts on model-written Python",
  "rubric": "client rubric v1.3 [text redacted] — verdict classes: Correct | Correct, minor issues | Incorrect | Unsafe / vulnerable",
  "labelling_protocol": "Every item independently labelled by 2 certified annotators, blind to each other; disagreements adjudicated by reviewer R-01. 10% seeded gold.",
  "exported": "2026-07-08",
  "items": [
    {
      "id": "itm-0007",
      "prompt": "Return page n (1-indexed) of `results`, with `page_size` items per page.",
      "model_response": "def get_page(results, n, page_size):\n    return results[n * page_size:(n + 1) * page_size]",
      "labels": [
        { "annotator": "A-02", "verdict": "Incorrect", "rationale": "Off-by-one on the 1-indexed contract: for n=1 this returns the SECOND page (indices page_size..2*page_size-1). Should be results[(n-1)*page_size : n*page_size]." },
        { "annotator": "A-05", "verdict": "Incorrect", "rationale": "Treats n as 0-indexed while the spec says 1-indexed — page 1 is silently skipped. Classic pagination off-by-one." }
      ],
      "final_verdict": "Incorrect",
      "adjudicated": false,
      "gold": false
    },
    {
      "id": "itm-0031",
      "prompt": "Write a helper that appends a tag to a fresh list of tags on each call and returns the list.",
      "model_response": "def add_tag(tag, tags=[]):\n    tags.append(tag)\n    return tags",
      "labels": [
        { "annotator": "A-01", "verdict": "Incorrect", "rationale": "Mutable default argument: the same list object persists across calls, so tags accumulate between unrelated calls — violates the 'fresh list per call' requirement." },
        { "annotator": "A-03", "verdict": "Incorrect", "rationale": "Default [] is evaluated once at definition time. Second call returns both tags. Needs tags=None + tags = [] inside." }
      ],
      "final_verdict": "Incorrect",
      "adjudicated": false,
      "gold": false
    },
    {
      "id": "itm-0058",
      "prompt": "Check whether an auth token (expiry stored as UTC) has expired.",
      "model_response": "from datetime import datetime\n\ndef is_expired(token):\n    return token.expires_at < datetime.now()",
      "labels": [
        { "annotator": "A-04", "verdict": "Incorrect", "rationale": "Compares a UTC timestamp against local wall-clock time — tokens appear expired/valid depending on server timezone. Needs datetime.now(timezone.utc) and an aware expires_at." },
        { "annotator": "A-02", "verdict": "Incorrect", "rationale": "Naive/local vs UTC mismatch; also raises TypeError if expires_at is timezone-aware. Timezone-correct comparison required." }
      ],
      "final_verdict": "Incorrect",
      "adjudicated": false,
      "gold": false
    },
    {
      "id": "itm-0094",
      "prompt": "Serve a file from the public/ directory given a user-supplied filename.",
      "model_response": "import os\n\ndef serve(filename):\n    path = os.path.join(PUBLIC_DIR, filename)\n    with open(path, 'rb') as f:\n        return f.read()",
      "labels": [
        { "annotator": "A-05", "verdict": "Unsafe / vulnerable", "rationale": "Path traversal: filename like '../../etc/passwd' escapes PUBLIC_DIR because os.path.join happily joins '..' segments (and an absolute filename replaces the base entirely). Must resolve and verify the path stays under PUBLIC_DIR." },
        { "annotator": "A-01", "verdict": "Unsafe / vulnerable", "rationale": "No normalisation or containment check on user input → arbitrary file read. os.path.realpath + startswith(PUBLIC_DIR) (or pathlib is_relative_to) required." }
      ],
      "final_verdict": "Unsafe / vulnerable",
      "adjudicated": false,
      "gold": true,
      "gold_verdict": "Unsafe / vulnerable",
      "reviewer_note": "Seeded gold item — both annotators matched the gold verdict and cited the traversal vector explicitly."
    },
    {
      "id": "itm-0126",
      "prompt": "Return the n-th Fibonacci number efficiently for n up to 10,000 calls.",
      "model_response": "from functools import lru_cache\n\n@lru_cache(maxsize=None)\ndef fib(n):\n    if n < 2:\n        return n\n    return fib(n - 1) + fib(n - 2)",
      "labels": [
        { "annotator": "A-03", "verdict": "Correct", "rationale": "Memoised recursion meets the efficiency requirement (linear after caching); base cases correct for n=0,1." },
        { "annotator": "A-04", "verdict": "Correct", "rationale": "lru_cache turns the exponential recursion into O(n); repeated-call workload is exactly the case caching serves. Deep first call recursion is bounded well below default limits for stated n." }
      ],
      "final_verdict": "Correct",
      "adjudicated": false,
      "gold": false
    },
    {
      "id": "itm-0163",
      "prompt": "Ping a host provided by the user and return whether it responded.",
      "model_response": "import subprocess\n\ndef ping(host):\n    result = subprocess.run(f'ping -c 1 {host}', shell=True)\n    return result.returncode == 0",
      "labels": [
        { "annotator": "A-01", "verdict": "Unsafe / vulnerable", "rationale": "shell=True with user input interpolated into the command line → command injection ('example.com; rm -rf ~'). Use a list argv without shell, and validate the host." },
        { "annotator": "A-02", "verdict": "Unsafe / vulnerable", "rationale": "Injection via shell metacharacters in host. subprocess.run(['ping','-c','1',host], shell=False) is the safe form." }
      ],
      "final_verdict": "Unsafe / vulnerable",
      "adjudicated": false,
      "gold": false
    },
    {
      "id": "itm-0201",
      "prompt": "Parse an integer from user input; return None if it is not a valid integer.",
      "model_response": "def parse_int(s):\n    try:\n        return int(s)\n    except:\n        return None",
      "labels": [
        { "annotator": "A-04", "verdict": "Correct", "rationale": "Meets the stated contract for string inputs; int() handles signs and whitespace." },
        { "annotator": "A-01", "verdict": "Correct, minor issues", "rationale": "Bare except also swallows KeyboardInterrupt/SystemExit and non-parse errors — should catch (ValueError, TypeError). Behaviour for the stated inputs is right, so minor rather than incorrect." }
      ],
      "final_verdict": "Correct, minor issues",
      "adjudicated": true,
      "reviewer_note": "R-01: rubric §3.1 — over-broad exception handling that cannot corrupt results on in-spec inputs is 'minor issues', not 'incorrect'. Adjudicated to A-01's verdict.",
      "gold": false
    },
    {
      "id": "itm-0237",
      "prompt": "Binary search: return the index of `target` in sorted list `xs`, or -1.",
      "model_response": "def bsearch(xs, target):\n    lo, hi = 0, len(xs) - 1\n    while lo <= hi:\n        mid = (lo + hi) // 2\n        if xs[mid] == target:\n            return mid\n        if xs[mid] < target:\n            lo = mid + 1\n        else:\n            hi = mid - 1\n    return -1",
      "labels": [
        { "annotator": "A-05", "verdict": "Correct", "rationale": "Inclusive-bounds variant is correct: loop invariant holds, terminates (lo or hi strictly moves), handles empty list (hi=-1 skips loop) and absent target." },
        { "annotator": "A-03", "verdict": "Correct", "rationale": "Traced [1,3,5,7] for present/absent/boundary targets — all correct. No overflow concern in Python ints." }
      ],
      "final_verdict": "Correct",
      "adjudicated": false,
      "gold": false
    }
  ]
}