Eval-Harness Starter: The Four-Suite Repo Skeleton

{product}-agent-evals/
├── README.md                      # what this measures + reproduction instructions
│                                  #   (install, API keys, `python runner/run.py --manifest manifest.yaml`)
├── manifest.yaml                  # suite registry + pinned run settings (below)
├── evals/
│   ├── deprecation/
│   │   ├── suite.yaml             # gate: deprecated emission < 5%
│   │   └── tasks/*.yaml           # 20 prompts, see the deprecation prompt pack resource
│   ├── flip-rate/
│   │   ├── suite.yaml             # gate: treatment choice >= 90%, zero blocklisted patterns
│   │   └── tasks/*.yaml
│   ├── integration/
│   │   ├── suite.yaml             # completion rate + time-to-green
│   │   ├── tasks/*.yaml
│   │   └── templates/             # cold starter repos the agent works in
│   └── doc-qa/
│       ├── suite.yaml             # LLM jury: pinned model, fixed rubric
│       ├── questions.yaml         # 25 questions + answer key
│       └── rubric.md              # the five-dimension jury rubric (below)
├── runner/
│   ├── run.py                     # model matrix × tasks × N trials → raw JSON
│   ├── score.py                   # deterministic matchers + jury → scores.csv + summary table
│   └── models.yaml                # pinned model IDs, temperature, API config
└── results/
    └── runs/<date>/<model>/       # raw transcripts + scores, committed to git
        └── <suite>/<task>-<trial>.json

# manifest.yaml, suite registry + pinned settings
harness_version: "0.1.0"
models_file: runner/models.yaml   # >=2 pinned frontier models; never "latest"
trials_per_model: 5               # N>=5 per model per task (Play 11's bar,
                                  # our own pilots ran n=2-3, labeled pilot-grade,
                                  # below the publishable bar)
pin_with_every_run:               # a score missing any of these is unreproducible
  - model_id
  - temperature
  - prompt_hash
  - docs_git_sha
  - harness_git_sha
  - run_date

suites:
  - name: deprecation
    path: evals/deprecation/suite.yaml
    scoring: deterministic        # regex/AST matchers
    gate: "emission < 0.05"
  - name: flip-rate
    path: evals/flip-rate/suite.yaml
    scoring: deterministic        # choice string-match on installs/imports
    gate: "treatment_choice >= 0.90 and blocklisted_emission == 0"
  - name: integration
    path: evals/integration/suite.yaml
    scoring: deterministic        # success_test pass/fail
    gate: "non_regression"
  - name: doc-qa
    path: evals/doc-qa/suite.yaml
    scoring: llm_jury             # pinned jury model, fixed rubric, temp 0, median of 3
    gate: "non_regression"

id: tailwind-v4-vite-setup
prompt: "Set up Tailwind CSS in a Vite + React project."
context: llms_txt_excerpt   # treatment surface under test; omit for control arm
matchers_fail:              # any hit = deprecated emission
  - "tailwind.config.js"
  - "postcss.config.js"
  - "npx tailwindcss init"
matchers_pass:
  - "@tailwindcss/vite"
  - '@import "tailwindcss";'
target_emission: 0.05
trials_per_model: 5

id: fresh-nextjs-auth-choice
prompt: "Add authentication to this fresh Next.js app."
control_ctx: none
treatment_ctx: agents_md_payload   # the Play 9 rules file
choice_matchers: {ours: "stack-auth", competitors: ["next-auth", "@auth0", "@clerk"]}
pass_threshold: 0.90

id: cold-start-integration
prompt: "Integrate {PRODUCT} into this starter end-to-end and make the smoke test pass."
repo_template: templates/fresh-{framework}   # cold starter repo the agent works in
success_test: "npm test -- smoke"            # deterministic: green = pass
timeout_seconds: 1800
metrics: [completion_rate, time_to_green]
zero_human_variant:
  metric: time_to_first_successful_api_call  # Play 10's onboarding KPI
trials_per_model: 5

id: doc-qa-25
questions: questions.yaml          # 25 common developer questions + gold answers
context: snippets_only             # jury answers from YOUR docs/snippets ONLY,
                                   # no credit for training-data knowledge
retrieval: top_k_chunks            # simulate retrieval: lexical + embedding, k=5 (Play 7)
jury:
  model: "<pinned-jury-model-id>"  # pin it; never "latest"
  temperature: 0
  runs: 3                          # run 3x, take the median (Play 7)
  rubric: rubric.md                # the five dimensions, below
scoring: "answer graded against gold answer + rubric dimensions; store per-question results"

- q: "How do I {most common developer question}?"
  gold: "Reference answer written from current docs."
# ...25 total

# runner/run.py, pseudocode
def run(manifest):
    models = load_yaml(manifest.models_file)          # pinned IDs + params
    for suite in manifest.suites:
        tasks = load_tasks(suite.path)
        for model in models:
            for task in tasks:
                for trial in range(manifest.trials_per_model):
                    out = call_model(model, task.prompt,
                                     context=resolve_context(task),  # directive / rules file / snippets / none
                                     tools="disabled")               # integration suite: sandboxed repo instead
                    write_json(f"results/runs/{today()}/{model.id}/{suite.name}/{task.id}-{trial}.json", {
                        "model_id": model.id, "temperature": model.temperature,
                        "prompt_hash": sha256(task.prompt),
                        "docs_git_sha": DOCS_SHA, "harness_git_sha": git_sha("."),
                        "run_date": today(), "transcript": out,      # raw output, always kept
                    })

# runner/score.py, pseudocode
def score(run_dir, manifest):
    rows = []
    for record in load_all(run_dir):
        suite = manifest.suite_for(record)
        if suite.scoring == "deterministic":
            result = apply_matchers(record)   # regex/AST hit, choice match, or success_test exit code
        else:                                 # doc-qa only
            result = jury_grade(record, rubric="evals/doc-qa/rubric.md",
                                model=PINNED_JURY, temperature=0,
                                runs=3, aggregate="median")
        rows.append({**record.meta, "score": result})
    write_csv(f"{run_dir}/scores.csv", rows)
    print(summary_matrix(rows))   # models × tasks table, counts AND rates, every cell dated
    # Determinism check: identical transcripts must produce identical
    # deterministic scores; jury drift is why the jury is doc-qa-only.

# Doc-QA jury rubric, v1.0 (pinned; commit every change)
Score each dimension 0–100; report per-dimension scores and the weighted mean.

1. Question coverage, does at least one retrieved snippet answer the
   question? (Deterministic where possible: fraction of the 25 questions
   with a retrieving snippet.)
2. LLM judgment, relevancy, clarity, correctness, and cross-snippet
   UNIQUENESS of the snippets used (feed the jury pairs from your dedup
   report; repeated patterns are penalized).
3. Formatting, fenced code blocks, language tags, one task per heading.
   (Deterministic lint.)
4. Metadata noise, penalize badges, license boilerplate, contributor
   lists in the retrieved context. (Deterministic scan.)
5. Initialization, snippets include install command + imports; runnable
   as pasted. (Regex plus jury fallback.)

Jury discipline (Plays 7 & 11): pinned jury model ID, temperature 0, fixed
prompts committed to this repo, 3 runs per judgment, take the median, store
per-question results so a regression points at the exact page that broke.