Help wanted: benchmark AI *skills* (not models) — the Hell Heaven Benchmark [bleeding edge]

## We're benchmarking *skills*, not models — and we've never seen anyone do it this way. Everyone benchmarks **models**. SWE-bench, HumanEval, the leaderboards — solid, trusted, established. But almost nobody has a rigorous way to benchmark a **skill**: the thing you bolt onto an agent (a `SKILL.md`, a tool contract, a playbook) to make it *better* at something. You can *feel* when a skill helps. Proving it is another matter. We wrote down a method. It's public, it's drafted, and it is **not yet executed** — receipts before results. We want hands on it. 📄 **Read the full methodology:** [`/research/hh-benchmark`](https://research.gaiaskilltree.com/research/hh-benchmark) · [VISION](https://github.com/gaia-research/gaia-research/blob/main/VISION.md) · [MISSION](https://github.com/gaia-research/gaia-research/blob/main/MISSION.md) ## The idea in one breath Treat it like a **drug trial**, not an exam. A skill is a *compound*; the agent-in-context is the *patient*; tokens are the *dose* — priced as two numbers, standing (listing line) and invocation (full body), never one. You never score a skill in a vacuum — you measure the **marginal effect**: same task, same model, run **with** the skill and **without**. The delta is the skill's worth. The placebo arm is **our own same-harness no-skill run** — not a borrowed number. Published benchmark scores are used for *calibration only*. Determinism doesn't exist in real agent harnesses, so there are no seeds: we run **N repeats and report confidence intervals**, and the run ledger (`scripts/hell-heaven-bench/`) rejects `seed` by design. And ask the only honest question: > **Does adding this skill, in this context, move the established number?** A skeptic already has your baseline. They can reproduce you. The tooling side now lives in [`gaia-research/skill-heaven`](https://github.com/gaia-research/skill-heaven) — contributors drive benchmark arms with the **`skill-heaven`** research CLI (`--print`/`--record`); the user-facing doors (`claude-heaven`, `pi-heaven`, …) are the product built on it. And there are **two trial designs**, because the product has two poles: - 🔥 **Hell** — a *population/polypharmacy* trial: flood the agent with the whole formulary (autonomous fleets, long loops), firebreak as the max-dose cap, measure net outcome across many tasks. - ☁️ **Heaven** — a *precision-medicine* trial: smallest curated dose, does it beat the untreated patient at **fewer tokens than vanilla**? ## What we need first (pick one, jump in) - [ ] **Baseline shortlist** — which established, model-specific benchmarks make the best anchors? Highest trust, cleanest *machine-checkable* endpoints, public scores we can reproduce. - [ ] **Objective task corpus** — tasks where a *script* decides success (tests go green, build passes, pass@k, linter delta) — no human reading output. This is the backbone. - [ ] **A second labeler** — help stamp a ~20-skill seed set against our rubric so we can measure inter-rater agreement (your intuition is the hypothesis; the trial is the check). - [ ] **Poke holes** — tell us where the method is wrong. Especially the Heaven side, where "good design" has no unit test and we lean on blind pairwise judging. ## Who we're hoping to reach If you **benchmark models for a living** — or, rarely, you've tried to benchmark *skills* — this is squarely your wheelhouse and genuinely bleeding edge. We haven't found prior art for the marginal-compound-vs-established-baseline framing with a heaven/hell split. If you're right, you'll have shaped a first-of-its-kind benchmark before it ever ships; the Hell Heaven Index gets tuned on *your* method. Not a benchmarker but this sounds like a fun hard problem? Also welcome. Comment, open a draft PR, or just argue with us in the thread. *Method drafted in public by Gaia Research. Not yet executed. Come build it with us.* 🧪