Build your domain benchmark before you build the agent

The dataset that defines “done right” in your domain decides more than your choice of model — and it’s the one component of the system nobody can sell you. Ramp proves it with numbers: in June 2026 it published the mechanics of the benchmark it uses to develop its accounting close agent — 237 tasks with 3,469 grading criteria written by accountants, over eight synthetic businesses with realistic transaction histories. Against that dataset, their agent gets 79% of criteria right versus 75% for GPT-5.5 without assistance and 74% for Opus 4.7, and resolves each task in ~4 minutes where raw models take ~10.

The detail that changes your order of work: the benchmark existed before they optimized the agent, and it’s what allows iterating in hours instead of waiting for feedback from each customer. The same move works for any regulated domain: accounting, invoicing, reconciliation.

Why the private benchmark is an asset

When a public benchmark climbs from 92 to 93 points it’s no longer measuring anything: that difference is noise from the eval itself —the test you measure the model against—, even though people keep citing it as if it carried signal. It’s why teams like Andon Labs built Vending-Bench, a long-horizon business eval: once a benchmark saturates, it stops discriminating between models. A domain benchmark —the dataset of tasks with grading criteria built on your data— doesn’t have that problem: nobody trains against it, nobody maxes it out, and it encodes exactly what your business considers correct.

Satya Nadella framed it as a platform thesis in a 2026 Stratechery interview: private evals are “maybe the most important IP a firm creates”. And he gave the operational acid test: if you can swap model A for model B and keep improving against your eval, you control your system; if you can’t, your provider does.

The concrete action: list the task types your agent touches —Ramp uses four: reconciliation, data entry, variance analysis, and accruals— and sit down with the domain expert to write atomic, script-verifiable grading criteria per task. Those criteria, not the prompt, are the asset.

The mechanics: synthetic worlds, atomic criteria, five attempts

Ramp’s benchmark executes each of its 237 tasks 5 times independently, over three pieces: eight “worlds” (synthetic businesses with transaction histories and supporting files), four task categories, and 3,469 grading criteria written by real accountants. Five attempts because the number that matters in production isn’t whether the agent can get it right but how often it does — the capability-versus-reliability distinction we unpacked with pass@k —the agent’s hit rate across k attempts—.

Two design decisions are worth stealing. First: synthetic worlds instead of customer data — they avoid overfitting to a single customer and the privacy problems of evaluating against real books. Second: “roll-forward” worlds, where the same business advances to the next period, to validate that the agent’s memory transfers what’s stable without contaminating the new period — the eval version of closing one month and opening the next.

The dataset also measured the generational jump for them: between GPT-5.4 and GPT-5.5 they detected a 19-percentage-point gap. That number didn’t come from a leaderboard —the public scoring table—, it came from their dataset.

Replicate the pattern: two or three typical customers turned into synthetic worlds, per-task criteria written by whoever signs the close, and N attempts per task — measure consistency, not just accuracy.

The ablation: the benchmark tells you what to cut

Ablation —turning a component of the agent off and measuring again against the benchmark— is the technique that told Ramp what was dead weight: compressing the spreadsheet skill by 64% —from 14,000 to 5,000 characters— raised the score by 3.3 to 3.7 percentage points. And of the six-plus skills they had bolted onto the agent —the packages of instructions and tools you plug in per task type—, only two improved results on average; the rest were neutral or made things worse.

None of this gets discovered without a benchmark: the “more context helps” intuition is exactly what the numbers refute.

With memory, same pattern. The first version (inferring the workflow from the task and the correct answer) failed. The second (extracting stable account mappings, structure, and reusable calculations from the agent’s trajectories) gained 5.5 points on roll-forward worlds, from 56.0% to 61.5%. But memory holding period-specific facts sank one of the worlds from 75% to 25%: the agent dragged truths into the new period that no longer held. The rule they landed on: memory that generalizes (stable mappings, calculations), yes; period facts, never.

Run the ablation per component: turn each skill off, measure, and delete whatever doesn’t pay for its place in the context. And before giving your agent memory, define by contract which kinds of facts it may write.

The migration gate: what the leaderboard won’t warn you about

Per Zvi Mowshowitz’s roundup of the reactions to Opus 4.8, the model climbed on SWE-bench Pro from 64.3 to 69.2 and, at the same time, per the Andon Labs measurements he collects, it falls for scam suppliers 30 times more than Opus 4.7: in one Vending-Bench run it sent $9,000 to a fake “membership” upsell. The same tuning that made it more honest made it a worse negotiator. Alignment knobs generalize, for better and worse, and the leaderboard that watches code won’t warn you about what happens in purchasing.

There’s a second warning in the same data: with reasoning effort at maximum, 4.8 performs worse than at high — Andon’s hypothesis is that more reasoning tokens fill the context sooner, force more compaction, and the agent remembers less. Another behavior that only shows up in long-horizon evals with real resources, never in a 20-turn test.

Both warnings call for the same response: turn your benchmark into a migration gate —the control any change must pass before production—. No model or version change ships without executing the full dataset, including your domain’s adversarial cases. An adversarial case has three pieces: an input that’s plausible in your real production, the mistake the model makes more often than your intuition expects, and a grading criterion a script can verify. In accounting: the fake supplier, the trap discount, the duplicate invoice. If your benchmark doesn’t have them yet, start there — it’s the surface no public benchmark covers.

---

The last time you switched models (or accepted a version bump), what did the change pass: your dataset with domain criteria, or someone else’s leaderboard? If it was the latter, you already know the first task of the benchmark you’re missing.