Who writes the 571 criteria

Harvey grades its due-diligence agents —the document review that precedes buying a company— against 571 yes-or-no criteria per data room. Questions like: does it cite the change-of-control clause in contract 14?

It publishes the number. It never says who wrote them.

Ramp did the same for its accounting-close agent and there it does say: they were written by “real-life accountants” (Ramp Engineering).

That one line decides whether your eval measures your domain or measures your own team. And it weighs: with rubrics like these, seven models go from 23.3% correct to 62.4% across 50 held-out data rooms, without touching the model. What changes is the code around them: 39.1 points on average (Harvey and Baseten, September 2026).

Writing those criteria is human work, done by someone with a job, a full calendar and a boss who never asked for this. Getting their hours and turning what they know into tested artifacts: that’s the forward deployed engineer’s work, the engineer who ships product embedded in the customer’s domain. Five published cases show what that person writes.

The expert writes the rubric, and the job title is worth naming

Ramp’s 3,469 grading criteria cover 237 tasks, and they’re only the eight scenarios it publishes from a larger benchmark (Ramp Engineering; I unpacked it here). Those scenarios are synthetic companies, same as Harvey’s data room: Aravon Bridge Bank, 2,270 documents across 82 folders and 31 million tokens. What’s synthetic is the test case. The criterion it’s judged by isn’t.

Harvey calls its own an expert rubric and leaves authorship blank. The fact it doesn’t publish is exactly the one that would tell you whether that rubric is worth anything: a list with no declared author could have been written by the team building the agent, in which case it measures what the agent already did well.

The change Ramp introduces is one of order. In the old split the expert arrives last: sees a demo, says “this is wrong”, and the judgement dies in the meeting. Arriving first, they write yes-or-no questions, and that shape makes the criterion reusable. The same list grades the agent, compares implementations and serves to tune the model by rewarding what scores against it. With that —and here the model does get touched— Harvey takes Qwen3.5 from 29.9% to 63.0%.

Worth watching who does what there. Scoring is done by a judge model: another model handed the answer and the rubric, returning a pass or fail per criterion. Deciding what counts as a pass is signed by a person.

In practice: take the criteria list from your eval —the test you use to decide whether the agent did the job— and write next to each block who signs it, with a name and a job title. If the whole column matches your engineering org chart, what you have is a list of what your agent already does well.

The reviewer stops reading diffs and writes the list of what breaks

Duckbill is fifteen people. They found themselves with 60 open PRs for a team of five and, instead of hiring reviewers, they changed the rule.

Since that change, human review is mandatory in five places. The public API. The MCP server —the Model Context Protocol, the standard an agent uses to discover and call external tools—. Auth, the design system and non-additive schema changes. Agent skills too. A script enforces it by labelling the PR, with the test-coverage floor at 85%.

They went from 353 to 684 merged PRs, 94% more. Median merge time fell from 26 hours with a human to 1 hour without one (Gergely Orosz, September 2026).

What changed is where the reviewer’s judgement lives. Risk used to be judged PR by PR, inside their head, and it didn’t scale past their working day. Now it’s judged once, as a list of paths, and the human is kept for high blast radius: how much breaks if the change goes wrong.

The opposite path is documented too. Add an automated reviewer without that list and engineers end up reviewing the review rather than the code.

Write your blast-radius list with whoever has eaten the outages, not with whoever holds the title, and have a script apply it. If it holds for two weeks —measure median merge time and reverts per label—, your team’s next opening isn’t for someone to read diffs.

The on-call engineer leaves their experience at the decision point

Over four weeks, Ramp’s on-call agent assisted 575 of the 1,220 PRs that fixed incidents. Each fix needed 37% fewer engineers. Responders rate it helpful 90% of the time (Ramp Engineering, September 2026).

Before that, here’s what it did. It looked at the logs, saw the oldest error was twenty minutes old and concluded the incident had started then. With absolute confidence. Query windows often default to a one-hour lookback, so it took the oldest data point that fit inside and built a theory around a deploy from twenty minutes earlier.

Anyone who has carried a pager spots it instantly. That’s the point: the person who has already eaten those shifts knows the trap, and what Ramp did was get it out of their head and put it where the agent goes wrong.

They did it in two places. First they wrote the question into the prompt: “Before acting on a query result, ask: ‘Could my query be silently excluding the data I’m looking for?’”. But over a long session that line dilutes, so they added a hook. An interceptor that stops the agent before it posts to Slack and drops in, right there, the order to load the traps list and work through it. Ramp calls them just-in-time instructions: rather than front-loading them all, they’re delivered when the agent is about to act.

The failure they fix is a rookie’s: stopping at the first plausible explanation. So that artifact isn’t a help doc, it’s a veteran’s experience executing at the exact moment the rookie would get it wrong.

In your next post-mortem, add the line “what did the human check that the agent wouldn’t have” and put a name on it. When the same name comes up three times, that person has stopped being an on-call resource: they’re the author of your next artifact.

From the customer you want two people, not a requirements document

The two people you need from a customer aren’t the one who signs the contract. They’re whoever grants read access to the real source and whoever owns the file the company sends its clients.

You can see it in how OpenAI shaped its financial-services product, with Morgan Stanley and Evercore as design partners: customers who co-design before general launch (OpenAI, September 2026). Two decisions came out of it that have nothing to do with the model. Indexing and hosting Daloopa, PitchBook, LSEG and Crunchbase data on its own infrastructure, because the data providers’ MCP connectors proved unreliable. And letting an admin publish the firm’s Excel, Word and PowerPoint templates, with information barriers between teams that may not share data.

A requirements document describes the work. A template is the work. That’s why the template gets versioned like code and the document ages in a folder.

The price is about people too: the data runs through the infrastructure of whoever sells you the product, and someone on your side has to be able to say no (Maya Zehavi makes the point).

Ask for those two names before kickoff and put their artifacts in the repo. If two weeks in all you have is a requirements document, the problem is who you’ve been talking to.

The FDE negotiates expert hours, not requirement lists

Ramp’s 3,469 criteria and Harvey’s 571 for one data room are hours someone took away from their real job. Getting those hours is what the FDE does —the engineer deployed at the customer’s site— and that constraint shows up on no architecture diagram.

Their deliverable is the conversion: expert hours in, tested artifacts out, artifacts that survive a model swap. The negotiation has a concrete currency: the expert gives up hours when they get back less of the work they already hate. That’s the sentence you use to ask for the time, rather than “we need to validate some use cases”.

It’s worth knowing what would break the argument too. If your domain changes its rules every quarter, the rubric depreciates before it pays for itself. And in Harvey’s own measurements, splitting the work across subagents raises generation cost per data room for six of the seven models (Harvey and Baseten), so none of this comes free. What doesn’t change is who knows what “done right” means.

Put the expert’s time in the plan as a line item with a unit and a date —“six hours of the senior accountant across two sessions, 80 criteria written”— rather than as “available for questions”. And measure criteria per week: it’s the only metric in this job that doesn’t depend on the model.

Four artifacts, four owners: the accountant’s rubric, the blast-radius list from whoever eats the outages, the traps list from the on-call veteran, the template from whoever owns the file. The model writes none of them, and all four survive replacing it. Who is that person in your company, the one who knows what “done right” means in your domain, and how many of their hours do you have booked in writing, with a date?