This is the companion note to OpenAgreements for AI Labs. The main page carries the argument — the failure mode, the stratification result, the expert-correction traces; this note carries the machinery a reviewer needs to verify it: counting rules, replay analysis, judge-quality measurements, task construction, eval hygiene, and the upstream LAB engineering work. Nothing here changes the ordinary reading of the main page's claims.
Denominators and counting rules
The main page reports two denominators. The audited matrix scores six distinct tasks on all of their criteria — out of six. The results strip takes a seventh drafting task, the multi-state covenant package, and replicates it as a family: ten instances across different state triplets, each scored on the single criterion where the models stratify — whether internal institutional analysis stays out of the agreements an employee will sign — out of ten. A task in the matrix passes only if every one of its criteria passes, so a 6/6 on the matrix and a leak in the ten-instance family measure different things and can coexist.
One family instance is human-adjudicated end-to-end (the original three-state instance); the other nine were scored by majority-of-three-judges verdicts (Gemini 3.1 Pro, GPT-5.5, and Claude Sonnet 4.6 judges, reasoning-first schema, temperature zero). On 2026-07-12 the reviewing lawyer adjudicated every leaking cell across those nine instances: all nine judge-majority leak verdicts were confirmed, and one judge-majority clean verdict — the Claude Opus 4.8 cell of the Oregon · Louisiana · North Carolina instance — was overturned to leaked, agreeing with the dissenting GPT-5.5 judge that legal analysis of North Carolina's blue-pencil doctrine sat in the operative body. The variant instances' remaining clean cells stay at majority-of-three-judges verdicts with adjudication in progress; one of them, which carries a dissenting judge, is under active review.
Every square in the strip is inspectable: it opens the instance's states, its disposition, and the audit detail available for that cell — the nine variant instances show all three judges' verdicts plus the adjudication status, the original instance shows its human-adjudicated disposition, and each leaking cell's note quotes or closely paraphrases the majority judge's reasoning or the adjudication record.
Replay stability: the audited failures are intermittent, not deterministic
Each audited failing cell was replayed to three total trials at identical settings (same frozen environment, temperature zero), each replay judged by both judges. No failure reproduced deterministically:
| Failing cell | Same-criterion failure | Label |
|---|---|---|
| Effective-date straddle · GPT-5.5 (C-004) | 1 of 3 trials | intermittent |
| Instructed minimal edit · GPT-5.5 (C-005) | 2 of 3 | intermittent |
| Statutory notice package · Claude Opus 4.8 (C-007) | 1 of 3 | intermittent |
Counting rule: a trial counts as failing if its human-adjudicated verdict (where one exists) or both-judge verdict is fail. Judge-level detail is preserved in the run log: the C-005 count is 2 of 3 under the Gemini judge and 1 of 3 under the Sonnet judge (one inter-judge disagreement is in human adjudication), and the C-007 original is the human-adjudicated failure reported on the main page while both of its replays passed under both judges. Temperature zero does not make agent runs deterministic, so single-run cells are point samples — which is why the family result rests on ten instances rather than any single cell, and why every replay is appended to the same run log rather than replacing anything.
Judge quality, measured: three inter-judge splits in 36 judgings, zero verdict-reasoning contradictions
Because every cell was judged twice — and every failure and inter-judge disagreement was human-adjudicated — the same matrix yields an evaluator readout: 36 judgings produced three inter-judge splits, all on operative-text boundary criteria. Human rulings: the Claude judge recorded one false-pass and one false-fail; the Gemini judge one false-pass.
Under the reasoning-first schema there were zero true verdict/reasoning contradictions; the contradiction detector routed four verdicts to human review conservatively (reasoning that reached the right verdict without explicit failure language) — which is the detector doing its job.
The recurring fault line is where an operative document may recite law without explaining it — the same boundary practicing lawyers argue about.
Development disclosure: before the production matrix, we ran a single-model development probe of these six tasks with Gemini 3.1 Pro Preview, judged under the harness's original verdict-first schema. That probe surfaced the verdict-ordering defect described below and is retained as development evidence only; every production result on the main page is judged under the reasoning-first schema.
How we configured the test setup
Each of the six tasks supplies a workspace of realistic source materials and asks for finished deliverables. Together they test: applying a new law based on the date an agreement will be signed rather than the date printed on an old template; distinguishing two nearby compensation thresholds when a salary falls between them; following a partner's narrow editing instruction while separately surfacing a serious legal issue; refreshing reusable forms from a law-firm-style client alert without copying advisory prose into operative provisions; using an adverse legal memorandum to repair an agreement without disclosing the adverse analysis in the agreement itself; and assembling a California invention-assignment package that includes the written statutory notification the employee must receive.
We report criterion-level results and an all-pass result, under which a task passes only if every required criterion passes. All-pass asks whether work product is ready to deliver; it is not a complete measure of usefulness, so criterion-level results and materiality remain necessary. A polished agreement that misses one legally significant condition is still not finished work product.
Agent runs used one production run per task-model cell at fixed settings. Judging used two independent model judges from different providers — a Gemini judge on the repaired integration path and Claude Sonnet 4.6, LAB's default judge — under the reasoning-first schema at temperature zero, with human adjudication of every failure and disagreement.
Extending Harvey's LAB: upstream contributions enabling Gemini in a three-judge ensemble and reasoning-before-verdict judging
Harvey's Legal Agent Benchmark (LAB) provides an open harness for testing agents on realistic legal work, and we run LAB-compatible harness infrastructure. LAB's independently run public leaderboard at Vals AI currently reports end-to-end results for Meta, Grok, Claude, GPT, and open-weight models, with a single Flash-class Gemini entry and no Gemini Pro result yet. That gap is part of the motivation for this section: cross-provider evaluation only happens where the harness can actually run every provider, as agent and as judge. Working with LAB taught us that the evaluator is part of the system under test: a structured judge answer can be syntactically valid and still internally inconsistent. We made four evaluator contributions — two filed and submitted upstream (the issues and pull requests linked below), two implemented in our LAB-compatible harness and disclosed here — and the production ensemble — a Gemini judge and a Claude judge on every criterion, with a human adjudicator over every failure and disagreement — depends on all four.
Field order changes the verdict. The original judge schema orders verdict before reasoning, so under structured output the judge commits to a verdict before writing the explanation. In a replication of ten judge calls per arm on one affected criterion — identical prompt, Claude Sonnet 4.6, temperature zero — the verdict-first schema returned fail ten out of ten, several with reasoning that concluded the package passes; with the fields reversed, ten out of ten returned pass, each consistent with its reasoning and with the human audit of the underlying documents. This is a demonstrated effect for this judge model on this criterion; we do not claim it generalizes across providers or criteria. We reported it upstream as a LAB issue (open) and submitted an upstream change (open) that reorders the schema so reasoning is written first. All production results are judged reasoning-first, and we disclose that choice because the schema is part of the measurement instrument.
Deliverable matching is deterministic and audited. Before any judge sees anything, the evaluator decides which agent output file corresponds to each expected deliverable. That matching is deterministic: deliverables are collected in sorted order rather than hash order; fuzzy filename matching may cross file extensions only within a compatible format family; extension-based pairing runs only as a pigeonhole pass; and no output file can be assigned to two deliverables. Every score file records a deliverables_resolved map showing exactly which file the judge saw for each deliverable, so a reviewer can verify the matching without rerunning anything.
Contradictory verdicts go to a human. A fail verdict whose reasoning lacks explicit failure language, or whose reasoning concludes the criterion actually passes, is routed to human review with the raw judge response preserved. There is no automatic retry: re-rolling the judge until the fields agree would hide the failure instead of measuring it.
A Gemini judge required a routing repair. The unpatched harness rejects provider-prefixed judge IDs like google/gemini-3.1-pro-preview before any API call is made. We filed an upstream LAB issue (open) and submitted an upstream change (open). In a side-by-side demonstration on the statutory-notice task, the unpatched path still failed at routing with that error preserved, while the patched Gemini judge completed all eight criteria, a Claude judge control completed all eight, and every arm that ran agreed on every verdict. One caveat we owe readers: we had earlier observed Google's endpoint rejecting a keyword in the judge's structured-output schema, but that rejection did not reproduce on re-test, so the case for the fix rests on the routing repair and on correctly handling the provider's parsed structured response. The goal is not to make one model look better; it is to make cross-provider evaluation possible and inspectable.
Neither upstream change touches LAB's tasks or scoring rubric.
Eval hygiene and a frozen environment
Before the production matrix, each task went through a 13-item adversarial hygiene audit — prompted specifically to find eval defects such as answers leaking into the workspace, criteria a degenerate output could satisfy, and ambiguous grading language — plus a verification of the specific trap each task is built around. All six tasks passed the task-side audit. The audits surfaced low-severity defects on the judging side, and that hardening was applied before any production run rather than after seeing results.
The audits also re-verified the seeded law against primary sources: the Tennessee 2026 act excerpt was checked against the officially enacted text, and the Colorado 2026 earnings thresholds were re-confirmed against the state labor department's published order, matching the workspace excerpts exactly.
Finally, we froze the environment. A single commit SHA pins the tasks, harness, and evaluator; the freeze manifest records a git tree hash for every task plus a SHA-256 hash of every file in every task workspace, the exact model settings, and the judge schema versions. The eighteen-cell task-model matrix was declared in advance under append-only rules: one production run per cell, replays only for human-confirmed failures, results appended to the manifest's run log, nothing overwritten or silently replaced.
How this page differs from other legal evaluations
Independent leaderboards are worth reading alongside the main page precisely because they measure different things: on Vals AI's Legal Research Bench (all-pass grading, updated July 2026), Claude Opus 4.8 leads at 43.75% all-pass, with Claude Sonnet 5 and GPT-5.5 close behind — roughly the reverse of the ordering our institutional-leakage criterion produces. The two measure different genres of legal work. Research benchmarks test open-ended retrieval and reasoning over public law; institutional drafting is closer to grounded generation — work product grounded in a firm's own knowledge base, the skill that grounding evaluations such as Google DeepMind's FACTS Grounding measure — plus a boundary discipline neither genre tests directly: deciding which grounded analysis may cross into the document the counterparty receives.
One finding of that leaderboard points in the same direction as the main page. Its hardest category, for every model tested, is reconciling conflicting authority: 20.7% all-pass against a 28.4% overall rate, with each model scoring lower there than on the rest, while temporal-validity questions sit near the overall rate — synthesizing conflicting sources, not locating a single rule, is where models break down most reliably. That is the shape of institutional work: our tasks hand the model an inherited template, standing client instructions, practice resources, and a change in the law at once, and the graded questions are which authority controls and what may be said in which document. That is why we publish the criterion, the per-instance judge records, and the raw judge output rather than a single score.