OpenAgreements for AI Labs

Abstract

Frontier models have been saturating evaluations: near-perfect needle-in-a-haystack recall at up to 10 million tokens of context (Gemini 1.5 technical report, 2024; Gemini 3 Pro model card, 2025), and LegalBench scores in excess of 80% (Guha et al., 2023; Vals AI leaderboard, 2026). Harvey's Legal Agent Benchmark (LAB, 2026) shows there's still room. For knowledge work, however, a trust gap remains, particularly in institutional contexts. We identify a concrete failure mode: models occasionally leak internal institutional analysis from a firm's knowledge base into outward-facing documents, in particular contracts. We observe sharp stratification across frontier models on this institutional-knowledge-leakage failure mode (1, 4, and 6 leaking instances out of ten across the three models tested). We offer a realistic institutional workspace with a corpus that simulates a firm's internal knowledge base. We also collect expert-correction traces intended for reinforcement-learning environments: training data for mitigating exactly this failure mode.

What this page covers

1. Headline result: sharp stratification in institutional-knowledge leakage (1, 4, and 6 of ten instances)
2. Results: an audited three-model matrix detects errors in keeping institutional knowledge internal
3. Why OpenAgreements is a realistic institutional knowledge base
4. How we configured the test setup
5. Extending Harvey's LAB: upstream contributions enabling Gemini in a three-judge ensemble and reasoning-before-verdict judging
6. Eval hygiene and a frozen environment
7. How OpenAgreements harvests expert corrections naturally in the course of maintaining its knowledge base
8. Future directions

Headline result: sharp stratification in institutional-knowledge leakage (1, 4, and 6 of ten instances)

The ten-instance multi-state covenant-package family extends the audited matrix's multi-state covenant task. Across the first ten task instances, the three models leaked internal institutional analysis into an outward-facing agreement at sharply different rates:

* One instance set is human-adjudicated (the original three-state instance); the other nine are majority-of-three-judges verdicts (Gemini, GPT-5.5, and Claude Sonnet judges) with human adjudication in progress.

These are the results from the first ten instances; we are continuing to run additional instances and will update this page. Each failure is a placement failure: accurate legal analysis written into the document an employee or counterparty receives — drafting advice, statutory threshold math, penalty exposure, or references to the firm's internal resources inside operative agreements.

Results: an audited three-model matrix detects errors in keeping institutional knowledge internal

We ran six tasks against three frontier models — Gemini 3.1 Pro Preview, GPT-5.5, and Claude Opus 4.8 — one production run per cell, inside a frozen environment (a single commit pins the tasks, harness, and evaluator; every workspace file is hash-manifested; results append to a run log and are never overwritten). Every run was judged twice, independently: once by the Gemini judge on the repaired integration path, once by Claude Sonnet 4.6, LAB's default judge — both under the reasoning-first schema, at temperature zero. Every criterion failure and every inter-judge disagreement then went to human adjudication, with the deciding passage quoted and the ruling recorded. Raw judge scores are preserved unchanged; the audited results below are the human-adjudicated layer on top.

* Both judges failed one criterion here (a date-recital reading); human adjudication overturned it as a rubric defect — the criterion required one of two equally natural date readings — and the rubric was corrected as part of the adjudication record. We report it as an evaluator finding, not a model failure.
† One judge failed this cell; human adjudication overturned the judge: the flagged passage is the statutory notification embedded in the agreement, which is exactly how market-standard California forms work.

What the failures were: placement and completeness, not wrong legal conclusions

Every audited failure is a placement or completeness failure, not a wrong legal conclusion:

• GPT-5.5, effective-date task: the drafting was correct — the void covenant came out, on the right statutory ground — but the file rationale never addressed why the client's prior form-approval memo no longer controlled, a required element. Right answer, incomplete explanation.
• GPT-5.5, minimal-edit task: the client email flagged that the covenant ‘may not be enforceable’ but never warned that presenting the void covenant carries a per-worker statutory penalty, and never asked for a decision. For a business reader, the penalty is the difference between ‘we’ll take our chances’ and ‘stop.’
• Claude Opus 4.8, statutory-notice task: the agreement body editorialized about public policy and unenforceability where operative text should simply state the law's operation — legal analysis leaking into a document an employee signs.

Judge quality, measured: three inter-judge splits in 36 judgings, zero verdict-reasoning contradictions

Because every cell was judged twice and adjudicated, the same matrix yields an evaluator readout: 36 judgings produced three inter-judge splits, all on operative-text boundary criteria. Human rulings: the Claude judge recorded one false-pass and one false-fail; the Gemini judge one false-pass. Under the reasoning-first schema there were zero true verdict/reasoning contradictions; the contradiction detector routed four verdicts to human review conservatively (reasoning that reached the right verdict without explicit failure language) — which is the detector doing its job. The recurring fault line is where an operative document may recite law without explaining it — the same boundary practicing lawyers argue about.

Development disclosure: before the production matrix, we ran a single-model development probe of these six tasks with Gemini 3.1 Pro Preview, judged under the harness's original verdict-first schema. That probe surfaced the verdict-ordering defect described below and is retained as development evidence only; every production result above is judged under the reasoning-first schema.

This is a six-task probe, not a leaderboard. We publish it because the mechanism — realistic institutional drafting, deterministic evaluation, disclosed judging, human adjudication with preserved raw output — is the contribution; the sample is small and we say so.

Why OpenAgreements is a realistic institutional knowledge base

Transactional lawyers work inside an institution: inherited templates, standing client instructions, practice resources, and deal point memos, with each matter arriving as deal-specific inputs like a term sheet or a change in the law. The work is to decide what controls, change what must change, preserve what should remain, and keep the legal analysis out of the document the counterparty will sign.

The OpenAgreements library supplies that institution in public form: primary-source-backed practice guides, checklists, and maintained agreement templates across all 50 states, the District of Columbia, and five U.S. territories. The corpus is not eval-only material. It is designed for ordinary practitioner drafting and review, and browser traffic attributable to people and user-directed AI sessions reaches thousands of fetches each month. Eval matters use synthetic facts and no client data, while the same public material that helps a person or agent do the work can be frozen at a known version and used to test whether the resulting work product is correct.

A public corpus invites an obvious objection: a model may have seen it in training. The tasks reduce that risk by drawing on areas of law that change frequently — several of the seeded rules took effect within months of the tasks being written — so producing correct work product requires reading the workspace rather than recalling training data.

How we configured the test setup

Each of the six tasks supplies a workspace of realistic source materials and asks for finished deliverables. Together they test: applying a new law based on the date an agreement will be signed rather than the date printed on an old template; distinguishing two nearby compensation thresholds when a salary falls between them; following a partner's narrow editing instruction while separately surfacing a serious legal issue; refreshing reusable forms from a law-firm-style client alert without copying advisory prose into operative provisions; using an adverse legal memorandum to repair an agreement without disclosing the adverse analysis in the agreement itself; and assembling a California invention-assignment package that includes the written statutory notification the employee must receive.

We report criterion-level results and an all-pass result, under which a task passes only if every required criterion passes. All-pass asks whether work product is ready to deliver; it is not a complete measure of usefulness, so criterion-level results and materiality remain necessary. A polished agreement that misses one legally significant condition is still not finished work product.

Agent runs used one production run per task-model cell at fixed settings. Judging is described in the results section: two independent model judges from different providers, reasoning-first schema, temperature zero, human adjudication of every failure and disagreement.

Extending Harvey's LAB: upstream contributions enabling Gemini in a three-judge ensemble and reasoning-before-verdict judging

Harvey's Legal Agent Benchmark (LAB) provides an open harness for testing agents on realistic legal work, and we run LAB-compatible harness infrastructure. Working with it taught us that the evaluator is part of the system under test: a structured judge answer can be syntactically valid and still internally inconsistent. We made four upstream contributions, and the production ensemble — a Gemini judge and a Claude judge on every criterion, with a human adjudicator over every failure and disagreement — depends on them.

Field order changes the verdict. The original judge schema orders verdict before reasoning, so under structured output the judge commits to a verdict before writing the explanation. In a replication of ten judge calls per arm on one affected criterion — identical prompt, Claude Sonnet 4.6, temperature zero — the verdict-first schema returned fail ten out of ten, several with reasoning that concluded the package passes; with the fields reversed, ten out of ten returned pass, each consistent with its reasoning and with the human audit of the underlying documents. This is a demonstrated effect for this judge model on this criterion; we do not claim it generalizes across providers or criteria. We reported it upstream as a LAB issue (open) and submitted an upstream change (open) that reorders the schema so reasoning is written first. All production results are judged reasoning-first, and we disclose that choice because the schema is part of the measurement instrument.

Deliverable matching is deterministic and audited. Before any judge sees anything, the evaluator decides which agent output file corresponds to each expected deliverable. That matching is deterministic: deliverables are collected in sorted order rather than hash order; fuzzy filename matching may cross file extensions only within a compatible format family; extension-based pairing runs only as a pigeonhole pass; and no output file can be assigned to two deliverables. Every score file records a deliverables_resolved map showing exactly which file the judge saw for each deliverable, so a reviewer can verify the matching without rerunning anything.

Contradictory verdicts go to a human. A fail verdict whose reasoning lacks explicit failure language, or whose reasoning concludes the criterion actually passes, is routed to human review with the raw judge response preserved. There is no automatic retry: re-rolling the judge until the fields agree would hide the failure instead of measuring it.

A Gemini judge can serve at all. The unpatched harness rejects provider-prefixed judge IDs like google/gemini-3.1-pro-preview before any API call is made. We filed an upstream LAB issue (open) and submitted an upstream change (open). In a side-by-side demonstration on the statutory-notice task, the unpatched path still failed at routing with that error preserved, while the patched Gemini judge completed all eight criteria, a Claude judge control completed all eight, and every arm that ran agreed on every verdict. One caveat we owe readers: we had earlier observed Google's endpoint rejecting a keyword in the judge's structured-output schema, but that rejection did not reproduce on re-test, so the case for the fix rests on the routing repair and on correctly handling the provider's parsed structured response. The goal is not to make one model look better; it is to make cross-provider evaluation possible and inspectable.

Eval hygiene and a frozen environment

Before the production matrix, each task went through a 13-item adversarial hygiene audit — prompted specifically to find eval defects such as answers leaking into the workspace, criteria a degenerate output could satisfy, and ambiguous grading language — plus a verification of the specific trap each task is built around. All six tasks passed. The audits surfaced low-severity defects on the judging side, and that hardening was applied before any production run rather than after seeing results.

The audits also re-verified the seeded law against primary sources: the Tennessee 2026 act excerpt was checked against the officially enacted text, and the Colorado 2026 earnings thresholds were re-confirmed against the state labor department's published order, matching the workspace excerpts exactly.

Finally, we froze the environment. A single commit SHA pins the tasks, harness, and evaluator; the freeze manifest records a git tree hash for every task plus a SHA-256 hash of every file in every task workspace, the exact model settings, and the judge schema versions. The eighteen-cell task-model matrix was declared in advance under append-only rules: one production run per cell, replays only for human-confirmed failures, results appended to the manifest's run log, nothing overwritten or silently replaced.

How OpenAgreements harvests expert corrections naturally in the course of maintaining its knowledge base

Maintaining an open legal knowledge base produces paired examples in the ordinary course of work: an AI-assisted proposal, the expert's correction, the resulting diff, and the contemporaneous rationale — including the primary-source law checks the expert ran before deciding. No annotation pipeline manufactures these; the repository's normal review process does, and each correction lands as a structured, attributable record. Here is one representative record from a Massachusetts non-compete maintenance review (the source repository, legal-explainer, is private; we reference its pull requests and issues by number):

{
  "record_id": "ec-2026-07-11-ma-noncompete-pilot",
  "source_repo": "legal-explainer (private)",
  "pr": 1738,
  "issues": [1561, 1718],
  "finding": "MA-T-006 — self-executing statutory limits recited in the operative body",
  "authorship": {
    "mode": "correction_of_ai_output",
    "proposal_model": "claude-fable-5",
    "proposal_commits": ["b335b18b", "bfbf0a2e"],
    "attribution_source": "machine-recorded reasoning trace on every commit",
    "correction_directed_by": "repository owner (technically trained lawyer)"
  },
  "before_ref": "commit bfbf0a2e — operative recitals of the choice-of-law bar and county-venue mandate of Mass. Gen. Laws ch. 149, § 24L(e)-(f)",
  "after_ref": "commit 10c5ef9f — recitals removed; clause keeps governing law plus mutual Suffolk County consent",
  "rationale": "Restating self-executing statutory limits binds the employer contractually even where the statute would not reach; the instrument should not surrender rights the statute leaves intact.",
  "law_check": {
    "question": "Must the instrument recite § 24L(e) and (f)?",
    "answer": "No — both are self-executing; the one venue term the parties can create by contract is mutual county consent.",
    "sources": ["Mass. Gen. Laws ch. 149, § 24L(e)", "Mass. Gen. Laws ch. 149, § 24L(f)"]
  },
  "review_medium": "spoken review entered via speech recognition, 2026-07-11"
}

Note that this record is the failure mode this page is about, caught at the knowledge-base layer: the AI proposal recited legal analysis in the operative body of a form, and the expert's correction moved the boundary back — what the law does of its own force belongs in the guide, not the instrument.

We also record the expert's spoken review and enter it via speech recognition. Spoken deliberation is roughly three times faster than writing, and it captures a data type that is not on the internet: a lawyer discussing a client-shaped issue, checking primary sources live, and landing on an answer.

Future directions

Replay-stability trials for the failing cells (three trials per cell at identical settings) will be appended to the run log and reported alongside the single-run matrix. We plan to publish selected expert-correction traces with frozen task inputs, rubrics, model outputs, and audited evaluations, and to extend the task families. We publish the method early because eval design improves when the tasks, rubrics, and evaluator failures can themselves be criticized.

If you work on legal evaluation, model training, or open legal infrastructure and want to help shape the next iteration, contact hello@openagreements.org.