# OpenAgreements for AI Labs[^about]

Public legal knowledge, drafting evaluations, and expert correction traces for teams building legal AI.

## Abstract {#abstract}

Frontier models have been saturating evaluations: near-perfect needle-in-a-haystack recall at up to 10 million tokens of context[^gemini-1-5-technical-report][^gemini-3-pro-model-card], and LegalBench scores in excess of 80%.[^legalbench-guha-2023][^vals-ai-legalbench-leaderboard] Harvey's Legal Agent Benchmark (LAB, 2026) shows there's still room.[^harvey-lab-initial-results] For knowledge work, however, a trust gap remains, particularly in institutional contexts. We identify a concrete failure mode: models occasionally leak internal institutional analysis from a firm's knowledge base into outward-facing documents, in particular contracts. We observe sharp stratification across frontier models on this institutional-knowledge-leakage failure mode (1, 4, and 6 leaking instances out of ten across the three models tested). We offer a realistic institutional workspace with a corpus that simulates a firm's internal knowledge base. We also collect expert-correction traces intended for reinforcement-learning environments: training data for mitigating exactly this failure mode.

**What this page covers**

1. [Headline result: sharp stratification in institutional-knowledge leakage (1, 4, and 6 of ten instances)](#headline-result-sharp-stratification-in-institutional-knowledge-leakage-1-4-and-6-of-ten-instances)
2. [Results: an audited three-model matrix detects errors in keeping institutional knowledge internal](#results-an-audited-three-model-matrix-detects-errors-in-keeping-institutional-knowledge-internal)
3. [Why OpenAgreements is a realistic institutional knowledge base](#why-openagreements-is-a-realistic-institutional-knowledge-base)
4. [How we configured the test setup](#how-we-configured-the-test-setup)
5. [Extending Harvey's LAB: upstream contributions enabling Gemini in a three-judge ensemble and reasoning-before-verdict judging](#extending-harvey-s-lab-upstream-contributions-enabling-gemini-in-a-three-judge-ensemble-and-reasoning-before-verdict-judging)
6. [Eval hygiene and a frozen environment](#eval-hygiene-and-a-frozen-environment)
7. [How OpenAgreements harvests expert corrections naturally in the course of maintaining its knowledge base](#how-openagreements-harvests-expert-corrections-naturally-in-the-course-of-maintaining-its-knowledge-base)
8. [Future directions](#future-directions)

## Headline result: sharp stratification in institutional-knowledge leakage (1, 4, and 6 of ten instances) {#headline-result-sharp-stratification-in-institutional-knowledge-leakage-1-4-and-6-of-ten-instances}

Across the first ten instances of the multi-state covenant-package family, the three models leaked internal institutional analysis into an outward-facing agreement at sharply different rates:

| Model | Leaked internal analysis into an outward-facing agreement\* |
| --- | --- |
| Gemini 3.1 Pro Preview | 1 of 10 instances |
| GPT-5.5 | 4 of 10 |
| Claude Opus 4.8 | 6 of 10 |

\* One instance set is human-adjudicated (the original three-state instance); the other nine are majority-of-three-judges verdicts (Gemini, GPT-5.5, and Claude Sonnet judges) with human adjudication in progress.

Per-instance verdicts (family order; fails are placement failures):

| Instance | States | Gemini 3.1 Pro Preview | GPT-5.5 | Claude Opus 4.8 | Adjudication |
| --- | --- | --- | --- | --- | --- |
| orig | California · Colorado · Texas | clean | **leaked** | **leaked** | human-adjudicated |
| v01 | Washington · Illinois · Georgia | clean | clean | **leaked** | judge-majority (adjudication in progress) |
| v02 | Minnesota · Florida · Ohio | clean | clean | clean | judge-majority (adjudication in progress) |
| v03 | Massachusetts · Nevada · Virginia | clean | clean | **leaked** | judge-majority (adjudication in progress) |
| v04 | Oregon · Louisiana · North Carolina | clean | clean | clean | judge-majority (adjudication in progress) |
| v05 | Tennessee · Arizona · New Jersey | clean | **leaked** | **leaked** | judge-majority (adjudication in progress) |
| v06 | Oklahoma · North Dakota · New York | clean | clean | clean | judge-majority (adjudication in progress) |
| v07 | Maine · Wisconsin · Michigan | clean | **leaked** | **leaked** | judge-majority (adjudication in progress) |
| v08 | New Hampshire · Alabama · Utah | **leaked** | **leaked** | **leaked** | judge-majority (adjudication in progress) |
| v09 | Maryland · Indiana · Missouri | clean | clean | clean | judge-majority (adjudication in progress) |

A note on the two denominators on this page. The [audited matrix below](#results-an-audited-three-model-matrix-detects-errors-in-keeping-institutional-knowledge-internal) scores six distinct tasks on all of their criteria — out of six. This section takes a seventh drafting task, the multi-state covenant package, and replicates it as a family: ten instances across different state triplets, each scored on the single criterion where the models stratify — whether internal institutional analysis stays out of the agreements an employee will sign — out of ten. One instance is human-adjudicated (the original); the other nine are majority-of-three-judges verdicts (Gemini, GPT-5.5, and Claude Sonnet judges) with human adjudication in progress. Every square above is inspectable: it opens the instance's states, its disposition, and the audit detail available for that cell — the nine variant instances show all three judges' verdicts, the original instance shows its human-adjudicated disposition, and each leaking cell's note quotes or closely paraphrases the majority judge's reasoning or the adjudication record.

These are the results from the first ten instances; we are continuing to run additional instances and will update this page. Each failure is a placement failure: accurate legal analysis written into the document an employee or counterparty receives — drafting advice, statutory threshold math, penalty exposure, or references to the firm's internal resources inside operative agreements.

## Results: an audited three-model matrix detects errors in keeping institutional knowledge internal {#results-an-audited-three-model-matrix-detects-errors-in-keeping-institutional-knowledge-internal}

We ran six tasks against three frontier models — Gemini 3.1 Pro Preview, GPT-5.5, and Claude Opus 4.8 — one production run per cell, inside a frozen environment (a single commit pins the tasks, harness, and evaluator; every workspace file is hash-manifested; results append to a run log and are never overwritten). Every run was judged twice, independently: once by the Gemini judge on the repaired integration path, once by Claude Sonnet 4.6, LAB's default judge — both under the reasoning-first schema, at temperature zero. Every criterion failure and every inter-judge disagreement then went to human adjudication, with the deciding passage quoted and the ruling recorded. Raw judge scores are preserved unchanged; the audited results below are the human-adjudicated layer on top.

| Task | Gemini 3.1 Pro Preview | GPT-5.5 | Claude Opus 4.8 |
| --- | --- | --- | --- |
| Multi-state term-sheet implementation | pass | pass | pass\* |
| Effective-date statutory straddle | pass | fail (C-004) | pass |
| Instructed minimal edit with duty to flag | pass | fail (C-005) | pass |
| Client-alert form refresh | pass | pass | pass |
| Adverse-memo agreement revision | pass | pass | pass |
| Statutory notice package | pass | pass† | fail (C-007) |
| **Audited tasks passed** | **6/6** | **4/6** | **5/6** |

\* Both judges failed one criterion here (a date-recital reading); human adjudication overturned it as a rubric defect — the criterion required one of two equally natural date readings — and the rubric was corrected as part of the adjudication record. We report it as an evaluator finding, not a model failure.\
† One judge failed this cell; human adjudication overturned the judge: the flagged passage is the statutory notification embedded in the agreement, which is exactly how market-standard California forms work.

This matrix's denominator is six distinct tasks — not the ten instances in the [headline section](#headline-result-sharp-stratification-in-institutional-knowledge-leakage-1-4-and-6-of-ten-instances), which replicate a seventh, separately audited covenant-package task. Here a task passes only if every one of its criteria passes, so a 6/6 on this matrix and a leak in the ten-instance family measure different things and can coexist.

### What the failures were: placement and completeness, not wrong legal conclusions {#what-the-failures-were-placement-and-completeness-not-wrong-legal-conclusions}

Every audited failure is a *placement or completeness* failure, not a wrong legal conclusion:

- **GPT-5.5, effective-date task:** the drafting was correct — the void covenant came out, on the right statutory ground — but the file rationale never addressed why the client's prior form-approval memo no longer controlled, a required element. Right answer, incomplete explanation.
- **GPT-5.5, minimal-edit task:** the client email flagged that the covenant ‘may not be enforceable’ but never warned that *presenting* the void covenant carries a per-worker statutory penalty, and never asked for a decision. For a business reader, the penalty is the difference between ‘we’ll take our chances’ and ‘stop.’
- **Claude Opus 4.8, statutory-notice task:** the agreement body editorialized about public policy and unenforceability where operative text should simply state the law's operation — legal analysis leaking into a document an employee signs.

### Replay stability: the audited failures are intermittent, not deterministic {#replay-stability-the-audited-failures-are-intermittent-not-deterministic}

Each audited failing cell was replayed to three total trials at identical settings (same frozen environment, temperature zero), each replay judged by both judges. No failure reproduced deterministically:

| Failing cell | Same-criterion failure | Label |
| --- | --- | --- |
| Effective-date straddle · GPT-5.5 (C-004) | 1 of 3 trials | intermittent |
| Instructed minimal edit · GPT-5.5 (C-005) | 2 of 3 | intermittent |
| Statutory notice package · Claude Opus 4.8 (C-007) | 1 of 3 | intermittent |

Counting rule: a trial counts as failing if its human-adjudicated verdict (where one exists) or both-judge verdict is fail. Judge-level detail is preserved in the run log: the C-005 count is 2 of 3 under the Gemini judge and 1 of 3 under the Sonnet judge (one inter-judge disagreement is in human adjudication), and the C-007 original is the human-adjudicated failure reported above while both of its replays passed under both judges. Temperature zero does not make agent runs deterministic, so single-run cells are point samples — which is why the [headline result](#headline-result-sharp-stratification-in-institutional-knowledge-leakage-1-4-and-6-of-ten-instances) rests on a ten-instance family rather than any single cell, and why every replay is appended to the same run log rather than replacing anything.

### Judge quality, measured: three inter-judge splits in 36 judgings, zero verdict-reasoning contradictions {#judge-quality-measured-three-inter-judge-splits-in-36-judgings-zero-verdict-reasoning-contradictions}

Because every cell was judged twice and adjudicated, the same matrix yields an evaluator readout: 36 judgings produced three inter-judge splits, all on operative-text boundary criteria. Human rulings: the Claude judge recorded one false-pass and one false-fail; the Gemini judge one false-pass. Under the reasoning-first schema there were zero true verdict/reasoning contradictions; the contradiction detector routed four verdicts to human review conservatively (reasoning that reached the right verdict without explicit failure language) — which is the detector doing its job. The recurring fault line is where an operative document may recite law without explaining it — the same boundary practicing lawyers argue about.

Development disclosure: before the production matrix, we ran a single-model development probe of these six tasks with Gemini 3.1 Pro Preview, judged under the harness's original verdict-first schema. That probe surfaced the verdict-ordering defect described [below](#extending-harvey-s-lab-upstream-contributions-enabling-gemini-in-a-three-judge-ensemble-and-reasoning-before-verdict-judging) and is retained as development evidence only; every production result above is judged under the reasoning-first schema.

This is a six-task probe, not a leaderboard. We publish it because the mechanism — realistic institutional drafting, deterministic evaluation, disclosed judging, human adjudication with preserved raw output — is the contribution; the sample is small and we say so.

## Why OpenAgreements is a realistic institutional knowledge base {#why-openagreements-is-a-realistic-institutional-knowledge-base}

Transactional lawyers work inside an institution: inherited templates, standing client instructions, practice resources, and deal point memos, with each matter arriving as deal-specific inputs like a term sheet or a change in the law. The work is to decide what controls, change what must change, preserve what should remain, and keep the legal analysis out of the document the counterparty will sign.

The [OpenAgreements](https://github.com/open-agreements/open-agreements) library supplies that institution in public form: primary-source-backed practice guides, checklists, and maintained agreement templates across all 50 states, the District of Columbia, and five U.S. territories. The corpus is not eval-only material. It is designed for ordinary practitioner drafting and review, and browser traffic attributable to people and user-directed AI sessions reaches thousands of fetches each month. Eval matters use synthetic facts and no client data, while the same public material that helps a person or agent do the work can be frozen at a known version and used to test whether the resulting work product is correct.

A public corpus invites an obvious objection: a model may have seen it in training. The tasks reduce that risk by drawing on areas of law that change frequently — several of the seeded rules took effect within months of the tasks being written — so producing correct work product requires reading the workspace rather than recalling training data.

## How we configured the test setup {#how-we-configured-the-test-setup}

Each of the six tasks supplies a workspace of realistic source materials and asks for finished deliverables. Together they test: applying a new law based on the date an agreement will be signed rather than the date printed on an old template; distinguishing two nearby compensation thresholds when a salary falls between them; following a partner's narrow editing instruction while separately surfacing a serious legal issue; refreshing reusable forms from a law-firm-style client alert without copying advisory prose into operative provisions; using an adverse legal memorandum to repair an agreement without disclosing the adverse analysis in the agreement itself; and assembling a California invention-assignment package that includes the written statutory notification the employee must receive.

We report criterion-level results and an all-pass result, under which a task passes only if every required criterion passes. All-pass asks whether work product is ready to deliver; it is not a complete measure of usefulness, so criterion-level results and materiality remain necessary. A polished agreement that misses one legally significant condition is still not finished work product.

Agent runs used one production run per task-model cell at fixed settings. Judging is described in the [results section](#results-an-audited-three-model-matrix-detects-errors-in-keeping-institutional-knowledge-internal): two independent model judges from different providers, reasoning-first schema, temperature zero, human adjudication of every failure and disagreement.

## Extending Harvey's LAB: upstream contributions enabling Gemini in a three-judge ensemble and reasoning-before-verdict judging {#extending-harvey-s-lab-upstream-contributions-enabling-gemini-in-a-three-judge-ensemble-and-reasoning-before-verdict-judging}

[Harvey's Legal Agent Benchmark (LAB)](https://github.com/harveyai/harvey-labs) provides an open harness for testing agents on realistic legal work, and we run LAB-compatible harness infrastructure. Working with it taught us that the evaluator is part of the system under test: a structured judge answer can be syntactically valid and still internally inconsistent. We made four upstream contributions, and the production ensemble — a Gemini judge and a Claude judge on every criterion, with a human adjudicator over every failure and disagreement — depends on them.

**Field order changes the verdict.** The original judge schema orders `verdict` before `reasoning`, so under structured output the judge commits to a verdict before writing the explanation. In a replication of ten judge calls per arm on one affected criterion — identical prompt, Claude Sonnet 4.6, temperature zero — the verdict-first schema returned `fail` ten out of ten, several with reasoning that concluded the package passes; with the fields reversed, ten out of ten returned `pass`, each consistent with its reasoning and with the human audit of the underlying documents. This is a demonstrated effect for this judge model on this criterion; we do not claim it generalizes across providers or criteria. We reported it upstream as [a LAB issue](https://github.com/harveyai/harvey-labs/issues/106) (open) and submitted [an upstream change](https://github.com/harveyai/harvey-labs/pull/105) (open) that reorders the schema so reasoning is written first. All production results are judged reasoning-first, and we disclose that choice because the schema is part of the measurement instrument.

**Deliverable matching is deterministic and audited.** Before any judge sees anything, the evaluator decides which agent output file corresponds to each expected deliverable. That matching is deterministic: deliverables are collected in sorted order rather than hash order; fuzzy filename matching may cross file extensions only within a compatible format family; extension-based pairing runs only as a pigeonhole pass; and no output file can be assigned to two deliverables. Every score file records a `deliverables_resolved` map showing exactly which file the judge saw for each deliverable, so a reviewer can verify the matching without rerunning anything.

**Contradictory verdicts go to a human.** A `fail` verdict whose reasoning lacks explicit failure language, or whose reasoning concludes the criterion actually passes, is routed to human review with the raw judge response preserved. There is no automatic retry: re-rolling the judge until the fields agree would hide the failure instead of measuring it.

**A Gemini judge can serve at all.** The unpatched harness rejects provider-prefixed judge IDs like `google/gemini-3.1-pro-preview` before any API call is made. We filed [an upstream LAB issue](https://github.com/harveyai/harvey-labs/issues/104) (open) and submitted [an upstream change](https://github.com/harveyai/harvey-labs/pull/100) (open). In a side-by-side demonstration on the statutory-notice task, the unpatched path still failed at routing with that error preserved, while the patched Gemini judge completed all eight criteria, a Claude judge control completed all eight, and every arm that ran agreed on every verdict. One caveat we owe readers: we had earlier observed Google's endpoint rejecting a keyword in the judge's structured-output schema, but that rejection did not reproduce on re-test, so the case for the fix rests on the routing repair and on correctly handling the provider's parsed structured response. The goal is not to make one model look better; it is to make cross-provider evaluation possible and inspectable.

## Eval hygiene and a frozen environment {#eval-hygiene-and-a-frozen-environment}

Before the production matrix, each task went through a 13-item adversarial hygiene audit — prompted specifically to find eval defects such as answers leaking into the workspace, criteria a degenerate output could satisfy, and ambiguous grading language — plus a verification of the specific trap each task is built around. All six tasks passed. The audits surfaced low-severity defects on the judging side, and that hardening was applied before any production run rather than after seeing results.

The audits also re-verified the seeded law against primary sources: the Tennessee 2026 act excerpt was checked against the officially enacted text, and the Colorado 2026 earnings thresholds were re-confirmed against the state labor department's published order, matching the workspace excerpts exactly.

Finally, we froze the environment. A single commit SHA pins the tasks, harness, and evaluator; the freeze manifest records a git tree hash for every task plus a SHA-256 hash of every file in every task workspace, the exact model settings, and the judge schema versions. The eighteen-cell task-model matrix was declared in advance under append-only rules: one production run per cell, replays only for human-confirmed failures, results appended to the manifest's run log, nothing overwritten or silently replaced.

## How OpenAgreements harvests expert corrections naturally in the course of maintaining its knowledge base {#how-openagreements-harvests-expert-corrections-naturally-in-the-course-of-maintaining-its-knowledge-base}

Maintaining an open legal knowledge base produces paired examples in the ordinary course of work: an AI-assisted proposal, the expert's correction, the resulting diff, and the contemporaneous rationale — including the primary-source law checks the expert ran before deciding. No annotation pipeline manufactures these; the repository's normal review process does, and each correction lands as a structured, attributable record. Here is one representative record from a Massachusetts non-compete maintenance review (the source repository, legal-explainer, is private; we reference its pull requests and issues by number):

```json
{
  "record_id": "ec-2026-07-11-ma-noncompete-pilot",
  "source_repo": "legal-explainer (private)",
  "pr": 1738,
  "issues": [1561, 1718],
  "finding": "MA-T-006 — self-executing statutory limits recited in the operative body",
  "authorship": {
    "mode": "correction_of_ai_output",
    "proposal_model": "claude-fable-5",
    "proposal_commits": ["b335b18b", "bfbf0a2e"],
    "attribution_source": "machine-recorded reasoning trace on every commit",
    "correction_directed_by": "repository owner (technically trained lawyer)"
  },
  "before_ref": "commit bfbf0a2e — operative recitals of the choice-of-law bar and county-venue mandate of Mass. Gen. Laws ch. 149, § 24L(e)-(f)",
  "after_ref": "commit 10c5ef9f — recitals removed; clause keeps governing law plus mutual Suffolk County consent",
  "rationale": "Restating self-executing statutory limits binds the employer contractually even where the statute would not reach; the instrument should not surrender rights the statute leaves intact.",
  "law_check": {
    "question": "Must the instrument recite § 24L(e) and (f)?",
    "answer": "No — both are self-executing; the one venue term the parties can create by contract is mutual county consent.",
    "sources": ["Mass. Gen. Laws ch. 149, § 24L(e)", "Mass. Gen. Laws ch. 149, § 24L(f)"]
  },
  "review_medium": "spoken review entered via speech recognition, 2026-07-11"
}
```

Note that this record is the failure mode this page is about, caught at the knowledge-base layer: the AI proposal recited legal analysis in the operative body of a form, and the expert's correction moved the boundary back — what the law does of its own force belongs in the guide, not the instrument.

Below are two more findings from the same Massachusetts review, as full layered traces. The leftmost tab of each finding is the human expert correction: a word-level legal redline computed on this page from the embedded AI-proposed and expert-corrected text, with the expert's reasoning attached in the margin. The supporting layers behind it — the dictated review (paraphrased from speech recognition), the machine-readable JSON record, the AI-summarized decision, and the primary-source check — sit in the remaining tabs. In the second finding, the primary-source check answered two questions the expert asked before signing off, and one answer corrected the expert's own first instinct.

Derived from Git — structured `correction_of_ai_output` commit messages.

**Finding MA-T-003 — Excluded workers: implied obligation made express** (NEEDED CORRECTION)

Massachusetts non-compete template · excluded-worker carve-out, § 24L(c).

Computed word-level redline (`~~strikethrough~~` = removed by the expert correction, `**++bold++**` = added):

> The noncompetition covenant in Section 2 does not apply to Employee if, as of the date Employee’s employment ends, Employee is within a category of workers as to whom noncompetition agreements are unenforceable under Section ~~24L(c), and Employer will not enforce it against Employee.~~ **++24L(c).++**

**Expert rationale (AI addition rejected).** If the covenant does not apply, non-enforcement follows. The AI’s added promise made an implied obligation express, creating an obligation against the form’s user with no validity gain. ‘Don’t draft the counterparty’s defenses into the instrument.’ (The AI’s removal of the independent-contractor recital, a separate part of the same proposal, was confirmed.)

**Human dictated review (paraphrased from speech recognition).**

> “Dropping the independent-contractor recital — agreed.”
>
> “But no to ‘and Employer will not enforce it against Employee.’ If the covenant doesn’t apply, non-enforcement follows. That’s an express obligation that’s already implied — we’d be drafting the counterparty’s defense into our own form. Take it out.”

**AI-summarized decision (verified by the expert).** The AI’s proposed removal of the independent-contractor recital is confirmed. The AI’s added promise, ‘and Employer will not enforce it against Employee,’ is rejected and removed: it creates an obligation against the form’s user with no validity gain. (Affected requirement: `REQ-…massachusetts.exclude-non-competes-for-excluded-worker-categories`.)

No primary source check was requested for this finding: the correction rests on drafting judgment (implied vs. express obligations), not a point of law.

Machine-readable record:

```json
{
  "record_id": "ec-2026-07-11-ma-noncompete-pilot",
  "repo": "legal-explainer (private)",
  "pr": 1738,
  "branch": "pilot/ma-expert-correction",
  "finding": "MA-T-003",
  "topic": "excluded-worker carve-out — implied obligation made express",
  "disposition": "NEEDS_CORRECTION",
  "affected_requirement": "REQ-…massachusetts.exclude-non-competes-for-excluded-worker-categories",
  "authorship": {
    "proposal": {
      "model": "claude-fable-5",
      "tool": "Claude Code",
      "commit": "bfbf0a2e"
    },
    "correction": {
      "authorship_mode": "correction_of_ai_output",
      "model": "claude-fable-5",
      "directed_by": "repository owner (technically trained lawyer)",
      "commit": "10c5ef9f"
    }
  },
  "review_capture": "voice transcript, 2026-07-11 (speech recognition; paraphrased)",
  "law_check": "none_requested — drafting judgment, not a point of law",
  "confirmed_element": "removal of the independent-contractor recital (AI proposal confirmed)",
  "rejected_element": "‘and Employer will not enforce it against Employee’ (AI addition)"
}
```

**Finding MA-T-001 — Garden-leave consideration: drafting posture** (NEEDED CORRECTION)

Massachusetts non-compete template · Mass. Gen. Laws ch. 149, § 24L.

Computed word-level redline (`~~strikethrough~~` = removed by the expert correction, `**++bold++**` = added):

> During the Restricted Period, **++if the Cover Terms specify garden leave,++** Employer ~~will~~ **++shall++** pay Employee garden-leave consideration equal to at least fifty percent of Employee’s highest annualized base salary within the two years preceding termination, paid on a pro-rata basis over the Restricted Period. **++This payment obligation becomes effective upon cessation of employment unless Employer waives the noncompetition covenant in writing, and, except in the event of a breach by Employee, Employer shall not unilaterally discontinue or otherwise fail or refuse to make the payments.++** If the Cover Terms specify other mutually agreed-upon consideration, ~~Employer will provide Employee~~ **++the parties stipulate++** that ~~consideration.~~ **++such consideration supports the covenant.++**

**Expert rationale.** An employer-presented form shouldn’t volunteer affirmative obligations the law doesn’t require, so the consideration is recast as a conditional stipulation of the Cover Terms consideration. The open-ended ‘other consideration → Employer will provide it’ promise was a blank check (other than what? no ascertainable trigger); it is deleted, and the parties can always amend.

**Human dictated review (paraphrased from speech recognition).**

> “This should be a conditional stipulation, not an employer obligation. An employer-presented form shouldn’t volunteer affirmative promises the law doesn’t require.”
>
> “And delete the ‘other mutually-agreed consideration → Employer will provide Employee that consideration’ sentence. Other than what? It’s a blank check — and unnecessary, the parties can always amend.”
>
> “Two things to check: does §24L require the consideration to be specified in the agreement, or merely paid? And can we structure it so the covenant simply lapses if the employer stops paying?”

**AI-summarized decision (verified by the expert).** Redraft as a conditional stipulation of the Cover Terms consideration. The garden-leave branch keeps the statutory payment promise, effective on cessation of employment unless Employer waives the non-compete in writing, with the no-unilateral-discontinuance term and its employee-breach exception. The open-ended ‘other consideration’ performance promise is deleted. (Affected requirement: `REQ-…massachusetts.require-garden-leave-or-agreed-consideration`.)

**Primary source check.** The expert asked two questions before signing off: does § 24L require the consideration to be specified in the agreement, or merely paid? And may the covenant simply lapse if the employer stops paying? Both were answered against the statutory text below; the second answer corrected the expert’s first instinct.

- **Mass. Gen. Laws ch. 149, § 24L(b)(vii)** — Section 24L(b)(vii) supports the rule that the consideration must be specified in the noncompetition agreement itself, and that a qualifying garden leave clause may not let the employer unilaterally discontinue the payments except on employee breach. “The noncompetition agreement shall be supported by a garden leave clause or other mutually-agreed upon consideration between the employer and the employee, provided that such consideration is specified in the noncompetition agreement. To constitute a garden leave clause within the meaning of this section, the agreement must (i) provide for the payment, consistent with the requirements for the payment of wages under section 148 of chapter 149 of the general laws, on a pro-rata basis during the entirety of the restricted period, of at least 50 percent of the employee’s highest annualized base salary paid by the employer within the 2 years preceding the employee’s termination; and (ii) except in the event of a breach by the employee, not permit an employer to unilaterally discontinue or otherwise fail or refuse to make the payments” (<https://malegislature.gov/Laws/GeneralLaws/PartI/TitleXXI/Chapter149/Section24L>)
- **Mass. Gen. Laws ch. 149, § 24L(a) (garden leave clause definition)** — Section 24L(a) supports the rule that the statute’s employer off-ramp is waiver: the garden-leave payment provision becomes effective upon termination of employment unless the employer waives the restriction. ““Garden leave clause”, a provision within a noncompetition agreement by which an employer agrees to pay the employee during the restricted period, provided that such provision shall become effective upon termination of employment unless the restriction upon post-employment activities are waived by the employer or ineffective under subsection (c)(iii).” (<https://malegislature.gov/Laws/GeneralLaws/PartI/TitleXXI/Chapter149/Section24L>)

Quoted verbatim from the official General Laws text (malegislature.gov); the L.1 quote ends where the statutory proviso on breach-extended restricted periods begins.

Machine-readable record:

```json
{
  "record_id": "ec-2026-07-11-ma-noncompete-pilot",
  "repo": "legal-explainer (private)",
  "pr": 1738,
  "branch": "pilot/ma-expert-correction",
  "finding": "MA-T-001",
  "topic": "garden-leave consideration — drafting posture",
  "disposition": "NEEDS_CORRECTION",
  "affected_requirement": "REQ-…massachusetts.require-garden-leave-or-agreed-consideration",
  "authorship": {
    "proposal": {
      "model": "claude-fable-5",
      "tool": "Claude Code",
      "commit": "bfbf0a2e"
    },
    "correction": {
      "authorship_mode": "correction_of_ai_output",
      "model": "claude-fable-5",
      "directed_by": "repository owner (technically trained lawyer)",
      "commit": "10c5ef9f"
    }
  },
  "review_capture": "voice transcript, 2026-07-11 (speech recognition; paraphrased)",
  "law_check": {
    "requested_by": "expert",
    "questions": [
      "does §24L require the consideration to be specified in the agreement, or merely paid?",
      "may the covenant lapse if the employer stops paying?"
    ],
    "answers": [
      "§24L(b)(vii): the consideration must be ‘specified in the noncompetition agreement’",
      "no — unilateral discontinuance is barred except on employee breach; the employer off-ramp is the §24L(a) waiver"
    ],
    "corrected_expert_instinct": true,
    "sources": [
      "Mass. Gen. Laws ch. 149, § 24L(a)",
      "Mass. Gen. Laws ch. 149, § 24L(b)(vii)"
    ]
  },
  "rejected_element": null
}
```

We also record the expert's spoken review and enter it via speech recognition. Spoken deliberation is roughly three times faster than writing, and it captures a data type that is not on the internet: a lawyer discussing a client-shaped issue, checking primary sources live, and landing on an answer.

## Future directions {#future-directions}

We plan to publish selected expert-correction traces with frozen task inputs, rubrics, model outputs, and audited evaluations; to publish the raw run data (transcripts, raw judge output, and adjudication records, with the layers kept separate); and to extend the task families, including further tranches of the ten-instance covenant-package family. We publish the method early because eval design improves when the tasks, rubrics, and evaluator failures can themselves be criticized.

If you work on legal evaluation, model training, or open legal infrastructure and want to help shape the next iteration, contact [hello@openagreements.org](mailto:hello@openagreements.org).


[^about]: By Steven Obiajulu, J.D. Published by [openagreements.org](https://openagreements.org). Last reviewed 2026-07-12. License: CC BY 4.0. Steven Obiajulu, J.D. wrote this essay. It states the author's views, synthesizes public sources, and is not legal advice. This article is for informational purposes only and does not create an attorney-client relationship. CC BY 4.0. Cite as Steven Obiajulu, *OpenAgreements for AI Labs*, OpenAgreements (last updated July 12, 2026), https://openagreements.org/for-labs.

[^gemini-1-5-technical-report]: **Gemini 1.5 technical report (2024)** — "Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities" *Gemini Team, Google, Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context, arXiv:2403.05530 (2024).* <https://arxiv.org/abs/2403.05530>

[^gemini-3-pro-model-card]: **Gemini 3 Pro model card (2025)** — "Gemini 3 Pro is now Google’s most advanced model for complex tasks, and can comprehend vast datasets and challenging problems from different information sources, including text, audio, images, video, and entire code repositories." *Google, Gemini 3 Pro Model Card (released Nov. 2025; updated May 2026).* <https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Pro-Model-Card.pdf>

[^legalbench-guha-2023]: **LegalBench (Guha et al., 2023)** — "LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals" *Neel Guha et al., LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models, arXiv:2308.11462 (2023).* <https://arxiv.org/abs/2308.11462>

[^vals-ai-legalbench-leaderboard]: **Vals AI LegalBench leaderboard (2026)** — "Evaluating language models on a wide range of open source legal reasoning tasks." *Vals AI, LegalBench leaderboard (last updated July 9, 2026).* <https://www.vals.ai/benchmarks/legal_bench>

[^harvey-lab-initial-results]: **Harvey Legal Agent Benchmark, initial results (2026)** — "Under our strict all-pass standard, the frontier models we evaluated complete less than 10% of tasks end-to-end in aggregate." *Harvey, Initial Results on Legal Agent Benchmark (May 26, 2026).* <https://www.harvey.ai/blog/legal-agent-benchmark-initial-results>