Abstract
Reinforcement learning needs expert corrections gathered where frontier AI models actually fail, and saturated question-answer benchmarks no longer supply them. Independent evaluation points at the gap: on Vals AI's Legal Research Bench, reconciling conflicting authority is the lowest-scoring category for every model tested, at 20.7% all-pass against a 28.4% overall rate. Maintaining the OpenAgreements legal knowledge base produces this data in the ordinary course of work. Frontier models draft practice guides and templates that receive thousands of fetches each month, and a lawyer reviews the drafts before publication, so expert corrections accrue as a byproduct rather than an annotation project. Each trace is captured at the time of entry from a Git-based workflow: the machine-recorded diff between the AI proposal and the expert correction, the expert's dictated rationale, the supporting primary-source citations, and the authorities considered and not applied, the reconciliation judgment the leaderboard finds hardest. The corrections concentrate on judgment rather than fact: what may cross into a counterparty-facing document, which authority controls an inherited form, when a statute's self-executing operation belongs in a guide rather than an instrument. As of July 12, 2026, the corpus holds eight structured correction records from one Massachusetts non-compete maintenance review (seven substantive corrections and one process finding), with the capture pipeline running on every subsequent review. A complete two-finding sample trace, layered from redline to machine-readable record, is included below.
Record schema and sample trace
Below are two example expert-correction traces produced by our workflow.
Derived from structured Git commit messages labeled correction_of_ai_output.
Excluded workers: implied obligation made express
Massachusetts non-compete template · excluded-worker carve-out, § 24L(c)
The noncompetition covenant in Section 2 does not apply to Employee if, as of the date Employee’s employment ends, Employee is within a category of workers as to whom noncompetition agreements are unenforceable under Section 24L(c), and Employer will not enforce it against Employee. 24L(c).
This excerpt scrolls. More below ↓
AI addition rejected
If the covenant does not apply, non-enforcement follows. The AI’s added promise made an implied obligation express, creating an obligation against the form’s user with no validity gain. ‘Don’t draft the counterparty’s defenses into the instrument.’ (The AI’s removal of the independent-contractor recital, a separate part of the same proposal, was confirmed.)
“Dropping the independent-contractor recital — agreed.”
“But no to ‘and Employer will not enforce it against Employee.’ If the covenant doesn’t apply, non-enforcement follows. That’s an express obligation that’s already implied — we’d be drafting the counterparty’s defense into our own form. Take it out.”
Paraphrased from the expert’s spoken review (captured via speech recognition); curated verbatim transcript available on request.
{
"record_id": "ec-2026-07-11-ma-noncompete-pilot",
"repo": "legal-explainer",
"repo_visibility": "private",
"pr": 1738,
"branch": "pilot/ma-expert-correction",
"finding": "MA-T-003",
"topic": "excluded-worker carve-out: implied obligation made express",
"disposition": "NEEDS_CORRECTION",
"affected_requirement": "REQ-…massachusetts.exclude-non-competes-for-excluded-worker-categories",
"rationale": "If the covenant does not apply, non-enforcement follows. The AI’s added promise made an implied obligation express, creating an obligation against the form’s user with no validity gain. ‘Don’t draft the counterparty’s defenses into the instrument.’ (The AI’s removal of the independent-contractor recital, a separate part of the same proposal, was confirmed.)",
"authorship": {
"proposal": {
"model": "claude-fable-5",
"tool": "Claude Code",
"commit": "bfbf0a2e"
},
"correction": {
"authorship_mode": "correction_of_ai_output",
"model": "claude-fable-5",
"directed_by": "repository_owner",
"reviewer_background": "technically_trained_lawyer",
"commit": "10c5ef9f"
}
},
"diff_inputs": {
"before_commit": "bfbf0a2e",
"before_role": "ai_proposal",
"after_commit": "10c5ef9f",
"after_role": "expert_correction"
},
"dictated_review_ref": {
"medium": "voice_transcript",
"date": "2026-07-11",
"fidelity": "paraphrased_from_speech_recognition",
"verbatim_transcript": "available_on_request"
},
"law_check": {
"requested": false,
"reason": "drafting judgment, not a point of law"
},
"confirmed_element": "removal of the independent-contractor recital",
"confirmed_element_source": "ai_proposal",
"rejected_element": "‘and Employer will not enforce it against Employee’",
"rejected_element_source": "ai_addition"
}Garden-leave consideration: drafting posture
Massachusetts non-compete template · Mass. Gen. Laws ch. 149, § 24L
During the Restricted Period, if the Cover Terms specify garden leave, Employer will shall pay Employee garden-leave consideration equal to at least fifty percent of Employee’s highest annualized base salary within the two years preceding termination, paid on a pro-rata basis over the Restricted Period. This payment obligation becomes effective upon cessation of employment unless Employer waives the noncompetition covenant in writing, and, except in the event of a breach by Employee, Employer shall not unilaterally discontinue or otherwise fail or refuse to make the payments. If the Cover Terms specify other mutually agreed-upon consideration, Employer will provide Employee the parties stipulate that consideration. such consideration supports the covenant.
This excerpt scrolls. More below ↓
An employer-presented form shouldn’t volunteer affirmative obligations the law doesn’t require, so the consideration is recast as a conditional stipulation of the Cover Terms consideration. The open-ended ‘other consideration → Employer will provide it’ promise was a blank check (other than what? no ascertainable trigger); it is deleted, and the parties can always amend.
“This should be a conditional stipulation, not an employer obligation. An employer-presented form shouldn’t volunteer affirmative promises the law doesn’t require.”
“And delete the ‘other mutually-agreed consideration → Employer will provide Employee that consideration’ sentence. Other than what? It’s a blank check — and unnecessary, the parties can always amend.”
“Two things to check: does §24L require the consideration to be specified in the agreement, or merely paid? And can we structure it so the covenant simply lapses if the employer stops paying?”
Paraphrased from the expert’s spoken review (captured via speech recognition); curated verbatim transcript available on request.
The expert asked two questions before signing off: does § 24L require the consideration to be specified in the agreement, or merely paid? And may the covenant simply lapse if the employer stops paying? Both were answered against the statutory text below; the second answer corrected the expert’s first instinct.
Primary source · Primary law · 2026-07-11
L.1 Mass. Gen. Laws ch. 149, § 24L(b)(vii)Section 24L(b)(vii) supports the rule that the consideration must be specified in the noncompetition agreement itself, and that a qualifying garden leave clause may not let the employer unilaterally discontinue the payments except on employee breach.
The noncompetition agreement shall be supported by a garden leave clause or other mutually-agreed upon consideration between the employer and the employee, provided that such consideration is specified in the noncompetition agreement. To constitute a garden leave clause within the meaning of this section, the agreement must (i) provide for the payment, consistent with the requirements for the payment of wages under section 148 of chapter 149 of the general laws, on a pro-rata basis during the entirety of the restricted period, of at least 50 percent of the employee’s highest annualized base salary paid by the employer within the 2 years preceding the employee’s termination; and (ii) except in the event of a breach by the employee, not permit an employer to unilaterally discontinue or otherwise fail or refuse to make the payments
See Mass. Gen. Laws ch. 149, § 24L(b)(vii)
Primary source · Primary law · 2026-07-11
L.2 Mass. Gen. Laws ch. 149, § 24L(a) (garden leave clause definition)Section 24L(a) supports the rule that the statute’s employer off-ramp is waiver: the garden-leave payment provision becomes effective upon termination of employment unless the employer waives the restriction.
“Garden leave clause”, a provision within a noncompetition agreement by which an employer agrees to pay the employee during the restricted period, provided that such provision shall become effective upon termination of employment unless the restriction upon post-employment activities are waived by the employer or ineffective under subsection (c)(iii).
See Mass. Gen. Laws ch. 149, § 24L(a)
Quoted verbatim from the official General Laws text (malegislature.gov); the L.1 quote ends where the statutory proviso on breach-extended restricted periods begins.
{
"record_id": "ec-2026-07-11-ma-noncompete-pilot",
"repo": "legal-explainer",
"repo_visibility": "private",
"pr": 1738,
"branch": "pilot/ma-expert-correction",
"finding": "MA-T-001",
"topic": "garden-leave consideration: drafting posture",
"disposition": "NEEDS_CORRECTION",
"affected_requirement": "REQ-…massachusetts.require-garden-leave-or-agreed-consideration",
"rationale": "An employer-presented form shouldn’t volunteer affirmative obligations the law doesn’t require, so the consideration is recast as a conditional stipulation of the Cover Terms consideration. The open-ended ‘other consideration → Employer will provide it’ promise was a blank check (other than what? no ascertainable trigger); it is deleted, and the parties can always amend.",
"authorship": {
"proposal": {
"model": "claude-fable-5",
"tool": "Claude Code",
"commit": "bfbf0a2e"
},
"correction": {
"authorship_mode": "correction_of_ai_output",
"model": "claude-fable-5",
"directed_by": "repository_owner",
"reviewer_background": "technically_trained_lawyer",
"commit": "10c5ef9f"
}
},
"diff_inputs": {
"before_commit": "bfbf0a2e",
"before_role": "ai_proposal",
"after_commit": "10c5ef9f",
"after_role": "expert_correction"
},
"dictated_review_ref": {
"medium": "voice_transcript",
"date": "2026-07-11",
"fidelity": "paraphrased_from_speech_recognition",
"verbatim_transcript": "available_on_request"
},
"law_check": {
"requested": true,
"requested_by": "expert",
"questions": [
"does §24L require the consideration to be specified in the agreement, or merely paid?",
"may the covenant lapse if the employer stops paying?"
],
"answers": [
"§24L(b)(vii): the consideration must be ‘specified in the noncompetition agreement’",
"no; unilateral discontinuance is barred except on employee breach, and the employer off-ramp is the §24L(a) waiver"
],
"corrected_expert_instinct": true,
"sources": [
"Mass. Gen. Laws ch. 149, § 24L(a)",
"Mass. Gen. Laws ch. 149, § 24L(b)(vii)"
]
},
"rejected_element": null
}Each trace carries the AI-proposed text, the expert-corrected text with the computed word-level redline, the expert's rationale, the dictated review (paraphrased from speech recognition), the primary-source check where the expert requested one, and the machine-readable JSON record.
A third finding from the same review, MA-T-006, records the leakage failure mode the companion evaluation write-up measures: the AI proposal recited the self-executing choice-of-law bar and county-venue mandate of Mass. Gen. Laws ch. 149, § 24L(e)–(f) in the operative body of a form, and the expert's correction removed them.
Motivation from external evidence
The clearest published signal comes from an evaluation OpenAgreements does not run: Vals AI's Legal Research Bench reports that reconciling conflicting authority is the hardest category for every model it tests, at 20.7% all-pass against a 28.4% overall rate, with each model scoring lower there than on the rest. Synthesizing conflicting sources, not locating a single rule, is where the tested models break down. Whether that is because reconciliation is inherently difficult or because models are least trained on it is an open question; either way, public training data that captures expert reconciliation is scarce.
The OpenAgreements evaluation points the same direction: in the companion evaluation write-up, frontier models leaked internal institutional analysis into outward-facing contracts at sharply different rates, and every leak is a reconciliation-adjacent judgment failure (correct law, wrong document). That result is cited second deliberately: the case for this dataset rests on the published external evidence, not on the OpenAgreements benchmark.
Methods and materials
Capture workflow
The maintenance workflow for the OpenAgreements legal knowledge base captures paired examples in the ordinary course of work: an AI-assisted proposal, the expert's correction, the resulting diff, and the contemporaneous rationale, including the primary-source law checks the expert ran before deciding. No separate annotation pipeline exists; each correction lands as a structured, attributable record with two-layer model attribution (machine-recorded for the corrected work; recollection-based and labeled as such for legacy content).
The maintenance reviews run on live legal change. Cases get superseded, distinguished, extended, or scoped; statutes replace common-law rules mid-form; an inherited template can be right in one state and void in a neighboring one. Keeping the forms current therefore requires the reconciliation judgment the leaderboard scores lowest, and the traces record how a practicing lawyer performs it: which authorities were consulted, which controlled, which were considered and not applied, and why.
The workflow also records the expert's spoken review, entered via speech recognition. The dictation-based capture format allows high throughput per attorney; spoken review has run roughly three times faster than written annotation in this workflow. The recordings capture a data type that is scarce on the open internet: a lawyer working through a realistic client problem, checking primary sources as questions arise, and reaching a conclusion.
Limitations
The corpus is early and limited: the current records are a sample from one Massachusetts non-compete maintenance review, published so the schema and capture method can be inspected against real examples. Dictated rationales are paraphrased from speech recognition (curated verbatim transcripts are available on request). The source repository, legal-explainer, is private; its pull requests and issues are referenced by number.
If you work on model training or data acquisition and want to shape the schema, the licensing, or the next tranches, contact hello@openagreements.org.
Sources
Data provider
1 Vals AI Legal Research Bench leaderboard (2026)Reconciling conflicting authority is the leaderboard's hardest category for every model tested: 20.7% all-pass against a 28.4% overall rate.
Reconciliation is the clearest difficulty signal in the benchmark. At 20.7% all-pass it is the hardest category of all, well below the 28.4% overall rate, and the gap holds for every single model: each one scores lower on reconciliation questions than on the rest.
See Vals AI, Legal Research Bench leaderboard (last updated July 9, 2026).