# Expert-correction traces from maintaining an open legal knowledge base[^about] Structured, attributable expert corrections of AI legal drafting, captured from a real maintenance workflow: machine-recorded diffs, dictated rationale, primary-source checks, and the authorities considered and not applied. ## Abstract {#abstract} Reinforcement learning needs expert corrections gathered where frontier AI models actually fail, and saturated question-answer benchmarks no longer supply them. Independent evaluation points at the gap: on Vals AI's Legal Research Bench, reconciling conflicting authority is the lowest-scoring category for every model tested, at 20.7% all-pass against a 28.4% overall rate.[^vals-ai-legal-research-leaderboard] Maintaining the OpenAgreements legal knowledge base produces this data in the ordinary course of work. Frontier models draft practice guides and templates that receive thousands of fetches each month, and a lawyer reviews the drafts before publication, so expert corrections accrue as a byproduct rather than an annotation project. Each trace is captured at the time of entry from a Git-based workflow: the machine-recorded diff between the AI proposal and the expert correction, the expert's dictated rationale, the supporting primary-source citations, and the authorities considered and not applied, the reconciliation judgment the leaderboard finds hardest. The corrections concentrate on judgment rather than fact: what may cross into a counterparty-facing document, which authority controls an inherited form, when a statute's self-executing operation belongs in a guide rather than an instrument. As of July 12, 2026, the corpus holds eight structured correction records from one Massachusetts non-compete maintenance review (seven substantive corrections and one process finding), with the capture pipeline running on every subsequent review. A complete two-finding sample trace, layered from redline to machine-readable record, is included below. ## Record schema and sample trace {#record-schema} Below are two example expert-correction traces produced by our workflow. Derived from structured Git commit messages labeled `correction_of_ai_output`. **Finding MA-T-003 — Excluded workers: implied obligation made express** (NEEDED CORRECTION) Massachusetts non-compete template · excluded-worker carve-out, § 24L(c). **Human expert correction (diff).** Computed word-level redline (`~~strikethrough~~` = removed by the expert correction, `**++bold++**` = added): > The noncompetition covenant in Section 2 does not apply to Employee if, as of the date Employee’s employment ends, Employee is within a category of workers as to whom noncompetition agreements are unenforceable under Section ~~24L(c), and Employer will not enforce it against Employee.~~ **++24L(c).++** **Expert rationale (AI addition rejected).** If the covenant does not apply, non-enforcement follows. The AI’s added promise made an implied obligation express, creating an obligation against the form’s user with no validity gain. ‘Don’t draft the counterparty’s defenses into the instrument.’ (The AI’s removal of the independent-contractor recital, a separate part of the same proposal, was confirmed.) **Human dictated review (paraphrased from speech recognition).** > “Dropping the independent-contractor recital — agreed.” > > “But no to ‘and Employer will not enforce it against Employee.’ If the covenant doesn’t apply, non-enforcement follows. That’s an express obligation that’s already implied — we’d be drafting the counterparty’s defense into our own form. Take it out.” JSON record: ```json { "record_id": "ec-2026-07-11-ma-noncompete-pilot", "repo": "legal-explainer", "repo_visibility": "private", "pr": 1738, "branch": "pilot/ma-expert-correction", "finding": "MA-T-003", "topic": "excluded-worker carve-out: implied obligation made express", "disposition": "NEEDS_CORRECTION", "affected_requirement": "REQ-…massachusetts.exclude-non-competes-for-excluded-worker-categories", "rationale": "If the covenant does not apply, non-enforcement follows. The AI’s added promise made an implied obligation express, creating an obligation against the form’s user with no validity gain. ‘Don’t draft the counterparty’s defenses into the instrument.’ (The AI’s removal of the independent-contractor recital, a separate part of the same proposal, was confirmed.)", "authorship": { "proposal": { "model": "claude-fable-5", "tool": "Claude Code", "commit": "bfbf0a2e" }, "correction": { "authorship_mode": "correction_of_ai_output", "model": "claude-fable-5", "directed_by": "repository_owner", "reviewer_background": "technically_trained_lawyer", "commit": "10c5ef9f" } }, "diff_inputs": { "before_commit": "bfbf0a2e", "before_role": "ai_proposal", "after_commit": "10c5ef9f", "after_role": "expert_correction" }, "dictated_review_ref": { "medium": "voice_transcript", "date": "2026-07-11", "fidelity": "paraphrased_from_speech_recognition", "verbatim_transcript": "available_on_request" }, "law_check": { "requested": false, "reason": "drafting judgment, not a point of law" }, "confirmed_element": "removal of the independent-contractor recital", "confirmed_element_source": "ai_proposal", "rejected_element": "‘and Employer will not enforce it against Employee’", "rejected_element_source": "ai_addition" } ``` **Finding MA-T-001 — Garden-leave consideration: drafting posture** (NEEDED CORRECTION) Massachusetts non-compete template · Mass. Gen. Laws ch. 149, § 24L. **Human expert correction (diff).** Computed word-level redline (`~~strikethrough~~` = removed by the expert correction, `**++bold++**` = added): > During the Restricted Period, **++if the Cover Terms specify garden leave,++** Employer ~~will~~ **++shall++** pay Employee garden-leave consideration equal to at least fifty percent of Employee’s highest annualized base salary within the two years preceding termination, paid on a pro-rata basis over the Restricted Period. **++This payment obligation becomes effective upon cessation of employment unless Employer waives the noncompetition covenant in writing, and, except in the event of a breach by Employee, Employer shall not unilaterally discontinue or otherwise fail or refuse to make the payments.++** If the Cover Terms specify other mutually agreed-upon consideration, ~~Employer will provide Employee~~ **++the parties stipulate++** that ~~consideration.~~ **++such consideration supports the covenant.++** **Expert rationale.** An employer-presented form shouldn’t volunteer affirmative obligations the law doesn’t require, so the consideration is recast as a conditional stipulation of the Cover Terms consideration. The open-ended ‘other consideration → Employer will provide it’ promise was a blank check (other than what? no ascertainable trigger); it is deleted, and the parties can always amend. **Human dictated review (paraphrased from speech recognition).** > “This should be a conditional stipulation, not an employer obligation. An employer-presented form shouldn’t volunteer affirmative promises the law doesn’t require.” > > “And delete the ‘other mutually-agreed consideration → Employer will provide Employee that consideration’ sentence. Other than what? It’s a blank check — and unnecessary, the parties can always amend.” > > “Two things to check: does §24L require the consideration to be specified in the agreement, or merely paid? And can we structure it so the covenant simply lapses if the employer stops paying?” **Primary-source check.** The expert asked two questions before signing off: does § 24L require the consideration to be specified in the agreement, or merely paid? And may the covenant simply lapse if the employer stops paying? Both were answered against the statutory text below; the second answer corrected the expert’s first instinct. - **Mass. Gen. Laws ch. 149, § 24L(b)(vii)** — Section 24L(b)(vii) supports the rule that the consideration must be specified in the noncompetition agreement itself, and that a qualifying garden leave clause may not let the employer unilaterally discontinue the payments except on employee breach. “The noncompetition agreement shall be supported by a garden leave clause or other mutually-agreed upon consideration between the employer and the employee, provided that such consideration is specified in the noncompetition agreement. To constitute a garden leave clause within the meaning of this section, the agreement must (i) provide for the payment, consistent with the requirements for the payment of wages under section 148 of chapter 149 of the general laws, on a pro-rata basis during the entirety of the restricted period, of at least 50 percent of the employee’s highest annualized base salary paid by the employer within the 2 years preceding the employee’s termination; and (ii) except in the event of a breach by the employee, not permit an employer to unilaterally discontinue or otherwise fail or refuse to make the payments” () - **Mass. Gen. Laws ch. 149, § 24L(a) (garden leave clause definition)** — Section 24L(a) supports the rule that the statute’s employer off-ramp is waiver: the garden-leave payment provision becomes effective upon termination of employment unless the employer waives the restriction. ““Garden leave clause”, a provision within a noncompetition agreement by which an employer agrees to pay the employee during the restricted period, provided that such provision shall become effective upon termination of employment unless the restriction upon post-employment activities are waived by the employer or ineffective under subsection (c)(iii).” () Quoted verbatim from the official General Laws text (malegislature.gov); the L.1 quote ends where the statutory proviso on breach-extended restricted periods begins. JSON record: ```json { "record_id": "ec-2026-07-11-ma-noncompete-pilot", "repo": "legal-explainer", "repo_visibility": "private", "pr": 1738, "branch": "pilot/ma-expert-correction", "finding": "MA-T-001", "topic": "garden-leave consideration: drafting posture", "disposition": "NEEDS_CORRECTION", "affected_requirement": "REQ-…massachusetts.require-garden-leave-or-agreed-consideration", "rationale": "An employer-presented form shouldn’t volunteer affirmative obligations the law doesn’t require, so the consideration is recast as a conditional stipulation of the Cover Terms consideration. The open-ended ‘other consideration → Employer will provide it’ promise was a blank check (other than what? no ascertainable trigger); it is deleted, and the parties can always amend.", "authorship": { "proposal": { "model": "claude-fable-5", "tool": "Claude Code", "commit": "bfbf0a2e" }, "correction": { "authorship_mode": "correction_of_ai_output", "model": "claude-fable-5", "directed_by": "repository_owner", "reviewer_background": "technically_trained_lawyer", "commit": "10c5ef9f" } }, "diff_inputs": { "before_commit": "bfbf0a2e", "before_role": "ai_proposal", "after_commit": "10c5ef9f", "after_role": "expert_correction" }, "dictated_review_ref": { "medium": "voice_transcript", "date": "2026-07-11", "fidelity": "paraphrased_from_speech_recognition", "verbatim_transcript": "available_on_request" }, "law_check": { "requested": true, "requested_by": "expert", "questions": [ "does §24L require the consideration to be specified in the agreement, or merely paid?", "may the covenant lapse if the employer stops paying?" ], "answers": [ "§24L(b)(vii): the consideration must be ‘specified in the noncompetition agreement’", "no; unilateral discontinuance is barred except on employee breach, and the employer off-ramp is the §24L(a) waiver" ], "corrected_expert_instinct": true, "sources": [ "Mass. Gen. Laws ch. 149, § 24L(a)", "Mass. Gen. Laws ch. 149, § 24L(b)(vii)" ] }, "rejected_element": null } ``` Each trace carries the AI-proposed text, the expert-corrected text with the computed word-level redline, the expert's rationale, the dictated review (paraphrased from speech recognition), the primary-source check where the expert requested one, and the machine-readable JSON record. A third finding from the same review, MA-T-006, records the leakage failure mode the companion evaluation write-up measures: the AI proposal recited the self-executing choice-of-law bar and county-venue mandate of Mass. Gen. Laws ch. 149, § 24L(e)–(f) in the operative body of a form, and the expert's correction removed them. ## Motivation from external evidence {#motivation} The clearest published signal comes from an evaluation OpenAgreements does not run: Vals AI's [Legal Research Bench](https://www.vals.ai/benchmarks/legal_research) reports that reconciling conflicting authority is the hardest category for every model it tests, at 20.7% all-pass against a 28.4% overall rate, with each model scoring lower there than on the rest.[^vals-ai-legal-research-leaderboard] Synthesizing conflicting sources, not locating a single rule, is where the tested models break down. Whether that is because reconciliation is inherently difficult or because models are least trained on it is an open question; either way, public training data that captures expert reconciliation is scarce. The OpenAgreements evaluation points the same direction: in the [companion evaluation write-up](/for-labs/evals), frontier models leaked internal institutional analysis into outward-facing contracts at sharply different rates, and every leak is a reconciliation-adjacent judgment failure (correct law, wrong document). That result is cited second deliberately: the case for this dataset rests on the published external evidence, not on the OpenAgreements benchmark. ## Methods and materials {#methods-and-materials} ### Capture workflow {#capture-workflow} The maintenance workflow for the OpenAgreements legal knowledge base captures paired examples in the ordinary course of work: an AI-assisted proposal, the expert's correction, the resulting diff, and the contemporaneous rationale, including the primary-source law checks the expert ran before deciding. No separate annotation pipeline exists; each correction lands as a structured, attributable record with two-layer model attribution (machine-recorded for the corrected work; recollection-based and labeled as such for legacy content). The maintenance reviews run on live legal change. Cases get superseded, distinguished, extended, or scoped; statutes replace common-law rules mid-form; an inherited template can be right in one state and void in a neighboring one. Keeping the forms current therefore requires the reconciliation judgment the leaderboard scores lowest, and the traces record how a practicing lawyer performs it: which authorities were consulted, which controlled, which were considered and not applied, and why. The workflow also records the expert's spoken review, entered via speech recognition. The dictation-based capture format allows high throughput per attorney; spoken review has run roughly three times faster than written annotation in this workflow. The recordings capture a data type that is scarce on the open internet: a lawyer working through a realistic client problem, checking primary sources as questions arise, and reaching a conclusion. ## Limitations {#data-limitations} The corpus is early and limited: the current records are a sample from one Massachusetts non-compete maintenance review, published so the schema and capture method can be inspected against real examples. Dictated rationales are paraphrased from speech recognition (curated verbatim transcripts are available on request). The source repository, legal-explainer, is private; its pull requests and issues are referenced by number. If you work on model training or data acquisition and want to shape the schema, the licensing, or the next tranches, contact [hello@openagreements.org](mailto:hello@openagreements.org). [^about]: By Steven Obiajulu, J.D. Published by [openagreements.org](https://openagreements.org). Last reviewed 2026-07-12. License: CC BY 4.0. Steven Obiajulu, J.D. wrote this essay. It states the author's views, synthesizes public sources, and is not legal advice. This article is for informational purposes only and does not create an attorney-client relationship. CC BY 4.0. Cite as Steven Obiajulu, *Expert-correction traces from maintaining an open legal knowledge base*, OpenAgreements (last updated July 12, 2026), https://openagreements.org/for-labs/data. [^vals-ai-legal-research-leaderboard]: **Vals AI Legal Research Bench leaderboard (2026)** — "Reconciliation is the clearest difficulty signal in the benchmark. At 20.7% all-pass it is the hardest category of all, well below the 28.4% overall rate, and the gap holds for every single model: each one scores lower on reconciliation questions than on the rest." *Vals AI, Legal Research Bench leaderboard (last updated July 9, 2026).*