Abstract
Frontier AI models have been saturating benchmarks: near-perfect needle-in-a-haystack recall at up to 10 million tokens of context, and LegalBench scores in excess of 80%. For knowledge work, however, a trust gap remains, particularly in institutional contexts. Harvey's Legal Agent Benchmark (LAB) grades realistic legal tasks end-to-end on an all-pass standard: each LAB scenario hands the model about eight documents on average (2 to 55 documents per task, across 1,749 tasks), and the frontier models evaluated at launch completed fewer than 10% of tasks. We identify a concrete failure mode one layer deeper into institutional work. In exploratory testing on an evaluation environment that includes a 367-document knowledge base in each scenario (127 practice guides, 176 templates, 53 checklists, and 11 surveys), we observed frontier models leaking internal institutional analysis into a scenario's outward-facing deliverables, in particular contracts. We replicated a covenant-drafting task where the leakage failure appeared as a family of ten instances across different state combinations, scoring three frontier models on the single leakage criterion, with up to three model judges per cell and human adjudication of every leak verdict. The models stratify: 1, 4, and 7 leaking instances out of ten (Gemini 3.1 Pro Preview, GPT-5.5, and Claude Opus 4.8 respectively). Every leak verdict is human-adjudicated; the remaining clean cells are majority-of-three-judges verdicts.
This paper reports the evaluation. Its companion paper, Expert-correction traces from maintaining an open legal knowledge base, reports the training-data capture that the same environment produces; the two stand independently: accepting the dataset does not require accepting this benchmark, and vice versa.
Results
Across the ten instances of the covenant-package family, the three models leaked internal institutional analysis into an outward-facing agreement at different rates:
Click any square to compare that instance across models.
PASS if the offer letters and agreement bodies contain operative drafting text and keep advisory explanations in client-email.md or rationale.md. Legally operative statutory notices may appear where they are part of the agreement text itself, but the drafting files should not cite practice-guide paths, explain why provisions were changed, discuss how courts analyze the covenants, or give advice to the client. FAIL if any operative drafting file includes source-path references, drafting commentary, client advice, or legal analysis that belongs in the email or rationale rather than the agreement body.
The agent successfully kept all operative drafting files free of legal advice, drafting commentary, practice-guide paths, and legal analysis. The files contain only the operative text of the offer letters and restrictive covenant agreements, with statutory references appropriately integrated as part of the agreement text itself.
The operative drafting files (offer letters and restrictive covenant agreements) contain only operative legal text, definitions, representations, and legally required statutory notices (e.g., DTSA immunity notice, California Labor Code §§ 2870-2872 notice, Colorado C.R.S. § 8-2-113 references embedded as operative terms). None of the drafting files contain practice-guide citations, explanations of why provisions were drafted a certain way, commentary on how courts analyze covenants, client advice, or drafting rationale. The statutory references appear as operative terms within the agreement body (e.g., 'as preserved by C.R.S. § 8-2-113(3)(b)' in a permitted-disclosures clause, or the DTSA safe harbor notice), which is standard operative drafting practice. There are no source-path references, no 'we changed this because...' commentary, and no legal analysis directed at the client embedded in the agreement text.
Human-adjudicated clean: no internal analysis crossed into the outward-facing agreements on the production run.
Cited the firm’s internal resource library inside the agreement text (human-adjudicated).
PASS if the offer letters and agreement bodies contain operative drafting text and keep advisory explanations in client-email.md or rationale.md. Legally operative statutory notices may appear where they are part of the agreement text itself, but the drafting files should not cite practice-guide paths, explain why provisions were changed, discuss how courts analyze the covenants, or give advice to the client. FAIL if any operative drafting file includes source-path references, drafting commentary, client advice, or legal analysis that belongs in the email or rationale rather than the agreement body.
The agent included drafting commentary and legal analysis directly in the operative agreement for Frank Rivera. Specifically, under the 'Colorado Drafting Limitation' section, the agreement states: 'Employee's annualized cash compensation for this role is below the 2026 Colorado thresholds stated in the knowledge library for non-competes and customer non-solicits.' This is an explanation of why certain provisions were omitted and references a knowledge library, which belongs in a client email or rationale document, not in the operative legal agreement.
FAIL: At least one operative agreement includes drafting/legal commentary that belongs in a client email or rationale, not the agreement body. For example, the Colorado agreement states that the employee’s compensation is below “the 2026 Colorado thresholds stated in the knowledge library” and explains that the agreement is therefore limited to certain covenants. That is an advisory/source-reference explanation rather than operative contract text.
Reviewing all six drafting files (three offer letters and three restrictive covenant agreements), the operative text contains only standard contractual provisions, definitions, covenants, and legally operative statutory notices (e.g., the DTSA immunity notice, the California Labor Code §§ 2870-2872 notice). None of the files contain source-path references, practice-guide citations, drafting commentary explaining why provisions were chosen, discussion of how courts analyze the covenants, or client advisory content. The 'Colorado Drafting Limitation' and 'California Drafting Limitation' sections state what the agreement does not include and why (compensation thresholds, California law), but these read as operative scope-limiting provisions rather than legal analysis or client advice—they tell the employee what restrictions apply, not the client why the drafter made certain choices. No file references knowledge library paths, explains drafting decisions to the client, or provides legal analysis that belongs in a separate rationale document.
This Agreement does **not** include a post-employment non-compete, customer non-solicitation, or no-business-with-customers covenant. Employee's annualized cash compensation for this role is below the 2026 Colorado thresholds stated in the knowledge library for non-competes and customer non-solicits. The Agreement is therefore limited to confidentiality, invention assignment, return of property, and a narrow active employee non-solicitation covenant.
frank-rivera-restrictive-covenant-agreement.md
Human-adjudicated leak: the agreement text cites the firm’s internal resource library.
Embedded statutory analysis, threshold math, and penalty exposure as ‘scope notes’ in all three agreements (human-adjudicated).
PASS if the offer letters and agreement bodies contain operative drafting text and keep advisory explanations in client-email.md or rationale.md. Legally operative statutory notices may appear where they are part of the agreement text itself, but the drafting files should not cite practice-guide paths, explain why provisions were changed, discuss how courts analyze the covenants, or give advice to the client. FAIL if any operative drafting file includes source-path references, drafting commentary, client advice, or legal analysis that belongs in the email or rationale rather than the agreement body.
The agent included drafting commentary and legal analysis directly in the operative drafting files (e.g., the 'California scope note', 'Colorado scope note', and 'Texas scope note' blockquotes in the restrictive covenant agreements). This violates the criterion to keep legal advice and explanations out of the agreement text.
FAIL: The operative restrictive covenant agreements include advisory/legal-analysis sections such as “California scope note,” “Colorado scope note,” and “Texas scope note” explaining statutory rules, thresholds, penalties, enforceability analysis, and why covenants were omitted or included. That commentary belongs in a client email or rationale, not in the agreement body, except for legally operative statutory notices. The offer letters are mostly operative, but the agreement files violate the criterion.
The operative drafting files (offer letters and restrictive covenant agreements) contain scope notes embedded directly in the Cover Terms sections of the restrictive covenant agreements. These scope notes (e.g., the California scope note, Colorado scope note, and Texas scope note) contain legal analysis, explanations of why certain covenants were omitted or included, references to specific statutory thresholds with dollar amounts, discussion of how courts analyze covenants, and client advisory content (e.g., 'Presenting a void covenant to a worker is itself unlawful and carries a $5,000-per-worker penalty'). This is drafting commentary and legal analysis that belongs in a client email or rationale document, not in the operative agreement body. These scope notes are not operative statutory notices required by law to appear in the agreement itself—they are explanatory commentary about drafting choices. This violates the criterion that drafting files should not 'explain why provisions were changed, discuss how courts analyze the covenants, or give advice to the client.'
Colorado scope note.** Under C.R.S. § 8-2-113, a post-employment non-compete is void unless the worker earns at least the highly compensated worker threshold ($130,014 for 2026), and a customer non-solicitation covenant is void unless the worker earns at least 60% of that threshold ($78,008.40 for 2026), in each case measured both at signing and at enforcement and only to protect trade secrets. Employee's annualized cash compensation ($72,000) is below both thresholds. Presenting a void covenant to a worker is itself unlawful and carries a $5,000-per-worker penalty (C.R.S. § 8-2-113(8)).
frank-rivera-restrictive-covenant-agreement.md
Human-adjudicated leak: statutory analysis, threshold math, and penalty exposure embedded as scope notes in all three agreements.
Every square is a button opening that instance’s record under all three models, for side-by-side comparison. clean leaked dark outline = human-adjudicated cell · squares in family order (original, then v01–v09)
Where a human adjudication exists it controls the cell’s verdict; raw judge output is preserved unchanged. Each instance is one scenario from a larger evaluation set and is not necessarily representative of overall performance.
One instance is human-adjudicated end-to-end (the original). Across the nine variant instances, every leaking cell has been human-adjudicated (2026-07-12): the nine judge-majority leak verdicts were confirmed, and one judge-majority clean verdict was overturned to leaked on review; the remaining clean cells are majority-of-three-judges verdicts. Every square above opens that cell's full record: the criterion, each judge's complete reasoning, and, where the cell has them, the human adjudication and verbatim excerpts of what leaked.
Each failure is a placement failure: accurate legal analysis written into the document an employee or counterparty receives, as drafting advice, statutory threshold math, penalty exposure, or references to the firm's internal resources inside operative agreements.
Methods and materials
Harvey's LAB and our harness
Harvey's Legal Agent Benchmark (LAB) provides an open harness for testing agents on realistic legal work: each scenario supplies a workspace of source documents, asks for finished deliverables, and grades them end-to-end on an all-pass standard. The strongest published result on LAB's independently run public leaderboard is currently 20% of tasks end-to-end, at roughly $0.80 per task (Meta's Muse Spark 1.1). We run LAB-compatible harness infrastructure, with the evaluator repairs described in Extending Harvey's LAB.
Task construction
The family replicates one task design, a multi-state covenant-drafting package, ten times with different state combinations. Each instance asks the model to draft restrictive-covenant agreements for employees in three states, working inside the institutional knowledge base described below, and is scored on a single criterion: whether internal analysis stayed out of the signed documents. The original instance surfaced the failure mode in exploratory testing; the nine variants were generated after that, an adaptive step toward where the models actually break, in the spirit of adaptive evaluation frameworks, and then frozen before scoring. The exploratory runs that preceded the freeze are excluded from every number on this page.
Judging
Agent runs used one production run per instance-model cell at fixed settings. Each cell carries up to three independent judge votes (Gemini 3.1 Pro, GPT-5.5, and Claude Sonnet 4.6), all under the reasoning-first schema at temperature zero. Claude Sonnet 4.6 judges because it is Harvey LAB's default judge; the Claude agent under test is Opus 4.8, deliberately distinct from the judge model. A human adjudicated every leak verdict, and the per-cell records on this page carry each judge's full reasoning alongside the adjudication text.
Materials: a realistic institutional knowledge base
Transactional lawyers work inside an institution: inherited templates, standing client instructions, practice resources, and deal point memos, with each matter arriving as deal-specific inputs like a term sheet or a change in the law. The work is to decide what controls, change what must change, preserve what should remain, and keep the legal analysis out of the document the counterparty will sign.
The OpenAgreements library supplies that institution in public form: primary-source-backed practice guides, checklists, and maintained agreement templates across all 50 states, the District of Columbia, and five U.S. territories. The corpus is not eval-only material. It is designed for ordinary practitioner drafting and review, and browser traffic attributable to people and user-directed AI sessions reaches thousands of fetches each month. Eval matters use synthetic facts and no client data, while the same public material that helps a person or agent do the work can be frozen at a known version and used to test whether the resulting work product is correct.
A workspace this shape also tests something retrieval alone does not supply: published results on reasoning-intensive legal benchmarks find that adding standard retrieval pipelines barely improves downstream accuracy over zero-shot performance — the graded question here is not whether the model can find the authority but what it may say, and where, once it has.
A public corpus invites an obvious objection: a model may have seen it in training. The tasks reduce that risk by drawing on restrictive-covenant law, which has changed substantially in most states over the past two years — the FTC's 2024 nationwide ban was set aside before it took effect, leaving the field to fast-moving state statutes — and several of the seeded rules took effect within months of the tasks being written, so producing correct work product favors reading the workspace over recalling training data.
The frozen environment
A single commit SHA pins the tasks, harness, and evaluator. The freeze manifest records a git tree hash for every task, a SHA-256 hash of every file in every task workspace, the exact model settings, and the judge schema versions. Results are appended to the manifest's run log under append-only rules; nothing is overwritten or silently replaced.
Extending Harvey's LAB
LAB's independently run public leaderboard at Vals AI currently reports end-to-end results for Meta, Grok, Claude, GPT, and open-weight models, with a single Flash-class Gemini entry and no Gemini Pro result yet. That gap is part of the motivation for this section: cross-provider evaluation only happens where the harness can actually run every provider, as agent and as judge. Working with LAB taught us that the evaluator is part of the system under test: a structured judge answer can be syntactically valid and still internally inconsistent. We made three evaluator contributions: two filed and submitted upstream (the routing repair and the field-order fix, with the issues and pull requests linked below), and one implemented in our LAB-compatible harness and disclosed here (contradiction routing). The production ensemble depends on all three.
A Gemini judge required a routing repair. The unpatched harness rejects provider-prefixed judge IDs like google/gemini-3.1-pro-preview before any API call is made. We filed an upstream LAB issue (open) and submitted an upstream change (open). In a side-by-side demonstration on one task from the same environment, the unpatched path still failed at routing with that error preserved, while the patched Gemini judge completed every criterion, a Claude judge control completed every criterion, and every arm that ran agreed on every verdict. One caveat we owe readers: we had earlier observed Google's endpoint rejecting a keyword in the judge's structured-output schema, but that rejection did not reproduce on re-test, so the case for the fix rests on the routing repair and on correctly handling the provider's parsed structured response. The goal is not to make one model look better; it is to make cross-provider evaluation possible and inspectable.
Contradictory verdicts go to a human. A fail verdict whose reasoning lacks explicit failure language, or whose reasoning concludes the criterion actually passes, is routed to human review with the raw judge response preserved. There is no automatic retry: re-rolling the judge until the fields agree would hide the failure instead of measuring it.
Field order changes the verdict. The original judge schema orders verdict before reasoning, so under structured output the judge commits to a verdict before writing the explanation. In a replication of ten judge calls per arm on one affected criterion (identical prompt, Claude Sonnet 4.6, temperature zero), the verdict-first schema returned fail ten out of ten, several with reasoning that concluded the package passes; with the fields reversed, ten out of ten returned pass, each consistent with its reasoning and with the human audit of the underlying documents. This is a demonstrated effect for this judge model on this criterion; we do not claim it generalizes across providers or criteria. We reported it upstream as a LAB issue (open) and submitted an upstream change (open) that reorders the schema so reasoning is written first. All production results are judged reasoning-first, and we disclose that choice because the schema is part of the measurement instrument.
Neither upstream change touches LAB's tasks or scoring rubric.
Relation to other legal evaluations
Independent leaderboards are worth reading alongside this paper precisely because they measure different things: on Vals AI's Legal Research Bench (all-pass grading, updated July 2026), Claude Opus 4.8 leads at 43.75% all-pass, with Claude Sonnet 5 and GPT-5.5 close behind — roughly the reverse of the ordering our institutional-leakage criterion produces. The two measure different genres of legal work. Research benchmarks test open-ended retrieval and reasoning over public law; institutional drafting is closer to grounded generation (work product grounded in a firm's own knowledge base, the skill that grounding evaluations such as Google DeepMind's FACTS Grounding measure), plus a boundary discipline neither genre tests directly: deciding which grounded analysis may cross into the document the counterparty receives.
The leakage failure is also a different error family than the hallucination taxonomies studied for legal research tools: every leaked passage here is accurate law in the wrong document, a judgment failure grounding evaluations do not measure.
One finding of that leaderboard points in the same direction as this paper: its hardest category, for every model tested, is reconciling conflicting authority — synthesizing conflicting sources, not locating a single rule, is where models break down most reliably. That is the shape of institutional work: our tasks hand the model an inherited template, standing client instructions, practice resources, and a change in the law at once, and the graded questions are which authority controls and what may be said in which document. The companion paper takes this up as a data question. That is why we publish the criterion, the per-instance judge records, and the raw judge output rather than a single score.
Limitations and future directions
This paper measures one failure mode, on one task family, across ten instances; it is not a leaderboard. We publish it because the contribution is the method (realistic institutional drafting, a frozen environment, disclosed judging, and human adjudication with preserved raw output); the sample is small and we say so. Human adjudication is complete for every leaking cell (2026-07-12); the clean cells remain majority-of-three-judges verdicts, one of which, carrying a dissenting judge, is under active review.
We tested three proprietary frontier models; open-weight models are future work, both because legal work's confidentiality pressures make locally deployable models practically relevant and because open-source evaluation keeps the trade-offs inspectable — LAB's public leaderboard already carries open-weight entries. We plan to extend the task family (further tranches of the covenant-package family are held out and scheduled), to publish the raw run data with the layers kept separate, and to keep publishing the method early, because eval design improves when the tasks, rubrics, and evaluator failures can themselves be criticized.
If you work on legal evaluation and want to help shape the next iteration, contact hello@openagreements.org.
Reproducibility
The task definitions and frozen-environment manifests behind this paper are published in the public fork at open-agreements/harvey-labs; the harness fixes are public as the upstream pull requests linked above. The tranche-1 raw run data (transcripts, raw judge output, and adjudication records, with the layers kept separate) lands in the same tree once the final pre-publication scrub review completes — conservatively staged, because published data cannot be unshared. The per-cell verdict data rendered on this page is generated from those run records by a manifest-driven exporter; the generated data file embeds its source commit and content digest, so a reader can verify the page against the record tree as it publishes.
Sources
Scholarship
3 Gemini 1.5 technical report (2024)Gemini 1.5 reported near-perfect needle-in-a-haystack recall on long-context retrieval at up to 10 million tokens of context.
Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities
See Gemini Team, Google, Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context, arXiv:2403.05530 (2024).
Vendor documentation
1 Gemini 3 Pro model card (2025)PDFGoogle describes Gemini 3 Pro as its most advanced model for complex tasks, comprehending vast datasets across text, audio, images, video, and entire code repositories.
Gemini 3 Pro is now Google’s most advanced model for complex tasks, and can comprehend vast datasets and challenging problems from different information sources, including text, audio, images, video, and entire code repositories.
See Google, Gemini 3 Pro Model Card (released Nov. 2025; updated May 2026).
Scholarship
2 Gemini 2.5 technical report (2025)Google's Gemini 2.5 report describes frontier long-context, multimodal, and reasoning capability combining to unlock new agentic workflows.
Its unique combination of long context, multimodal and reasoning capabilities can be combined to unlock new agentic workflows.
See Gemini Team, Google, Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities, arXiv:2507.06261 (2025).
Scholarship
4 LegalBench (Guha et al., 2023)LegalBench is a 162-task legal-reasoning benchmark whose tasks were designed and hand-crafted by legal professionals, who took the leading role in its construction.
Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting.
See Neel Guha et al., LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models, arXiv:2308.11462 (2023).
Data provider
5 Vals AI LegalBench leaderboard (2026)Independently measured LegalBench accuracy exceeds 80% for the top frontier models: the leaderboard's highest is 88.56%, with the top Gemini models above 87%.
The highest-performing model is Claude Fable 5, with an accuracy of 88.56%.
See Vals AI, LegalBench leaderboard (last updated July 9, 2026).
Vendor documentation
6 Harvey Legal Agent Benchmark, initial results (2026)On LAB's strict all-pass standard, the frontier models evaluated at launch completed fewer than 10% of tasks end-to-end, showing substantial room for improvement on realistic legal work.
Under our strict all-pass standard, the frontier models we evaluated complete less than 10% of tasks end-to-end in aggregate.
See Harvey, Initial Results on Legal Agent Benchmark (May 26, 2026).
Data provider
7 Vals AI Harvey LAB leaderboard (2026)On LAB's independently run public leaderboard, the strongest published end-to-end result is Meta's Muse Spark 1.1 at 20.0% task resolution at roughly $0.80 per test, while top models pass around 85% of individual criteria — end-to-end resolution, not criterion recall, is the bottleneck.
Top models satisfy most individual criteria, with criteria pass rates around 85%.
See Vals AI, Harvey LAB leaderboard (last updated July 9, 2026).
Lawyer commentary · Commentary
10 The Federal FTC Non-Compete Rule: Where It Stands (OpenAgreements)The FTC's 2024 rule would have banned most employee non-competes nationwide; a federal court set it aside before its effective date and the rule was struck from the CFR, leaving non-competes governed by fast-moving state law.
The FTC issued a rule in 2024 that would have banned most employee non-competes nationwide, but a federal court set it aside before its effective date, the FTC then acceded to that vacatur, and the Commission conformed the CFR to the court's order by striking the rule from the books.
See OpenAgreements, The Federal FTC Non-Compete Rule: Where It Stands (last reviewed July 1, 2026).
Scholarship
11 FACTS Grounding (Jacovi et al., 2025)FACTS Grounding is Google DeepMind's benchmark and public leaderboard measuring whether long-form model responses stay factually grounded in a provided context document — the genre of skill institutional drafting draws on.
We introduce FACTS Grounding, an online leaderboard and associated benchmark that evaluates language models’ ability to generate text that is factually accurate with respect to given context in the user prompt.
See Alon Jacovi et al., The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input, arXiv:2501.03200 (2025).
Data provider
13 Vals AI Legal Research Bench leaderboard (2026)Reconciling conflicting authority is the leaderboard's hardest category for every model tested: 20.7% all-pass against a 28.4% overall rate.
Reconciliation is the clearest difficulty signal in the benchmark. At 20.7% all-pass it is the hardest category of all, well below the 28.4% overall rate, and the gap holds for every single model: each one scores lower on reconciliation questions than on the rest.
See Vals AI, Legal Research Bench leaderboard (last updated July 9, 2026).
Scholarship
12 Hallucination-Free? (Magesh et al., 2024)The first systematic assessment of proprietary legal research tools found meaningful hallucination rates persist even in retrieval-augmented products, motivating granular failure taxonomies for legal AI rather than a binary correct/incorrect metric.
While hallucinations are reduced relative to general-purpose chatbots (GPT-4), we find that the AI research tools made by LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI) each hallucinate between 17% and 33% of the time.
See Varun Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, arXiv:2405.20362 (2024).
Scholarship
9 L-MARS (Wang & Yuan, 2025)On reasoning-intensive legal benchmarks, standard retrieval pipelines barely improve downstream question-answering accuracy over zero-shot performance.
Recent work has shown that on reasoning-intensive legal benchmarks, standard retrieval pipelines barely improve downstream QA accuracy.
See Ziqi Wang & Boqin Yuan, L-MARS: Legal Multi-Agent Workflow with Orchestrated Reasoning and Agentic Search, arXiv:2509.00761 (2025).
Scholarship
8 Task elicitation (Brown et al., 2025)Adaptive evaluation frameworks that iterate toward a model's observed weaknesses profile capabilities more effectively than static test sets.
Task elicitation generates new natural language profiles of model capabilities and weaknesses, found adaptively.
See Davis Brown et al., Adaptively Profiling Models with Task Elicitation, arXiv:2503.01986 (2025).
Scholarship
14 Evaluating AI for Law (Bhambhoria et al., 2024)Open-source legal AI systems keep the accuracy and transparency trade-offs inspectable, complementing evaluation of proprietary models.
The paper advocates for creating open-source legal AI systems to improve accuracy, transparency, and narrative diversity, addressing general AI’s shortcomings in legal contexts.
See Rohan Bhambhoria et al., Evaluating AI for Law: Bridging the Gap with Open-Source Solutions, arXiv:2404.12349 (2024).