Fine-tuning a lightweight verifier for institutional-knowledge leakage in legal drafting

Abstract

Knowledge supplied to a drafting model can leak into the document it produces: accurate internal analysis appears in contract language intended for a counterparty. A second model can look for that failure, but prompting alone produced a noisy critic. We fine-tuned a lightweight verifier on 52 lawyer-adjudicated findings and evaluated it on 27 source-separated lawyer-gold findings. Accuracy rose from 40.7% (11/27) to 77.8% (21/27). The tuned verifier corrected ten paired decisions without a regression (exact two-sided McNemar p = 0.001953125). Most of the gain came from rejecting false positives: specificity rose from 7.7% (1/13) to 76.9% (10/13), while recall moved from 71.4% (10/14) to 78.6% (11/14). A separate ten-document first-pass pilot found a precision and coverage tradeoff that supports combining a broad detector with the tuned verifier.

The critic fine-tuning experiment, blinded lawyer audit of 36 unique proposed repairs, and a source-disjoint raw-document first-pass pilot are complete.

The failure mode

Institutional-knowledge leakage, also called instruction bleed, is a placement error, not necessarily a factual error. A model can use a knowledge library correctly but carry internal material into operative language in a document intended for a counterparty. In legal work, that can expose negotiating posture, internal playbooks, litigation or enforcement strategy, vulnerability analysis, or penalty exposure. A related earlier eval measures the institutional-knowledge leakage failure mode across three AI models.

The auto-rater prompt asks whether operative contract language reveals those internal materials. It is designed to isolate that failure mode while leaving the rest of the contract untouched.

From quality control to a trainable verifier

Over two weeks of ordinary maintenance in July 2026, a Gemini 3.1 Flash-Lite auto-rater ran as an automated quality-control check over AI-generated legal content. A lawyer adjudicated its findings while maintaining the underlying forms. That produced both positives and harder calibration examples: plausible critiques the lawyer rejected.

The tuned task is deliberately narrow. Given one proposed finding, the cited passage when retained, the frozen document, and the auto-rater prompt, the critic returns whether that specific finding is supported. It may not search for a different violation or rescue an inaccurate rationale.

The source documents were state-configured versions of a restrictive-covenant agreement. Whole state agreement files were assigned to either training or evaluation to reduce cross-split leakage from related clauses. Oklahoma evaluation candidates were excluded because the same Oklahoma agreement appeared in training. Duplicates and non-body material were also excluded before analysis. No source group was excluded based on its result.

The tuning job updated the weights of the Gemini 3.1 Flash-Lite critic. It did not update the weights of the original drafting model. Criticism is nevertheless part of the drafting system: the reviewer’s output can block, revise, or return a draft before release.

Results

At the row level, the tuned critic corrected ten decisions without a regression (exact two-sided McNemar p = 0.001953125). Improvements appeared in 5 of ten jurisdiction-file groups (South Dakota, Tennessee, Texas, Vermont, Washington), with none regressing; the exact group sign test gives p = 0.0625. Leaving out each source group in turn kept the accuracy gain positive, ranging from 27.3 to 41.7 percentage points. A stricter five-token-shingle analysis collapsed the 27 passages into 21 near-duplicate clause families; 9 families improved and none regressed. These are sensitivity groupings, not a basis for dropping rows.

Precision rose from 45.5% to 78.6%. The practical change is therefore not just a higher score: fewer ordinary acknowledgments and operative limitations are mislabeled as leaked strategy, while one additional true finding is recovered.

The contrast matters. A separate clean-room experiment trained only on 144 fictional examples and held out all 93 lawyer-gold rows. Flash-Lite moved from 49/93 to 53/93 (p = .125), while a Flash-class model moved from 56/93 to 54/93 (p = .791). Fully synthetic examples did not reproduce the lawyer-gold result.

Raw-document first-pass pilot

We next ran both models directly over ten frozen, tuning-disjoint agreement snapshots without supplying a candidate finding. Counsel reviewed the deduplicated union of 20 findings blind to detector identity: 10 were valid leakage findings and 10 were false positives.

The untuned detector proposed 18 unique findings, 9 valid (50.0% precision). The tuned verifier proposed 6, 4 valid (66.7% precision). The untuned detector surfaced 9 of the 10 counsel-valid findings in the union; the tuned verifier surfaced 4. One valid passage was unique to the tuned verifier, six were unique to the untuned detector, and three were shared.

This is not an absolute recall estimate because counsel adjudicated model-proposed findings rather than every clause in every document. It does show a useful system design: use the broader untuned model for discovery, then use the tuned verifier to filter candidate findings before returning precise criticism to the drafting model.

Can accepted criticism support better drafts?

Counsel completed a blinded review of all 36 unique proposed repairs produced from findings accepted by either verifier. 33/36 were leakage-free (95% Wilson CI 78.2% to 97.1%); 36/36 introduced no new legal error; 35/36 made no unnecessary material change; and 35/36 were publishable targeted repairs under the no-regret standard.

The blinded repair review used four frozen decision questions and a rationale field:

• Leakage-free? the repaired operative body contains none of the instruction bleed prohibited by the frozen rule: internal legal analysis; litigation or enforcement strategy; employer-side vulnerability; statutory penalty exposure; court-reformation strategy; warnings that a provision may be void or risky; explanations of how courts analyze the covenant; source-path references; or practice-guide or treatise commentary. Direct operative terms, party acknowledgments, and legally required notices are allowed.
• No new legal error? the proposed edits introduce no new legally inaccurate, misleading, internally inconsistent, or legally noncompliant language in the touched provisions or their necessary cross-references. Judge the change caused by the edit; do not count unrelated pre-existing defects.
• No unnecessary material change? the proposed edits do not change substantive rights, duties, remedies, scope, duration, definitions, or risk allocation beyond what is reasonably necessary to remove the leakage and keep the drafting coherent. A direct operative replacement for leaked explanatory language is allowed when it preserves the intended legal effect.
• Would you publish this targeted repair? considering the touched language and its necessary consequences, the proposed edits are a net-positive, no-regret correction relative to the pre-edit version and counsel would ship them as a focused patch if that version were already live. Ignore unrelated defects outside the touched language; this is not an absolute whole-document publication-quality judgment.
• Rationale: Record the decisive reason for the rulings, especially any No ruling. This is a repair-focused explanation, not a complete audit of the agreement.

The system-level claim is that a specialized critic can improve the quality-control loop around a stronger drafting model. In a drafting workflow, a critic can block release, identify a precise defect, and return the document for revision. The quality of the released draft depends on that review step much as junior work depends on senior review.

Relation to model ensembles

OpenRouter’s Fusion experiment is adjacent evidence that differently trained models make complementary errors: a synthesis model combined multiple candidate answers and outperformed each contributing frontier model on its domain test.1 Our design is not answer synthesis. It specializes a cheaper verifier on one observed failure mode and feeds that criticism back into drafting quality control. The common proposition is narrower: correlated self-review leaves value on the table.

Limitations

The candidate-verification evaluation is small: 27 rows from ten state agreement files. The row-level exact paired test gives p = 0.001953125; the source-group sign test gives p = 0.0625. The first-pass pilot adds ten source-disjoint documents, but counsel adjudicated the union of proposed findings rather than every clause. Its union-valid coverage is therefore not absolute recall, and a document without a proposed finding is not presumed clean.

Methods and materials

Gemini 3.1 Flash-Lite produced the candidate critiques. The source documents were generated principally with Claude, although exact drafting-model provenance was not retained for every document. Gemini 3.1 Flash-Lite was tuned for 40 epochs with learning-rate multiplier 2 and adapter size four. The blinded repair review used criteria version repair-review-v1.0.0 with definition SHA-256 b43bf99726ea751b9a1ada4c314abaf8bcd39a12123ef6347000b5c1625be4b4.

All public figures on this page are generated from source artifacts pinned in hashed manifests. The first-pass pilot used one temperature-zero call per arm and supplied the full frozen document and rule, but no candidate finding. The underlying lawyer-gold workbook is available upon request.