Abstract
When a language model rates the fairness of a contract provision, does the rating depend on what the clause does, or on who it does it to? We constructed 105 clause groups across three contracting relationships (employment, residential lease, B2B SaaS) in which the two one-sided variants of each clause are the same text with the party role tokens swapped (schematically, 'The Company may terminate this Agreement without notice' versus 'The Employee may terminate this Agreement without notice'), so the two variants are textually identical apart from the party labels. Four frontier models (GPT-5.1, Claude Sonnet 5, Gemini 3.1 Pro Preview, Gemini 3.1 Flash Lite) rated each variant's fairness under the original role labels and under counterbalanced, opaque Party A / Party B labels, over 10,000 determinations in all.
Three of the four models rated the identical text as significantly less fair when it favored the stronger party (employer or landlord): group-level mean asymmetry +0.86 points on a 10-point scale for GPT-5.1 (95% CI 0.51 to 1.31), +0.50 for Claude Sonnet 5, +0.21 for Gemini 3.1 Pro Preview, with Gemini 3.1 Flash Lite showing no reliable tilt (+0.09, n.s.). Under the opaque labels the asymmetry collapsed to between 0.00 and 0.06 in all four models (90 percent confidence intervals, the equivalence-test convention, all inside ±0.13). The tilt is carried by the party role labels.
We do not label the effect bias. A model that treats an employer terminating without notice differently from an employee quitting without notice may be correctly calibrated to asymmetric legal and economic reality even though the sentences match. What this evaluation establishes is narrower: the differential treatment rides on the role-label presentation. It survives when the words are identical, and it vanishes when the role labels, together with the agreement framing that names them, are replaced with opaque ones.
Results
Labeled role names produced a fairness asymmetry in three of four models; opaque Party A / Party B labels collapsed it in all four. Per-model verdicts on the five hypotheses:
| Hypothesis | Flash Lite | Pro Preview | GPT-5.1 | Sonnet 5 |
|---|---|---|---|---|
| H1 — labeled fairness tilt > 0 | no (+0.09) | yes (+0.21) | yes (+0.86) | yes (+0.50) |
| H2 — labels carry part of the tilt | no | yes (+0.19) | yes (+0.80) | yes (+0.51) |
| H3 — the weak-party label specifically | no | no (−0.01) | yes (+0.24) | no |
| H4 — no text-level tilt under Party A/B (TOST ±0.5) | yes (0.00) | yes (+0.03) | yes (+0.06) | yes (0.00) |
| H5 — advocacy-condition tilt survives anonymization | no | no | no | no |
Decision rule and scope, fixed before data collection: employment and landlord–tenant groups pooled (up to 70 complete-case group pairs per cell; four of the Claude Sonnet 5 cells have n=69 after missing-data exclusions), with the entity-vs-entity B2B family as a descriptive comparator. A hypothesis passes on a group-level mean whose seeded BCa bootstrap 95% CI excludes zero (Holm-corrected across the five hypotheses within each model); the no-text-tilt hypothesis used a two-one-sided-tests (TOST) equivalence criterion at ±0.5 points, reported with the conventional 90 percent intervals.
Three descriptive patterns worth reporting alongside the confirmatory cells. First, the tilt is larger in the individual-versus-entity families than in B2B for three of four models (GPT-5.1: employment 1.29, landlord–tenant 0.43, B2B 0.34). Second, sorting the 35 clause archetypes by labeled asymmetry concentrates the effect in procedural and dignity clauses — a one-way obligation to give written reasons on termination (+2.25, pooled across the four models and both confirmatory families, n=8 model-family cells per archetype), a jury-trial waiver binding one party only (+2.13), a one-way records-retention burden (+2.00) — while most purely economic clauses sit near zero. Third, direction identification is essentially intact without labels: models still read which party a clause favors at 97 to 100 percent accuracy under Party A/B; what changes is how unfair they say it is.
Methods and materials
Mirror-by-construction items. Each of 35 clause archetypes (termination, cure periods, fee shifting, venue, indemnification, consent standards, and so on) is a template with two role slots. The strong-party variant and the weak-party variant are rendered from the same template with the slots filled in opposite ways, so the variants are byte-identical up to role tokens — a validator re-renders every committed item from its template and fails on any byte difference. Each group also carries a mutual base variant that applies the clause to both parties symmetrically, for 315 rated items across the 105 groups. This removes by construction the confound that dominated our pilot: hand-written variant pairs where one side's edit was simply harsher than the other's.
Counterbalanced opaque labels. In the opaque-label conditions, which party becomes Party A versus Party B is assigned per group by seeded randomization, balanced 18/17 within each family, so a generic preference for the first-named party cannot masquerade as a role-label effect.
Analysis plan fixed in advance. The design document — hypotheses H1 through H5 as algebraic group-level estimands, the confirmatory scope, the equivalence margin, the multiplicity correction, and the sampling policy — was committed to version control alone, before any item text existed. The items, generator, validator, and analysis code were committed before the first API call. Post-data amendments are limited to three logged, content-neutral changes: two JSON parse-repair rungs for one model's malformed output (recovering 82 of 97 unparseable responses from cached bytes without re-calling any model), and a third that fixed a display-layer bug in the descriptive report and added the per-archetype descriptive table the design promised but the pre-data commit omitted — each recorded in the design file with its rationale, and none touching any estimand, decision rule, or scope. Missing data after repairs: 15 of 10,080 responses (0.15 percent), never resampled.
What the machinery does not decide. One human-judgment layer remains: each archetype carries a one-line declaration of which slot-holder the one-sided clause favors. These 35 declarations are published with the item set for audit. And the central interpretive question — whether label-conditioned ratings are miscalibration or correct legal judgment — is left open by design; it requires practitioner ground truth, which is the next phase of this work.
Limits
Text symmetry is not consequence symmetry: identical sentences can describe different real-world events depending on who acts, and a fairness rater is entitled to know that. Results are scoped to this fixed 35-archetype benchmark; templates are deliberately family-generic (no salaries, rents, or service-specific terms), which buys the mirror guarantee at some cost in realism. Ratings come from one sample per prompt: the Gemini models were called at temperature 0 with JSON output mode, while the GPT-5.1 and Claude Sonnet 5 calls set no temperature (their current APIs reject the parameter for these models), so no within-prompt variance is measured anywhere. The opaque-label condition removes the role labels and incidental framing (agreement type), so the label effect is measured as a presentation bundle. This evaluation measures reading — rating existing provisions — not drafting; whether generation shows the same label-conditioned tilt is an open question we consider harder and more consequential.
Relation to our other work
Like the institutional-knowledge leakage evaluation, this study reports a narrow, verifiable failure-mode measurement rather than a leaderboard: one mechanism, isolated with controls, negative results included. The expert-correction dataset described in our data paper supplies the practitioner-labeling pipeline that the open calibration question above requires.