Sample · 3 tasks · 1 NPC

Contract Clause Evaluation

Frontier models graded against a written rubric, with explicit obligations, with verdicts that say fit for use or not fit for use.

Preliminary Provisions

Definitions

Intepretation variation 1

fail · strict fail
Coverage
83%
Verdict
FAIL
Confidence
0.98
Tokens
130 / 1,542
in / out

Prompt · Candidate · Rubric

Prompt
Candidate · gpt-5.4 (high)
Rubric · gold standard

Obligation coverage

ObligationPresentEvidenceNote
Interpret the agreement in accordance with its termsNOThe clause provides specific interpretation rules, but it does not state a general obligation to interpret the agreement in accordance with its terms.
Apply good faith in interpretationYESThe candidate's interpretation framework substantively reflects a good-faith approach to construing the agreement.
Consider the economic purpose of each provision in interpretationYESThe candidate's interpretation rules are consistent with construing provisions in light of their economic and commercial purpose.
Consider the parties' common intent at execution in interpretationYESThe candidate's interpretation rules give effect to the parties' common intent at the time of execution.
Disregard headings and titles in interpretationYESThe table of contents, clause headings and any descriptive titles are included for convenience only and shall not affect the interpretation of this Agreement.The clause expressly states that headings and titles do not affect interpretation.
Read singular and plural terms interchangeablyYESwords in the singular include the plural and vice versa;The clause expressly provides that singular and plural terms are to be read interchangeably.
Treat internal references to sections, clauses, annexes, and schedules as references to this agreementYESa reference to a Clause, Section, Schedule, Annex or Appendix is to a clause, section, schedule, annex or appendix of or to this AgreementThis expressly states that internal references to those items are references within this agreement.
Construe illustrative words as non-limitingYESa reference to “including”, “includes”, “in particular”, “for example” or similar words is to be construed without limitation and shall not limit the generality of the words preceding themThe clause expressly states that illustrative words are non-limiting.
Treat references to laws and regulations as including amendments, replacements, and re-enactmentsYESa reference to any statute, law, regulation, ordinance, rule, code or other legislative provision includes ... any amendment, consolidation, re-enactment or replacement of itThe clause expressly states that references to laws and regulations include amendments, re-enactments, and replacements.
Apply assigned meanings to capitalized terms wherever defined in the agreementNOThe clause gives interpretation rules and some definitions, but it does not state that capitalized terms carry their assigned meanings wherever defined in the agreement.
Exclude interpretation against a party based on drafting responsibilityYESNo rule of construction shall apply to the disadvantage of a party because that party was responsible for the preparation of this Agreement or any part of it.This expressly excludes construing the agreement against a party due to having drafted it.
Give precedence to the main body over annexes and schedules unless expressly stated otherwiseYESIf there is any inconsistency between the body of this Agreement and any Schedule, Annex or Appendix, the body of this Agreement shall prevail to the extent of the inconsistency, unless this Agreement expressly provides otherwise.This directly gives the main body precedence over schedules/annexes unless expressly stated otherwise.

Judge reasoning

The candidate is not fit for use against this reference because it omits several core substantive interpretation rules that are central to the gold-standard clause: interpretation in accordance with the agreement's terms, and the rule on capitalized terms having their assigned meanings wherever defined. Those omissions are not cured elsewhere in the draft.

Missing obligations

  • Interpret the agreement in accordance with its terms
  • Apply assigned meanings to capitalized terms wherever defined in the agreement

Hallucinations

  • References to a party include its permitted successors and permitted assigns
  • Periods of days are deemed to mean clear days, with deadlines rolling to the next Business Day

Drafting issues

  • Clause 1(l) and Clause 7 duplicate the same non-limiting 'including/in particular' rule unnecessarily