Agent evaluation · International arbitration

Methodology and results

This report presents an agentic legal evaluation of international-arbitration work across twelve tasks. The harness uses structured task packages and expert rubrics to assess both answer quality and agentic execution.

Scope. We built the evaluation harness, task preparation, scoring implementation and LLM judge independently. Results are specific to the evaluated task packages, candidate models and judge configuration.
RESULTS OVERVIEW

Weighted results across all tasks

The leaderboard ranks models by their equal-weight mean across the twelve task-level weighted rubric scores. It is not an all-or-nothing task pass rate: each score reflects the rubric weight earned, with high-, medium- and low-importance criteria worth 3, 2 and 1 points respectively.

Weighted rubric score

Overall scores give each task equal weight.

0%25%50%75%100%
1
Claude Opus 5
12/12 tasks · 36 runs
80.1%
2
Claude Opus 5.5
12/12 tasks · 36 runs
77.7%
3
Gemini 3.8 Flash
12/12 tasks · 36 runs
73.3%
4
GPT-6 Astra
12/12 tasks · 36 runs
72.5%
5
GPT-5.6 Sol
12/12 tasks · 36 runs
69.4%
6
Claude Sonnet 5
12/12 tasks · 36 runs
68.2%
7
Grok 4.6
12/12 tasks · 36 runs
67.2%
8
Gemini 3.6 Flash
12/12 tasks · 36 runs
65.5%
9
GPT-5.6 Luna
12/12 tasks · 36 runs
61.5%

Green highlights the leading model. On individual tasks, capped lines show the lowest and highest observed scores across runs; they are not confidence intervals.

Detailed task results

The weighted mean is shown first for emphasis. Each task cell gives the mean weighted rubric score and completed-run coverage. Starred figures have incomplete task coverage and are not directly comparable with complete twelve-task means.

ModelWeighted mean001M001L002S002M002L003S005S006M006L007S007M007L
Claude Opus 580.1%12/12 tasks · 36 runs83.5%3/3 runs81.7%3/3 runs88.2%3/3 runs66.7%3/3 runs81.5%3/3 runs88.9%3/3 runs98.2%3/3 runs75.8%3/3 runs75.7%3/3 runs71.1%3/3 runs59.3%3/3 runs90.9%3/3 runs
Claude Opus 5.577.7%12/12 tasks · 36 runs78.5%3/3 runs88.9%3/3 runs77.2%3/3 runs78.8%3/3 runs70.4%3/3 runs80.2%3/3 runs91.9%3/3 runs76.8%3/3 runs80.5%3/3 runs68.7%3/3 runs52.7%3/3 runs87.4%3/3 runs
Gemini 3.8 Flash73.3%12/12 tasks · 36 runs79.7%3/3 runs88.1%3/3 runs91.1%3/3 runs67.7%3/3 runs80.9%3/3 runs81.0%3/3 runs68.5%3/3 runs68.7%3/3 runs56.9%3/3 runs72.7%3/3 runs34.6%3/3 runs89.6%3/3 runs
GPT-6 Astra72.5%12/12 tasks · 36 runs75.9%3/3 runs72.6%3/3 runs76.4%3/3 runs51.9%3/3 runs72.8%3/3 runs84.1%3/3 runs81.1%3/3 runs76.8%3/3 runs81.6%3/3 runs77.1%3/3 runs45.7%3/3 runs74.5%3/3 runs
GPT-5.6 Sol69.4%12/12 tasks · 36 runs68.6%3/3 runs76.2%3/3 runs74.3%3/3 runs37.6%3/3 runs73.5%3/3 runs84.9%3/3 runs82.9%3/3 runs70.2%3/3 runs74.5%3/3 runs66.3%3/3 runs42.8%3/3 runs81.4%3/3 runs
Claude Sonnet 568.2%12/12 tasks · 36 runs76.2%3/3 runs71.0%3/3 runs83.1%3/3 runs46.6%3/3 runs59.9%3/3 runs84.1%3/3 runs79.3%3/3 runs70.7%3/3 runs59.2%3/3 runs71.9%3/3 runs39.9%3/3 runs76.6%3/3 runs
Grok 4.667.2%12/12 tasks · 36 runs67.0%3/3 runs57.1%3/3 runs81.0%3/3 runs47.1%3/3 runs67.3%3/3 runs91.3%3/3 runs74.8%3/3 runs63.1%3/3 runs59.6%3/3 runs85.9%3/3 runs30.5%3/3 runs81.4%3/3 runs
Gemini 3.6 Flash65.5%12/12 tasks · 36 runs71.6%3/3 runs77.4%3/3 runs72.2%3/3 runs48.7%3/3 runs86.4%3/3 runs81.0%3/3 runs65.8%3/3 runs61.1%3/3 runs55.8%3/3 runs60.2%3/3 runs30.0%3/3 runs75.3%3/3 runs
GPT-5.6 Luna61.5%12/12 tasks · 36 runs62.5%3/3 runs65.1%3/3 runs55.7%3/3 runs27.0%3/3 runs66.0%3/3 runs80.2%3/3 runs77.5%3/3 runs66.2%3/3 runs67.4%3/3 runs63.9%3/3 runs33.7%3/3 runs72.7%3/3 runs
Headline score

Percentage of available rubric weight earned in a run, averaged across completed and fully judged runs. High-importance criteria carry 3 points, medium-importance criteria 2 points and low-importance criteria 1 point. A passed criterion earns its full weight; an unmet criterion earns zero.

Range and variation

Per-task ranges show observed differences across runs. With at most three repetitions, they indicate consistency but are not statistical confidence intervals.

Coverage

Three runs were requested per model and task. Blank cells mean no completed work product was available to score.

CROSS-CASE FINDINGS

What the twelve tasks indicate

Quality

Claude Opus 5 had the highest equal-weight twelve-task mean at 80.1%. The next-highest model was Claude Opus 5.5 at 77.7%.

Efficiency trade-off

Claude Opus 5 achieved the strongest quality but used 93.4 mean model turns and 111.7 mean tool calls. Among the remaining models averaging no more than 20 turns per task, GPT-6 Astra provided the strongest quality-efficiency balance: 72.5% with 11.4 turns and 50.1 tool calls.

Task sensitivity

Scores depended strongly on the task. Across the fully covered models, 003S had the highest cross-model mean (84.0%), while 007M was the most difficult (41.0%). Model ranking should therefore be read task by task, not as one universal capability score.

Twelve-task comparison

This table puts overall quality beside the amount of agent activity required to produce it. The score is the equal-weight mean of the twelve task-level weighted scores; turns and tool calls are averaged across those same tasks.

ModelTwelve-task meanMean turnsMean tool calls
Claude Opus 580.1%93.4111.7
Claude Opus 5.577.7%22.942.8
Gemini 3.8 Flash73.3%32.731.7
GPT-6 Astra72.5%11.450.1
GPT-5.6 Sol69.4%6.230.8
Claude Sonnet 568.2%93.2117.9
Grok 4.667.2%7.818.0
Gemini 3.6 Flash65.5%25.724.7
GPT-5.6 Luna61.5%5.022.5
Limitations
  • This is an independently implemented agentic legal evaluation. Results are specific to this harness, task package and judge configuration.
  • The default judge was Claude Sonnet 5. Its assessment of Sonnet candidate outputs creates a potential same-family evaluation effect.
  • Three requested runs per task provide an initial consistency signal, not a statistically robust estimate. Reported scores use completed, fully judged runs only.
  • Candidate models used the same task materials, agent tools and output contract, but provider-specific reasoning and runtime settings were not identical.
  • Rubric categories summarize criterion outcomes; they do not replace expert review of the generated legal work products.
METHODOLOGY

How the agent harness and evaluation worked

The tasks were derived from real-world international arbitration matters and organized into controlled matter rooms. Client-facing case descriptions and downloadable work products in this report are anonymized.

1. Task construction

Under this agentic legal evaluation structure, each task combined a client-style instruction, a controlled matter room and an expert rubric. The twelve tasks covered two deliverables in Case 001, three in Case 002, one each in Cases 003 and 005, and two in Case 006, and three in Case 007. Where a reference answer was available, it supported task and rubric design but was not shown to the candidate agent.

2. Small, medium and long tasks

The labels describe the scope of the work rather than the quality expected. Small tasks are focused assignments with a compact deliverable; medium tasks require broader analysis or evidence review across more materials; long tasks require the most extensive source synthesis and a fuller legal work product. The labels are relative within this evaluation and do not impose one universal page or token limit.

3. Controlled agent harness

For each run, the harness created an isolated workspace and gave the candidate model the task, matter files and a defined DOCX output contract. Six active tools supported source inventory, bounded source reading, matter-room search, spreadsheet inspection, calculation and workspace notes. A seventh registered tool for external legal research was disabled for these evaluations. The reference answer and rubric were withheld from the candidate.

Case 007 used locally validated OCR for scanned pages. The 007S Opus 5.5 and 007L Grok follow-up cohorts added two precise line/page reading tools (eight active tools, nine registered), task-derived evidence-coverage guidance and a final-answer completion check. Other displayed groups retain their recorded harness version; these execution differences should be considered when comparing results. The default judge and expert rubrics were unchanged.

4. Independent executions

This report includes three completed independent runs for every candidate model and task. Each run started from the same instructions and matter materials without access to another run's work. The harness retained the submitted DOCX and recorded operational measures including elapsed time, model turns, tool calls, sources read and citation support.

5. LLM judge evaluation

After a candidate submitted its final document, the harness passed that work product—together with the task, expert rubric and reference evidence—to the default judge, Claude Sonnet 5. The judge evaluated every rubric criterion individually, recording whether it was satisfied and grounding the decision in the candidate response and reference materials. For 002M, compound criteria were checked component by component, with automated source-quote validation and an AI-assisted consistency review; all required components had to be satisfied. Equivalent wording counted unless the rubric expressly required an exact phrase. The judge assessed the completed output only; it did not participate in or alter the candidate run.

6. Validation and reporting

Deterministic checks confirmed that the required artifact existed and opened successfully. A run entered the results only after both the candidate artifact and criterion-level judgment were complete. The headline percentage is the rubric weight earned divided by the total available weight: high-importance criteria are worth 3 points, medium-importance criteria 2 points and low-importance criteria 1 point. Task results average that weighted score across completed runs; category results group the underlying criterion decisions by rubric theme.

CHANGE LOG

Evaluation releases

The current Version 1.2 contains twelve tasks, nine candidate models and three completed, fully judged runs per model and task (324 displayed runs). Earlier attempts and superseded cohorts remain retained internally.

Version 1.2Current · September 30, 2026
  • Extended Cases 001–003 with 005S, 006M/006L and the updated 007S/007M/007L task packages.
  • Added Claude Opus 5.5 and Gemini 3.8 Flash, bringing coverage to nine candidate models.
  • Refreshed 001M/001L and 002S/002M/002L using the September 30 task packages; added a full evaluation of 006M.
  • Reviewed 002M rubric decisions component by component and validated supporting source quotes. Its approved latest three-run cohorts remain unchanged.
  • Added Case 007 (007S, 007M and 007L), evaluated across all nine models with three runs per model/task.
  • Recalculated overall results across all twelve tasks with equal task weighting; task scores retain the rubric’s 3/2/1 importance weights.
Version 1.1Previous
  • Expanded the preliminary release to Cases 001–003.
  • Added GPT-6 Astra and Gemini 3.6 Flash.
  • Included controlled-harness reruns following task and source-configuration corrections.
  • Open archived v1.1 report →
Version 1.0Historical
  • Initial preliminary single-task evaluation, retained as the original published snapshot.
  • Its preliminary results were superseded by subsequent validated evaluations.
  • Open archived v1.0 report →

Contract dispute strategy

A multilingual telecommunications dispute requiring a respondent pleading and a client strategy memorandum. Matter languages: English and French.

TASK 001M

Respondent pleading

Draft a structured respondent pleading addressing the contractual history, termination, waiver, quantum, counterclaims, procedure and relief sought.

Model rankingMean weighted rubric score
1
Claude Opus 5
3 completed runs · range 82.8%–85.1%
83.5%
2
Gemini 3.8 Flash
3 completed runs · range 77.0%–81.6%
79.7%
3
Claude Opus 5.5
3 completed runs · range 78.2%–79.3%
78.5%
4
Claude Sonnet 5
3 completed runs · range 72.4%–79.3%
76.2%
5
GPT-6 Astra
3 completed runs · range 73.6%–77.0%
75.9%
6
Gemini 3.6 Flash
3 completed runs · range 69.0%–75.9%
71.6%
7
GPT-5.6 Sol
3 completed runs · range 67.8%–70.1%
68.6%
8
Grok 4.6
3 completed runs · range 64.4%–70.1%
67.0%
9
GPT-5.6 Luna
3 completed runs · range 62.1%–63.2%
62.5%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Claude Opus 5 recorded the highest mean score at 83.5%. The difference between the highest and lowest completed model means was 21.1 points. GPT-5.6 Luna had the lowest mean candidate latency (69.8 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Claude Opus 583.5%786.8 s79.7102.748.7100.0%
Gemini 3.8 Flash79.7%168.4 s38.337.329.7100.0%
Claude Opus 5.578.5%401.9 s18.047.733.3100.0%
Claude Sonnet 576.2%274.2 s51.779.047.0100.0%
GPT-6 Astra75.9%481.7 s13.060.039.3100.0%
Gemini 3.6 Flash71.6%128.5 s26.025.016.0100.0%
GPT-5.6 Sol68.6%150.6 s6.730.022.3100.0%
Grok 4.667.0%85.9 s6.314.37.3100.0%
GPT-5.6 Luna62.5%69.8 s4.317.715.0100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Claude Opus 5
Relief & Remedies
50.0%
Template & Procedural Compliance
70.4%
Citations & Evidentiary Grounding
80.0%
Factual Chronology of the Dispute
85.7%
Document Selection & Sourcing
100.0%
Governing Legal Framework
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Gemini 3.8 Flash
Relief & Remedies
50.0%
Document Selection & Sourcing
66.7%
Template & Procedural Compliance
74.1%
Factual Chronology of the Dispute
76.2%
Citations & Evidentiary Grounding
80.0%
Governing Legal Framework
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Claude Opus 5.5
Document Selection & Sourcing
50.0%
Citations & Evidentiary Grounding
60.0%
Template & Procedural Compliance
63.0%
Factual Chronology of the Dispute
85.7%
Legal Analysis - Core Issues
86.7%
Governing Legal Framework
100.0%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
Claude Sonnet 5
Relief & Remedies
50.0%
Template & Procedural Compliance
63.0%
Factual Chronology of the Dispute
66.7%
Citations & Evidentiary Grounding
80.0%
Legal Analysis - Core Issues
86.7%
Document Selection & Sourcing
100.0%
Governing Legal Framework
100.0%
Parties & Procedural History
100.0%
GPT-6 Astra
Document Selection & Sourcing
50.0%
Template & Procedural Compliance
59.3%
Citations & Evidentiary Grounding
60.0%
Factual Chronology of the Dispute
81.0%
Relief & Remedies
83.3%
Legal Analysis - Core Issues
86.7%
Governing Legal Framework
100.0%
Parties & Procedural History
100.0%
Gemini 3.6 Flash
Document Selection & Sourcing
50.0%
Relief & Remedies
50.0%
Citations & Evidentiary Grounding
60.0%
Factual Chronology of the Dispute
61.9%
Template & Procedural Compliance
63.0%
Governing Legal Framework
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
GPT-5.6 Sol
Relief & Remedies
33.3%
Document Selection & Sourcing
50.0%
Citations & Evidentiary Grounding
53.3%
Template & Procedural Compliance
63.0%
Factual Chronology of the Dispute
66.7%
Legal Analysis - Core Issues
80.0%
Governing Legal Framework
100.0%
Parties & Procedural History
100.0%
Grok 4.6
Document Selection & Sourcing
50.0%
Relief & Remedies
50.0%
Citations & Evidentiary Grounding
53.3%
Factual Chronology of the Dispute
57.1%
Template & Procedural Compliance
63.0%
Legal Analysis - Core Issues
86.7%
Governing Legal Framework
100.0%
Parties & Procedural History
100.0%
GPT-5.6 Luna
Factual Chronology of the Dispute
38.1%
Citations & Evidentiary Grounding
46.7%
Document Selection & Sourcing
50.0%
Relief & Remedies
50.0%
Template & Procedural Compliance
59.3%
Governing Legal Framework
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

TASK 001L

Client memorandum

Prepare a confidential and privileged client memorandum on the contractual dispute, legal and financial risks, and recommended strategy, using verified facts and contractual wording.

Model rankingMean weighted rubric score
1
Claude Opus 5.5
3 completed runs · range 85.7%–91.7%
88.9%
2
Gemini 3.8 Flash
3 completed runs · range 88.1%–88.1%
88.1%
3
Claude Opus 5
3 completed runs · range 76.2%–85.7%
81.7%
4
Gemini 3.6 Flash
3 completed runs · range 76.2%–78.6%
77.4%
5
GPT-5.6 Sol
3 completed runs · range 71.4%–78.6%
76.2%
6
GPT-6 Astra
3 completed runs · range 63.1%–83.3%
72.6%
7
Claude Sonnet 5
3 completed runs · range 65.5%–75.0%
71.0%
8
GPT-5.6 Luna
3 completed runs · range 59.5%–72.6%
65.1%
9
Grok 4.6
3 completed runs · range 38.1%–72.6%
57.1%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Claude Opus 5.5 recorded the highest mean score at 88.9%. The difference between the highest and lowest completed model means was 31.7 points. GPT-5.6 Luna had the lowest mean candidate latency (52.3 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Claude Opus 5.588.9%243.0 s15.031.720.398.8%
Gemini 3.8 Flash88.1%216.4 s32.731.719.097.5%
Claude Opus 581.7%498.9 s80.3102.052.0100.0%
Gemini 3.6 Flash77.4%157.9 s35.334.312.394.3%
GPT-5.6 Sol76.2%146.9 s7.732.314.798.6%
GPT-6 Astra72.6%358.8 s11.350.729.395.3%
Claude Sonnet 571.0%183.7 s40.759.033.398.4%
GPT-5.6 Luna65.1%52.3 s6.325.711.398.5%
Grok 4.657.1%84.2 s11.326.010.394.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Claude Opus 5.5
Governing Legal Framework
50.0%
Factual Chronology of the Dispute
75.0%
Template & Procedural Compliance
80.0%
Document Selection & Sourcing
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
Gemini 3.8 Flash
Governing Legal Framework
50.0%
Factual Chronology of the Dispute
75.0%
Template & Procedural Compliance
80.0%
Document Selection & Sourcing
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
Claude Opus 5
Parties & Procedural History
33.3%
Governing Legal Framework
50.0%
Factual Chronology of the Dispute
58.3%
Template & Procedural Compliance
80.0%
Document Selection & Sourcing
94.4%
Legal Analysis - Core Issues
100.0%
Relief & Remedies
100.0%
Gemini 3.6 Flash
Factual Chronology of the Dispute
50.0%
Governing Legal Framework
50.0%
Template & Procedural Compliance
80.0%
Legal Analysis - Core Issues
85.2%
Document Selection & Sourcing
94.4%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
GPT-5.6 Sol
Factual Chronology of the Dispute
50.0%
Governing Legal Framework
50.0%
Document Selection & Sourcing
77.8%
Template & Procedural Compliance
80.0%
Legal Analysis - Core Issues
96.3%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
GPT-6 Astra
Governing Legal Framework
16.7%
Factual Chronology of the Dispute
62.5%
Document Selection & Sourcing
66.7%
Parties & Procedural History
66.7%
Template & Procedural Compliance
80.0%
Legal Analysis - Core Issues
88.9%
Relief & Remedies
100.0%
Claude Sonnet 5
Governing Legal Framework
0.0%
Factual Chronology of the Dispute
50.0%
Template & Procedural Compliance
80.0%
Legal Analysis - Core Issues
81.5%
Document Selection & Sourcing
88.9%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
GPT-5.6 Luna
Governing Legal Framework
16.7%
Parties & Procedural History
33.3%
Factual Chronology of the Dispute
45.8%
Document Selection & Sourcing
72.2%
Legal Analysis - Core Issues
77.8%
Template & Procedural Compliance
80.0%
Relief & Remedies
100.0%
Grok 4.6
Governing Legal Framework
16.7%
Factual Chronology of the Dispute
29.2%
Parties & Procedural History
33.3%
Document Selection & Sourcing
50.0%
Relief & Remedies
66.7%
Template & Procedural Compliance
80.0%
Legal Analysis - Core Issues
81.5%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

CAIP arbitration defence

A cross-border sale-of-goods arbitration involving force majeure, exhibit assessment and a substantive reply. Matter languages: English, French, Spanish and Chinese.

TASK 002S

Force majeure memorandum

Prepare a five-to-six-page English memorandum on contractual force majeure under French law for a respondent in a CAIP arbitration.

Model rankingMean weighted rubric score
1
Gemini 3.8 Flash
3 completed runs · range 86.1%–97.5%
91.1%
2
Claude Opus 5
3 completed runs · range 84.8%–89.9%
88.2%
3
Claude Sonnet 5
3 completed runs · range 69.6%–91.1%
83.1%
4
Grok 4.6
3 completed runs · range 79.7%–82.3%
81.0%
5
Claude Opus 5.5
3 completed runs · range 63.3%–87.3%
77.2%
6
GPT-6 Astra
3 completed runs · range 74.7%–78.5%
76.4%
7
GPT-5.6 Sol
3 completed runs · range 72.2%–77.2%
74.3%
8
Gemini 3.6 Flash
3 completed runs · range 70.9%–74.7%
72.2%
9
GPT-5.6 Luna
3 completed runs · range 50.6%–62.0%
55.7%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Gemini 3.8 Flash recorded the highest mean score at 91.1%. The difference between the highest and lowest completed model means was 35.4 points. GPT-5.6 Luna had the lowest mean candidate latency (102.1 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Gemini 3.8 Flash91.1%175.9 s34.333.39.0100.0%
Claude Opus 588.2%920.4 s164.7183.369.0100.0%
Claude Sonnet 583.1%478.8 s108.7148.059.0100.0%
Grok 4.681.0%168.9 s7.315.04.0100.0%
Claude Opus 5.577.2%308.4 s24.343.724.0100.0%
GPT-6 Astra76.4%351.6 s13.056.735.7100.0%
GPT-5.6 Sol74.3%131.6 s6.340.016.7100.0%
Gemini 3.6 Flash72.2%133.4 s20.719.711.093.3%
GPT-5.6 Luna55.7%102.1 s5.028.722.389.6%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Gemini 3.8 Flash
Document Selection & Sourcing
0.0%
Legal Authority & Precedent
66.7%
Template & Procedural Compliance
80.0%
Governing Legal Framework
95.2%
Citations & Evidentiary Grounding
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Legal Analysis - Core Issues
100.0%
Negative Space
100.0%
Relief & Remedies
100.0%
Claude Opus 5
Document Selection & Sourcing
0.0%
Template & Procedural Compliance
60.0%
Governing Legal Framework
90.5%
Citations & Evidentiary Grounding
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Legal Analysis - Core Issues
100.0%
Legal Authority & Precedent
100.0%
Negative Space
100.0%
Relief & Remedies
100.0%
Claude Sonnet 5
Document Selection & Sourcing
0.0%
Negative Space
77.8%
Governing Legal Framework
81.0%
Citations & Evidentiary Grounding
83.3%
Legal Authority & Precedent
83.3%
Template & Procedural Compliance
86.7%
Legal Analysis - Core Issues
87.5%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Relief & Remedies
100.0%
Grok 4.6
Document Selection & Sourcing
33.3%
Negative Space
55.6%
Factual Chronology of the Dispute
66.7%
Legal Analysis - Core Issues
79.2%
Governing Legal Framework
85.7%
Template & Procedural Compliance
86.7%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
Legal Authority & Precedent
100.0%
Relief & Remedies
100.0%
Claude Opus 5.5
Document Selection & Sourcing
0.0%
Legal Authority & Precedent
50.0%
Template & Procedural Compliance
73.3%
Negative Space
77.8%
Legal Analysis - Core Issues
79.2%
Governing Legal Framework
81.0%
Citations & Evidentiary Grounding
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Relief & Remedies
100.0%
GPT-6 Astra
Document Selection & Sourcing
33.3%
Legal Authority & Precedent
50.0%
Factual Chronology of the Dispute
66.7%
Negative Space
66.7%
Governing Legal Framework
76.2%
Legal Analysis - Core Issues
79.2%
Template & Procedural Compliance
80.0%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
Relief & Remedies
100.0%
GPT-5.6 Sol
Document Selection & Sourcing
0.0%
Negative Space
33.3%
Legal Authority & Precedent
50.0%
Legal Analysis - Core Issues
79.2%
Template & Procedural Compliance
80.0%
Governing Legal Framework
85.7%
Citations & Evidentiary Grounding
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Relief & Remedies
100.0%
Gemini 3.6 Flash
Document Selection & Sourcing
0.0%
Hedging Guards
0.0%
Negative Space
44.4%
Legal Authority & Precedent
50.0%
Governing Legal Framework
71.4%
Template & Procedural Compliance
80.0%
Legal Analysis - Core Issues
87.5%
Citations & Evidentiary Grounding
100.0%
Factual Chronology of the Dispute
100.0%
Relief & Remedies
100.0%
GPT-5.6 Luna
Document Selection & Sourcing
0.0%
Factual Chronology of the Dispute
0.0%
Negative Space
33.3%
Legal Authority & Precedent
50.0%
Relief & Remedies
50.0%
Legal Analysis - Core Issues
58.3%
Governing Legal Framework
61.9%
Citations & Evidentiary Grounding
66.7%
Template & Procedural Compliance
66.7%
Hedging Guards
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

TASK 002M

Exhibit review

Assess the evidential weight of each exhibit MA001–MA017 submitted with the claimant’s case and present the assessment as a usable table.

Model rankingMean weighted rubric score
1
Claude Opus 5.5
3 completed runs · range 69.8%–85.7%
78.8%
2
Gemini 3.8 Flash
3 completed runs · range 66.7%–69.8%
67.7%
3
Claude Opus 5
3 completed runs · range 58.7%–73.0%
66.7%
4
GPT-6 Astra
3 completed runs · range 46.0%–57.1%
51.9%
5
Gemini 3.6 Flash
3 completed runs · range 46.0%–52.4%
48.7%
6
Grok 4.6
3 completed runs · range 41.3%–52.4%
47.1%
7
Claude Sonnet 5
3 completed runs · range 38.1%–50.8%
46.6%
8
GPT-5.6 Sol
3 completed runs · range 34.9%–39.7%
37.6%
9
GPT-5.6 Luna
3 completed runs · range 19.0%–31.7%
27.0%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Claude Opus 5.5 recorded the highest mean score at 78.8%. The difference between the highest and lowest completed model means was 51.9 points. Grok 4.6 had the lowest mean candidate latency (63.1 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Claude Opus 5.578.8%457.3 s26.740.317.3100.0%
Gemini 3.8 Flash67.7%181.9 s26.725.720.7100.0%
Claude Opus 566.7%658.2 s79.3108.357.0100.0%
GPT-6 Astra51.9%423.1 s13.060.737.3100.0%
Gemini 3.6 Flash48.7%133.7 s15.014.012.0100.0%
Grok 4.647.1%63.1 s6.010.35.797.2%
Claude Sonnet 546.6%298.2 s86.0105.314.397.0%
GPT-5.6 Sol37.6%107.0 s4.716.39.7100.0%
GPT-5.6 Luna27.0%74.1 s4.09.77.3100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Claude Opus 5.5
Hedging Guards
0.0%
Legal Authority & Precedent
33.3%
Legal Analysis - Core Issues
66.7%
Negative Space
66.7%
Governing Legal Framework
91.7%
Citations & Evidentiary Grounding
100.0%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Gemini 3.8 Flash
Hedging Guards
0.0%
Negative Space
33.3%
Legal Analysis - Core Issues
38.9%
Governing Legal Framework
75.0%
Legal Authority & Precedent
83.3%
Factual Chronology of the Dispute
88.9%
Citations & Evidentiary Grounding
100.0%
Document Selection & Sourcing
100.0%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Claude Opus 5
Hedging Guards
0.0%
Legal Authority & Precedent
33.3%
Negative Space
33.3%
Legal Analysis - Core Issues
38.9%
Governing Legal Framework
75.0%
Citations & Evidentiary Grounding
100.0%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
GPT-6 Astra
Hedging Guards
0.0%
Legal Authority & Precedent
0.0%
Parties & Procedural History
0.0%
Factual Chronology of the Dispute
33.3%
Legal Analysis - Core Issues
33.3%
Document Selection & Sourcing
50.0%
Governing Legal Framework
75.0%
Negative Space
83.3%
Citations & Evidentiary Grounding
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Gemini 3.6 Flash
Hedging Guards
0.0%
Legal Authority & Precedent
0.0%
Legal Analysis - Core Issues
16.7%
Factual Chronology of the Dispute
44.4%
Governing Legal Framework
50.0%
Negative Space
50.0%
Citations & Evidentiary Grounding
100.0%
Document Selection & Sourcing
100.0%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Grok 4.6
Hedging Guards
0.0%
Legal Analysis - Core Issues
16.7%
Legal Authority & Precedent
16.7%
Factual Chronology of the Dispute
22.2%
Parties & Procedural History
33.3%
Negative Space
50.0%
Governing Legal Framework
58.3%
Citations & Evidentiary Grounding
100.0%
Document Selection & Sourcing
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Claude Sonnet 5
Hedging Guards
0.0%
Legal Authority & Precedent
0.0%
Legal Analysis - Core Issues
11.1%
Negative Space
33.3%
Governing Legal Framework
58.3%
Document Selection & Sourcing
66.7%
Relief & Remedies
66.7%
Template & Procedural Compliance
77.8%
Factual Chronology of the Dispute
88.9%
Citations & Evidentiary Grounding
100.0%
Parties & Procedural History
100.0%
GPT-5.6 Sol
Factual Chronology of the Dispute
0.0%
Hedging Guards
0.0%
Legal Authority & Precedent
0.0%
Parties & Procedural History
0.0%
Legal Analysis - Core Issues
27.8%
Document Selection & Sourcing
50.0%
Governing Legal Framework
50.0%
Negative Space
50.0%
Template & Procedural Compliance
55.6%
Citations & Evidentiary Grounding
100.0%
Relief & Remedies
100.0%
GPT-5.6 Luna
Hedging Guards
0.0%
Legal Authority & Precedent
0.0%
Parties & Procedural History
0.0%
Factual Chronology of the Dispute
11.1%
Legal Analysis - Core Issues
11.1%
Governing Legal Framework
16.7%
Negative Space
33.3%
Document Selection & Sourcing
50.0%
Template & Procedural Compliance
77.8%
Citations & Evidentiary Grounding
100.0%
Relief & Remedies
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

TASK 002L

Arbitration reply

Draft an approximately 20-page English reply using the supplied legal authorities RL-1–RL-23 and client factual materials R-1–R-24.

Model rankingMean weighted rubric score
1
Gemini 3.6 Flash
3 completed runs · range 85.2%–88.9%
86.4%
2
Claude Opus 5
3 completed runs · range 77.8%–83.3%
81.5%
3
Gemini 3.8 Flash
3 completed runs · range 79.6%–83.3%
80.9%
4
GPT-5.6 Sol
3 completed runs · range 68.5%–79.6%
73.5%
5
GPT-6 Astra
3 completed runs · range 68.5%–79.6%
72.8%
6
Claude Opus 5.5
3 completed runs · range 66.7%–72.2%
70.4%
7
Grok 4.6
3 completed runs · range 53.7%–77.8%
67.3%
8
GPT-5.6 Luna
3 completed runs · range 63.0%–72.2%
66.0%
9
Claude Sonnet 5
3 completed runs · range 55.6%–66.7%
59.9%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Gemini 3.6 Flash recorded the highest mean score at 86.4%. The difference between the highest and lowest completed model means was 26.5 points. Grok 4.6 had the lowest mean candidate latency (107.0 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Gemini 3.6 Flash86.4%249.9 s44.743.741.7100.0%
Claude Opus 581.5%1,142.9 s120.7160.779.0100.0%
Gemini 3.8 Flash80.9%250.4 s57.756.751.0100.0%
GPT-5.6 Sol73.5%241.9 s8.363.057.7100.0%
GPT-6 Astra72.8%658.9 s18.0112.761.3100.0%
Claude Opus 5.570.4%674.1 s41.088.055.3100.0%
Grok 4.667.3%107.0 s5.312.79.097.4%
GPT-5.6 Luna66.0%271.1 s5.347.746.0100.0%
Claude Sonnet 559.9%298.1 s31.777.760.0100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Gemini 3.6 Flash
Governing Legal Framework
50.0%
Relief & Remedies
55.6%
Negative Space
83.3%
Template & Procedural Compliance
86.7%
Citations & Evidentiary Grounding
100.0%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Claude Opus 5
Negative Space
33.3%
Governing Legal Framework
50.0%
Relief & Remedies
66.7%
Template & Procedural Compliance
80.0%
Citations & Evidentiary Grounding
100.0%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Gemini 3.8 Flash
Governing Legal Framework
50.0%
Negative Space
50.0%
Relief & Remedies
55.6%
Template & Procedural Compliance
86.7%
Legal Analysis - Core Issues
93.3%
Citations & Evidentiary Grounding
100.0%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Parties & Procedural History
100.0%
GPT-5.6 Sol
Negative Space
16.7%
Parties & Procedural History
33.3%
Governing Legal Framework
50.0%
Citations & Evidentiary Grounding
66.7%
Relief & Remedies
66.7%
Template & Procedural Compliance
80.0%
Legal Analysis - Core Issues
93.3%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
GPT-6 Astra
Parties & Procedural History
0.0%
Citations & Evidentiary Grounding
33.3%
Governing Legal Framework
50.0%
Negative Space
50.0%
Factual Chronology of the Dispute
66.7%
Relief & Remedies
66.7%
Template & Procedural Compliance
73.3%
Document Selection & Sourcing
100.0%
Hedging Guards
100.0%
Legal Analysis - Core Issues
100.0%
Claude Opus 5.5
Negative Space
0.0%
Governing Legal Framework
50.0%
Citations & Evidentiary Grounding
66.7%
Relief & Remedies
66.7%
Legal Analysis - Core Issues
80.0%
Template & Procedural Compliance
80.0%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Parties & Procedural History
100.0%
Grok 4.6
Citations & Evidentiary Grounding
0.0%
Governing Legal Framework
33.3%
Negative Space
33.3%
Relief & Remedies
55.6%
Parties & Procedural History
66.7%
Template & Procedural Compliance
73.3%
Legal Analysis - Core Issues
86.7%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
GPT-5.6 Luna
Parties & Procedural History
0.0%
Negative Space
16.7%
Citations & Evidentiary Grounding
33.3%
Governing Legal Framework
50.0%
Relief & Remedies
66.7%
Template & Procedural Compliance
73.3%
Legal Analysis - Core Issues
80.0%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Claude Sonnet 5
Citations & Evidentiary Grounding
0.0%
Negative Space
0.0%
Governing Legal Framework
33.3%
Relief & Remedies
55.6%
Legal Analysis - Core Issues
66.7%
Template & Procedural Compliance
80.0%
Document Selection & Sourcing
100.0%
Factual Chronology of the Dispute
100.0%
Hedging Guards
100.0%
Parties & Procedural History
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

Arbitration clause drafting

A clause-drafting exercise for an international telecommunications agreement, applying client instructions to governing-law and dispute-resolution provisions. Matter languages: English and French.

Input documents used for the reported runs
  • Client instructions
  • Pseudonymized roaming framework agreement
  • ICC Arbitration Rules 2026 / Mediation Rules 2014
  • SCC Arbitration Rules 2023
TASK 003S

Clause drafting

Revise governing-law and arbitration clauses in a roaming framework agreement, including French law, Paris arbitration, SCC procedure, a sole arbitrator and emergency relief.

Model rankingMean weighted rubric score
1
Grok 4.6
3 completed runs · range 90.5%–92.9%
91.3%
2
Claude Opus 5
3 completed runs · range 85.7%–90.5%
88.9%
3
GPT-5.6 Sol
3 completed runs · range 78.6%–90.5%
84.9%
4
Claude Sonnet 5
3 completed runs · range 83.3%–85.7%
84.1%
5
GPT-6 Astra
3 completed runs · range 83.3%–85.7%
84.1%
6
Gemini 3.6 Flash
3 completed runs · range 78.6%–85.7%
81.0%
7
Gemini 3.8 Flash
3 completed runs · range 73.8%–85.7%
81.0%
8
GPT-5.6 Luna
3 completed runs · range 73.8%–83.3%
80.2%
9
Claude Opus 5.5
3 completed runs · range 78.6%–83.3%
80.2%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Grok 4.6 recorded the highest mean score at 91.3%. The difference between the highest and lowest completed model means was 11.1 points. GPT-5.6 Luna had the lowest mean candidate latency (44.2 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Grok 4.691.3%57.6 s11.317.33.0100.0%
Claude Opus 588.9%406.2 s44.348.713.0100.0%
GPT-5.6 Sol84.9%72.5 s5.09.73.3100.0%
Claude Sonnet 584.1%499.2 s176.7182.03.0100.0%
GPT-6 Astra84.1%259.7 s7.716.05.3100.0%
Gemini 3.6 Flash81.0%49.5 s12.011.03.0100.0%
Gemini 3.8 Flash81.0%105.3 s21.020.03.7100.0%
GPT-5.6 Luna80.2%44.2 s6.312.04.3100.0%
Claude Opus 5.580.2%224.5 s12.315.34.0100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Grok 4.6
Template & Procedural Compliance
66.7%
Legal Analysis - Core Issues
83.3%
Document Selection & Sourcing
91.7%
Governing Legal Framework
100.0%
Claude Opus 5
Document Selection & Sourcing
83.3%
Legal Analysis - Core Issues
88.9%
Governing Legal Framework
100.0%
Template & Procedural Compliance
100.0%
GPT-5.6 Sol
Template & Procedural Compliance
66.7%
Legal Analysis - Core Issues
77.8%
Document Selection & Sourcing
83.3%
Governing Legal Framework
100.0%
Claude Sonnet 5
Template & Procedural Compliance
66.7%
Document Selection & Sourcing
79.2%
Legal Analysis - Core Issues
83.3%
Governing Legal Framework
100.0%
GPT-6 Astra
Template & Procedural Compliance
66.7%
Document Selection & Sourcing
79.2%
Legal Analysis - Core Issues
83.3%
Governing Legal Framework
100.0%
Gemini 3.6 Flash
Legal Analysis - Core Issues
66.7%
Document Selection & Sourcing
79.2%
Governing Legal Framework
100.0%
Template & Procedural Compliance
100.0%
Gemini 3.8 Flash
Template & Procedural Compliance
33.3%
Legal Analysis - Core Issues
77.8%
Document Selection & Sourcing
79.2%
Governing Legal Framework
100.0%
GPT-5.6 Luna
Template & Procedural Compliance
66.7%
Document Selection & Sourcing
75.0%
Legal Analysis - Core Issues
77.8%
Governing Legal Framework
100.0%
Claude Opus 5.5
Template & Procedural Compliance
33.3%
Document Selection & Sourcing
75.0%
Legal Analysis - Core Issues
83.3%
Governing Legal Framework
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

Arbitrator fee determination

An Italian ad hoc arbitration fee dispute requiring analysis of non-acceptance, Article 814(2), the available fee evidence and the documented procedural posture. Matter language: Italian.

TASK 005S

Arbitrator fee memorandum

Assess whether an arbitrator's self-liquidated fee binds the parties under Article 814(2) of the Italian Code of Civil Procedure, apply the available fee-scale evidence and describe the documented procedural status.

Model rankingMean weighted rubric score
1
Claude Opus 5
3 completed runs · range 94.6%–100.0%
98.2%
2
Claude Opus 5.5
3 completed runs · range 91.9%–91.9%
91.9%
3
GPT-5.6 Sol
3 completed runs · range 81.1%–86.5%
82.9%
4
GPT-6 Astra
3 completed runs · range 81.1%–81.1%
81.1%
5
Claude Sonnet 5
3 completed runs · range 73.0%–83.8%
79.3%
6
GPT-5.6 Luna
3 completed runs · range 75.7%–81.1%
77.5%
7
Grok 4.6
3 completed runs · range 70.3%–81.1%
74.8%
8
Gemini 3.8 Flash
3 completed runs · range 59.5%–78.4%
68.5%
9
Gemini 3.6 Flash
3 completed runs · range 59.5%–73.0%
65.8%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Claude Opus 5 recorded the highest mean score at 98.2%. The difference between the highest and lowest completed model means was 32.4 points. GPT-5.6 Luna had the lowest mean candidate latency (53.8 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Claude Opus 598.2%368.3 s58.772.713.0100.0%
Claude Opus 5.591.9%206.4 s15.729.310.3100.0%
GPT-5.6 Sol82.9%109.7 s5.328.011.0100.0%
GPT-6 Astra81.1%244.4 s11.044.013.3100.0%
Claude Sonnet 579.3%420.6 s173.0183.010.7100.0%
GPT-5.6 Luna77.5%53.8 s5.318.011.3100.0%
Grok 4.674.8%77.7 s8.725.78.0100.0%
Gemini 3.8 Flash68.5%132.0 s27.026.011.3100.0%
Gemini 3.6 Flash65.8%101.6 s15.314.311.0100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Claude Opus 5
Parties & Procedural History
96.3%
Governing Legal Framework
100.0%
Legal Analysis - Core Issues
100.0%
Legal Authority & Precedent
100.0%
Claude Opus 5.5
Legal Analysis - Core Issues
75.0%
Governing Legal Framework
100.0%
Legal Authority & Precedent
100.0%
Parties & Procedural History
100.0%
GPT-5.6 Sol
Legal Analysis - Core Issues
75.0%
Parties & Procedural History
81.5%
Governing Legal Framework
100.0%
Legal Authority & Precedent
100.0%
GPT-6 Astra
Legal Analysis - Core Issues
75.0%
Parties & Procedural History
77.8%
Governing Legal Framework
100.0%
Legal Authority & Precedent
100.0%
Claude Sonnet 5
Legal Analysis - Core Issues
66.7%
Parties & Procedural History
81.5%
Governing Legal Framework
100.0%
Legal Authority & Precedent
100.0%
GPT-5.6 Luna
Parties & Procedural History
70.4%
Legal Analysis - Core Issues
75.0%
Governing Legal Framework
100.0%
Legal Authority & Precedent
100.0%
Grok 4.6
Legal Analysis - Core Issues
66.7%
Parties & Procedural History
70.4%
Governing Legal Framework
100.0%
Legal Authority & Precedent
100.0%
Gemini 3.8 Flash
Parties & Procedural History
63.0%
Legal Analysis - Core Issues
66.7%
Governing Legal Framework
100.0%
Legal Authority & Precedent
100.0%
Gemini 3.6 Flash
Legal Analysis - Core Issues
58.3%
Parties & Procedural History
63.0%
Governing Legal Framework
100.0%
Legal Authority & Precedent
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

Jurisdiction and merits exposure

An Italian distribution dispute requiring analysis of the tribunal's jurisdiction ruling and a separate opinion on unresolved merits exposure. Matter languages: Italian and English.

TASK 006M

Jurisdiction ruling opinion

Assess what the tribunal's jurisdiction ruling establishes and leaves undecided, and evaluate its grounds using the supplied Italian-language record.

Model rankingMean weighted rubric score
1
Claude Opus 5.5
3 completed runs · range 68.2%–81.8%
76.8%
2
GPT-6 Astra
3 completed runs · range 75.8%–77.3%
76.8%
3
Claude Opus 5
3 completed runs · range 68.2%–81.8%
75.8%
4
Claude Sonnet 5
3 completed runs · range 62.1%–77.3%
70.7%
5
GPT-5.6 Sol
3 completed runs · range 65.2%–77.3%
70.2%
6
Gemini 3.8 Flash
3 completed runs · range 63.6%–74.2%
68.7%
7
GPT-5.6 Luna
3 completed runs · range 63.6%–68.2%
66.2%
8
Grok 4.6
3 completed runs · range 60.6%–65.2%
63.1%
9
Gemini 3.6 Flash
3 completed runs · range 56.1%–65.2%
61.1%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Claude Opus 5.5 recorded the highest mean score at 76.8%. The difference between the highest and lowest completed model means was 15.7 points. Grok 4.6 had the lowest mean candidate latency (53.7 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Claude Opus 5.576.8%193.3 s10.021.716.7100.0%
GPT-6 Astra76.8%357.0 s9.322.017.0100.0%
Claude Opus 575.8%371.4 s44.053.720.7100.0%
Claude Sonnet 570.7%181.7 s64.075.715.0100.0%
GPT-5.6 Sol70.2%180.7 s4.318.315.0100.0%
Gemini 3.8 Flash68.7%126.6 s21.020.015.0100.0%
GPT-5.6 Luna66.2%57.7 s3.016.015.0100.0%
Grok 4.663.1%53.7 s5.011.710.7100.0%
Gemini 3.6 Flash61.1%101.1 s19.318.315.3100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Claude Opus 5.5
Legal Authority & Precedent
33.3%
Governing Legal Framework
57.1%
Parties & Procedural History
80.0%
Legal Analysis - Core Issues
83.3%
Hedging Guards
88.9%
Citations & Evidentiary Grounding
100.0%
Negative Space
100.0%
GPT-6 Astra
Legal Authority & Precedent
0.0%
Governing Legal Framework
38.1%
Negative Space
66.7%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Claude Opus 5
Legal Authority & Precedent
0.0%
Governing Legal Framework
57.1%
Negative Space
66.7%
Parties & Procedural History
73.3%
Legal Analysis - Core Issues
94.4%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
Claude Sonnet 5
Governing Legal Framework
28.6%
Legal Authority & Precedent
33.3%
Legal Analysis - Core Issues
83.3%
Negative Space
83.3%
Parties & Procedural History
86.7%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
GPT-5.6 Sol
Legal Authority & Precedent
0.0%
Governing Legal Framework
42.9%
Parties & Procedural History
60.0%
Negative Space
83.3%
Legal Analysis - Core Issues
94.4%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
Gemini 3.8 Flash
Legal Authority & Precedent
0.0%
Governing Legal Framework
28.6%
Citations & Evidentiary Grounding
66.7%
Negative Space
66.7%
Parties & Procedural History
73.3%
Legal Analysis - Core Issues
94.4%
Hedging Guards
100.0%
GPT-5.6 Luna
Governing Legal Framework
23.8%
Legal Authority & Precedent
66.7%
Legal Analysis - Core Issues
72.2%
Parties & Procedural History
80.0%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
Negative Space
100.0%
Grok 4.6
Legal Authority & Precedent
0.0%
Governing Legal Framework
28.6%
Parties & Procedural History
60.0%
Legal Analysis - Core Issues
83.3%
Negative Space
83.3%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
Gemini 3.6 Flash
Governing Legal Framework
9.5%
Legal Authority & Precedent
33.3%
Parties & Procedural History
60.0%
Legal Analysis - Core Issues
83.3%
Citations & Evidentiary Grounding
100.0%
Hedging Guards
100.0%
Negative Space
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

TASK 006L

Merits exposure opinion

Assess the respondent's exposure to contractual pecuniary-loss and reputational-damage claims after an arbitral tribunal declined jurisdiction, using only the supplied Italian-language record and without predicting a future merits outcome.

Model rankingMean weighted rubric score
1
GPT-6 Astra
3 completed runs · range 79.8%–85.4%
81.6%
2
Claude Opus 5.5
3 completed runs · range 79.8%–82.0%
80.5%
3
Claude Opus 5
3 completed runs · range 69.7%–79.8%
75.7%
4
GPT-5.6 Sol
3 completed runs · range 73.0%–76.4%
74.5%
5
GPT-5.6 Luna
3 completed runs · range 62.9%–70.8%
67.4%
6
Grok 4.6
3 completed runs · range 58.4%–61.8%
59.6%
7
Claude Sonnet 5
3 completed runs · range 52.8%–68.5%
59.2%
8
Gemini 3.8 Flash
3 completed runs · range 49.4%–64.0%
56.9%
9
Gemini 3.6 Flash
3 completed runs · range 50.6%–60.7%
55.8%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. GPT-6 Astra recorded the highest mean score at 81.6%. The difference between the highest and lowest completed model means was 25.8 points. GPT-5.6 Luna had the lowest mean candidate latency (38.3 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
GPT-6 Astra81.6%196.3 s7.723.317.0100.0%
Claude Opus 5.580.5%227.2 s10.724.315.7100.0%
Claude Opus 575.7%229.4 s32.340.018.0100.0%
GPT-5.6 Sol74.5%97.9 s4.316.014.0100.0%
GPT-5.6 Luna67.4%38.3 s4.017.015.0100.0%
Grok 4.659.6%56.7 s5.710.77.7100.0%
Claude Sonnet 559.2%58.2 s5.315.013.797.6%
Gemini 3.8 Flash56.9%124.4 s18.717.715.097.6%
Gemini 3.6 Flash55.8%103.8 s18.017.014.3100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

GPT-6 Astra
Governing Legal Framework
25.0%
Document Selection & Sourcing
50.0%
Parties & Procedural History
58.3%
Hedging Guards
75.0%
Citations & Evidentiary Grounding
77.8%
Legal Analysis - Core Issues
90.5%
Legal Authority & Precedent
100.0%
Negative Space
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Claude Opus 5.5
Governing Legal Framework
50.0%
Parties & Procedural History
62.5%
Legal Authority & Precedent
66.7%
Hedging Guards
75.0%
Citations & Evidentiary Grounding
77.8%
Legal Analysis - Core Issues
81.0%
Document Selection & Sourcing
83.3%
Negative Space
83.3%
Relief & Remedies
93.3%
Template & Procedural Compliance
100.0%
Claude Opus 5
Governing Legal Framework
41.7%
Legal Authority & Precedent
50.0%
Legal Analysis - Core Issues
61.9%
Parties & Procedural History
62.5%
Hedging Guards
75.0%
Citations & Evidentiary Grounding
77.8%
Document Selection & Sourcing
100.0%
Negative Space
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
GPT-5.6 Sol
Governing Legal Framework
25.0%
Parties & Procedural History
37.5%
Document Selection & Sourcing
50.0%
Legal Authority & Precedent
50.0%
Hedging Guards
75.0%
Legal Analysis - Core Issues
76.2%
Citations & Evidentiary Grounding
88.9%
Negative Space
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
GPT-5.6 Luna
Parties & Procedural History
20.8%
Document Selection & Sourcing
50.0%
Governing Legal Framework
50.0%
Legal Authority & Precedent
50.0%
Citations & Evidentiary Grounding
55.6%
Legal Analysis - Core Issues
61.9%
Hedging Guards
75.0%
Relief & Remedies
86.7%
Negative Space
100.0%
Template & Procedural Compliance
100.0%
Grok 4.6
Governing Legal Framework
25.0%
Citations & Evidentiary Grounding
33.3%
Parties & Procedural History
33.3%
Document Selection & Sourcing
50.0%
Legal Authority & Precedent
50.0%
Legal Analysis - Core Issues
52.4%
Hedging Guards
58.3%
Relief & Remedies
93.3%
Negative Space
100.0%
Template & Procedural Compliance
100.0%
Claude Sonnet 5
Citations & Evidentiary Grounding
22.2%
Governing Legal Framework
41.7%
Legal Analysis - Core Issues
42.9%
Hedging Guards
50.0%
Legal Authority & Precedent
50.0%
Parties & Procedural History
58.3%
Document Selection & Sourcing
66.7%
Relief & Remedies
66.7%
Negative Space
100.0%
Template & Procedural Compliance
100.0%
Gemini 3.8 Flash
Governing Legal Framework
16.7%
Citations & Evidentiary Grounding
33.3%
Legal Analysis - Core Issues
47.6%
Document Selection & Sourcing
50.0%
Hedging Guards
50.0%
Legal Authority & Precedent
50.0%
Parties & Procedural History
54.2%
Relief & Remedies
73.3%
Negative Space
83.3%
Template & Procedural Compliance
100.0%
Gemini 3.6 Flash
Governing Legal Framework
8.3%
Citations & Evidentiary Grounding
22.2%
Hedging Guards
41.7%
Document Selection & Sourcing
50.0%
Legal Authority & Precedent
50.0%
Legal Analysis - Core Issues
52.4%
Parties & Procedural History
54.2%
Relief & Remedies
66.7%
Negative Space
100.0%
Template & Procedural Compliance
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

Algerian-seated arbitration response

A dispute over outstanding invoices requiring a matter summary, reply strategy and complete response to the request for arbitration. Matter languages: English and French.

TASK 007S

Matter summary and information request

Summarize the dispute and identify the client documents and information needed to prepare the response.

Model rankingMean weighted rubric score
1
Grok 4.6
3 completed runs · range 81.9%–89.2%
85.9%
2
GPT-6 Astra
3 completed runs · range 74.7%–79.5%
77.1%
3
Gemini 3.8 Flash
3 completed runs · range 67.5%–81.9%
72.7%
4
Claude Sonnet 5
3 completed runs · range 66.3%–75.9%
71.9%
5
Claude Opus 5
3 completed runs · range 61.4%–78.3%
71.1%
6
Claude Opus 5.5
3 completed runs · range 66.3%–69.9%
68.7%
7
GPT-5.6 Sol
3 completed runs · range 57.8%–72.3%
66.3%
8
GPT-5.6 Luna
3 completed runs · range 57.8%–69.9%
63.9%
9
Gemini 3.6 Flash
3 completed runs · range 54.2%–68.7%
60.2%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Grok 4.6 recorded the highest mean score at 85.9%. The difference between the highest and lowest completed model means was 25.7 points. Grok 4.6 had the lowest mean candidate latency (66.7 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Grok 4.685.9%66.7 s7.714.35.3100.0%
GPT-6 Astra77.1%280.2 s11.748.322.3100.0%
Gemini 3.8 Flash72.7%119.4 s31.730.720.0100.0%
Claude Sonnet 571.9%316.0 s133.0160.720.0100.0%
Claude Opus 571.1%569.1 s90.0106.027.0100.0%
Claude Opus 5.568.7%283.0 s32.755.018.7100.0%
GPT-5.6 Sol66.3%135.7 s6.032.719.0100.0%
GPT-5.6 Luna63.9%68.5 s5.021.017.7100.0%
Gemini 3.6 Flash60.2%114.7 s26.725.718.0100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Grok 4.6
Legal Analysis - Core Issues
66.7%
Governing Legal Framework
86.7%
Factual Chronology of the Dispute
88.9%
Parties & Procedural History
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
GPT-6 Astra
Legal Analysis - Core Issues
38.1%
Factual Chronology of the Dispute
85.2%
Parties & Procedural History
85.7%
Governing Legal Framework
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Gemini 3.8 Flash
Legal Analysis - Core Issues
23.8%
Parties & Procedural History
81.0%
Governing Legal Framework
86.7%
Factual Chronology of the Dispute
92.6%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Claude Sonnet 5
Legal Analysis - Core Issues
33.3%
Template & Procedural Compliance
66.7%
Parties & Procedural History
71.4%
Governing Legal Framework
80.0%
Factual Chronology of the Dispute
96.3%
Relief & Remedies
100.0%
Claude Opus 5
Legal Analysis - Core Issues
28.6%
Factual Chronology of the Dispute
77.8%
Governing Legal Framework
86.7%
Parties & Procedural History
90.5%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Claude Opus 5.5
Legal Analysis - Core Issues
23.8%
Governing Legal Framework
80.0%
Parties & Procedural History
81.0%
Factual Chronology of the Dispute
81.5%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
GPT-5.6 Sol
Legal Analysis - Core Issues
42.9%
Factual Chronology of the Dispute
63.0%
Parties & Procedural History
66.7%
Governing Legal Framework
86.7%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
GPT-5.6 Luna
Legal Analysis - Core Issues
23.8%
Parties & Procedural History
71.4%
Governing Legal Framework
73.3%
Factual Chronology of the Dispute
74.1%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Gemini 3.6 Flash
Legal Analysis - Core Issues
14.3%
Factual Chronology of the Dispute
66.7%
Parties & Procedural History
66.7%
Governing Legal Framework
80.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

TASK 007M

Reply strategy

Develop the factual and legal response strategy to the request for arbitration concerning outstanding invoices.

Model rankingMean weighted rubric score
1
Claude Opus 5
3 completed runs · range 53.1%–65.4%
59.3%
2
Claude Opus 5.5
3 completed runs · range 50.6%–54.3%
52.7%
3
GPT-6 Astra
3 completed runs · range 38.3%–50.6%
45.7%
4
GPT-5.6 Sol
3 completed runs · range 39.5%–48.1%
42.8%
5
Claude Sonnet 5
3 completed runs · range 33.3%–45.7%
39.9%
6
Gemini 3.8 Flash
3 completed runs · range 29.6%–39.5%
34.6%
7
GPT-5.6 Luna
3 completed runs · range 29.6%–37.0%
33.7%
8
Grok 4.6
3 completed runs · range 25.9%–33.3%
30.5%
9
Gemini 3.6 Flash
3 completed runs · range 23.5%–35.8%
30.0%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Claude Opus 5 recorded the highest mean score at 59.3%. The difference between the highest and lowest completed model means was 29.2 points. GPT-5.6 Luna had the lowest mean candidate latency (64.5 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Claude Opus 559.3%889.4 s164.0183.339.0100.0%
Claude Opus 5.552.7%322.7 s34.758.717.0100.0%
GPT-6 Astra45.7%272.7 s10.756.322.0100.0%
GPT-5.6 Sol42.8%127.1 s7.342.019.0100.0%
Claude Sonnet 539.9%311.7 s137.3182.720.097.6%
Gemini 3.8 Flash34.6%120.4 s38.737.719.7100.0%
GPT-5.6 Luna33.7%64.5 s6.334.318.7100.0%
Grok 4.630.5%78.1 s10.728.08.793.9%
Gemini 3.6 Flash30.0%146.0 s45.044.019.3100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Claude Opus 5
Governing Legal Framework
16.7%
Document Selection & Sourcing
19.0%
Factual Chronology of the Dispute
66.7%
Legal Analysis - Core Issues
76.9%
Claude Opus 5.5
Governing Legal Framework
0.0%
Document Selection & Sourcing
28.6%
Factual Chronology of the Dispute
58.3%
Legal Analysis - Core Issues
64.1%
GPT-6 Astra
Governing Legal Framework
0.0%
Document Selection & Sourcing
28.6%
Factual Chronology of the Dispute
50.0%
Legal Analysis - Core Issues
59.0%
GPT-5.6 Sol
Governing Legal Framework
0.0%
Factual Chronology of the Dispute
30.6%
Document Selection & Sourcing
33.3%
Legal Analysis - Core Issues
59.0%
Claude Sonnet 5
Governing Legal Framework
16.7%
Document Selection & Sourcing
19.0%
Factual Chronology of the Dispute
44.4%
Legal Analysis - Core Issues
48.7%
Gemini 3.8 Flash
Document Selection & Sourcing
0.0%
Governing Legal Framework
0.0%
Factual Chronology of the Dispute
44.4%
Legal Analysis - Core Issues
46.2%
GPT-5.6 Luna
Governing Legal Framework
0.0%
Document Selection & Sourcing
19.0%
Legal Analysis - Core Issues
38.5%
Factual Chronology of the Dispute
41.7%
Grok 4.6
Governing Legal Framework
0.0%
Document Selection & Sourcing
23.8%
Factual Chronology of the Dispute
33.3%
Legal Analysis - Core Issues
41.0%
Gemini 3.6 Flash
Governing Legal Framework
0.0%
Document Selection & Sourcing
14.3%
Legal Analysis - Core Issues
33.3%
Factual Chronology of the Dispute
41.7%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.

TASK 007L

Reply to the request for arbitration

Prepare a complete reply using the pleadings, contract, correspondence, exhibits, procedural rules and available templates.

Model rankingMean weighted rubric score
1
Claude Opus 5
3 completed runs · range 90.9%–90.9%
90.9%
2
Gemini 3.8 Flash
3 completed runs · range 88.3%–92.2%
89.6%
3
Claude Opus 5.5
3 completed runs · range 87.0%–88.3%
87.4%
4
GPT-5.6 Sol
3 completed runs · range 76.6%–90.9%
81.4%
5
Grok 4.6
3 completed runs · range 70.1%–87.0%
81.4%
6
Claude Sonnet 5
3 completed runs · range 74.0%–79.2%
76.6%
7
Gemini 3.6 Flash
3 completed runs · range 72.7%–80.5%
75.3%
8
GPT-6 Astra
3 completed runs · range 72.7%–75.3%
74.5%
9
GPT-5.6 Luna
3 completed runs · range 63.6%–79.2%
72.7%

Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.

Interpretation. Claude Opus 5 recorded the highest mean score at 90.9%. The difference between the highest and lowest completed model means was 18.2 points. GPT-5.6 Luna had the lowest mean candidate latency (66.4 seconds).

Agent performance

These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.

ModelScoreLatencyTurnsTool callsSources readCitation support
Claude Opus 590.9%943.4 s162.3179.339.7100.0%
Gemini 3.8 Flash89.6%195.0 s45.044.020.0100.0%
Claude Opus 5.587.4%393.5 s34.358.018.7100.0%
GPT-5.6 Sol81.4%137.4 s8.340.719.7100.0%
Grok 4.681.4%90.6 s8.329.715.7100.0%
Claude Sonnet 576.6%279.9 s110.3146.720.096.7%
Gemini 3.6 Flash75.3%139.6 s30.029.020.0100.0%
GPT-6 Astra74.5%333.7 s11.050.722.3100.0%
GPT-5.6 Luna72.7%66.4 s4.722.319.0100.0%

Rubric-category performance

How rubric weighting works. Each high-importance criterion is worth 3 points, each medium-importance criterion 2 points and each low-importance criterion 1 point. A passed criterion earns its full weight and an unmet criterion earns zero. The headline score is the total weight earned divided by the total weight available, averaged across completed runs.

The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.

Claude Opus 5
Governing Legal Framework
71.4%
Parties & Procedural History
75.0%
Factual Chronology of the Dispute
100.0%
Legal Analysis - Core Issues
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
Gemini 3.8 Flash
Governing Legal Framework
66.7%
Relief & Remedies
83.3%
Factual Chronology of the Dispute
100.0%
Legal Analysis - Core Issues
100.0%
Parties & Procedural History
100.0%
Template & Procedural Compliance
100.0%
Claude Opus 5.5
Governing Legal Framework
66.7%
Legal Analysis - Core Issues
83.3%
Parties & Procedural History
83.3%
Factual Chronology of the Dispute
100.0%
Relief & Remedies
100.0%
Template & Procedural Compliance
100.0%
GPT-5.6 Sol
Legal Analysis - Core Issues
66.7%
Relief & Remedies
66.7%
Governing Legal Framework
71.4%
Parties & Procedural History
75.0%
Factual Chronology of the Dispute
92.6%
Template & Procedural Compliance
100.0%
Grok 4.6
Governing Legal Framework
61.9%
Legal Analysis - Core Issues
75.0%
Relief & Remedies
83.3%
Template & Procedural Compliance
83.3%
Parties & Procedural History
91.7%
Factual Chronology of the Dispute
92.6%
Claude Sonnet 5
Relief & Remedies
33.3%
Governing Legal Framework
57.1%
Parties & Procedural History
58.3%
Legal Analysis - Core Issues
75.0%
Template & Procedural Compliance
91.7%
Factual Chronology of the Dispute
100.0%
Gemini 3.6 Flash
Relief & Remedies
50.0%
Governing Legal Framework
52.4%
Legal Analysis - Core Issues
58.3%
Factual Chronology of the Dispute
96.3%
Parties & Procedural History
100.0%
Template & Procedural Compliance
100.0%
GPT-6 Astra
Relief & Remedies
33.3%
Legal Analysis - Core Issues
50.0%
Governing Legal Framework
57.1%
Parties & Procedural History
83.3%
Factual Chronology of the Dispute
100.0%
Template & Procedural Compliance
100.0%
GPT-5.6 Luna
Legal Analysis - Core Issues
50.0%
Relief & Remedies
50.0%
Governing Legal Framework
61.9%
Parties & Procedural History
66.7%
Factual Chronology of the Dispute
88.9%
Template & Procedural Compliance
91.7%
Generated work products

Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.