Claude Opus 5 had the highest equal-weight twelve-task mean at 80.1%. The next-highest model was Claude Opus 5.5 at 77.7%.
Agent evaluation · International arbitration
Methodology and resultsThis report presents an agentic legal evaluation of international-arbitration work across twelve tasks. The harness uses structured task packages and expert rubrics to assess both answer quality and agentic execution.
Weighted results across all tasks
The leaderboard ranks models by their equal-weight mean across the twelve task-level weighted rubric scores. It is not an all-or-nothing task pass rate: each score reflects the rubric weight earned, with high-, medium- and low-importance criteria worth 3, 2 and 1 points respectively.
Overall scores give each task equal weight.
Green highlights the leading model. On individual tasks, capped lines show the lowest and highest observed scores across runs; they are not confidence intervals.
Detailed task results
The weighted mean is shown first for emphasis. Each task cell gives the mean weighted rubric score and completed-run coverage. Starred figures have incomplete task coverage and are not directly comparable with complete twelve-task means.
| Model | Weighted mean | 001M | 001L | 002S | 002M | 002L | 003S | 005S | 006M | 006L | 007S | 007M | 007L |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Opus 5 | 80.1%12/12 tasks · 36 runs | 83.5%3/3 runs | 81.7%3/3 runs | 88.2%3/3 runs | 66.7%3/3 runs | 81.5%3/3 runs | 88.9%3/3 runs | 98.2%3/3 runs | 75.8%3/3 runs | 75.7%3/3 runs | 71.1%3/3 runs | 59.3%3/3 runs | 90.9%3/3 runs |
| Claude Opus 5.5 | 77.7%12/12 tasks · 36 runs | 78.5%3/3 runs | 88.9%3/3 runs | 77.2%3/3 runs | 78.8%3/3 runs | 70.4%3/3 runs | 80.2%3/3 runs | 91.9%3/3 runs | 76.8%3/3 runs | 80.5%3/3 runs | 68.7%3/3 runs | 52.7%3/3 runs | 87.4%3/3 runs |
| Gemini 3.8 Flash | 73.3%12/12 tasks · 36 runs | 79.7%3/3 runs | 88.1%3/3 runs | 91.1%3/3 runs | 67.7%3/3 runs | 80.9%3/3 runs | 81.0%3/3 runs | 68.5%3/3 runs | 68.7%3/3 runs | 56.9%3/3 runs | 72.7%3/3 runs | 34.6%3/3 runs | 89.6%3/3 runs |
| GPT-6 Astra | 72.5%12/12 tasks · 36 runs | 75.9%3/3 runs | 72.6%3/3 runs | 76.4%3/3 runs | 51.9%3/3 runs | 72.8%3/3 runs | 84.1%3/3 runs | 81.1%3/3 runs | 76.8%3/3 runs | 81.6%3/3 runs | 77.1%3/3 runs | 45.7%3/3 runs | 74.5%3/3 runs |
| GPT-5.6 Sol | 69.4%12/12 tasks · 36 runs | 68.6%3/3 runs | 76.2%3/3 runs | 74.3%3/3 runs | 37.6%3/3 runs | 73.5%3/3 runs | 84.9%3/3 runs | 82.9%3/3 runs | 70.2%3/3 runs | 74.5%3/3 runs | 66.3%3/3 runs | 42.8%3/3 runs | 81.4%3/3 runs |
| Claude Sonnet 5 | 68.2%12/12 tasks · 36 runs | 76.2%3/3 runs | 71.0%3/3 runs | 83.1%3/3 runs | 46.6%3/3 runs | 59.9%3/3 runs | 84.1%3/3 runs | 79.3%3/3 runs | 70.7%3/3 runs | 59.2%3/3 runs | 71.9%3/3 runs | 39.9%3/3 runs | 76.6%3/3 runs |
| Grok 4.6 | 67.2%12/12 tasks · 36 runs | 67.0%3/3 runs | 57.1%3/3 runs | 81.0%3/3 runs | 47.1%3/3 runs | 67.3%3/3 runs | 91.3%3/3 runs | 74.8%3/3 runs | 63.1%3/3 runs | 59.6%3/3 runs | 85.9%3/3 runs | 30.5%3/3 runs | 81.4%3/3 runs |
| Gemini 3.6 Flash | 65.5%12/12 tasks · 36 runs | 71.6%3/3 runs | 77.4%3/3 runs | 72.2%3/3 runs | 48.7%3/3 runs | 86.4%3/3 runs | 81.0%3/3 runs | 65.8%3/3 runs | 61.1%3/3 runs | 55.8%3/3 runs | 60.2%3/3 runs | 30.0%3/3 runs | 75.3%3/3 runs |
| GPT-5.6 Luna | 61.5%12/12 tasks · 36 runs | 62.5%3/3 runs | 65.1%3/3 runs | 55.7%3/3 runs | 27.0%3/3 runs | 66.0%3/3 runs | 80.2%3/3 runs | 77.5%3/3 runs | 66.2%3/3 runs | 67.4%3/3 runs | 63.9%3/3 runs | 33.7%3/3 runs | 72.7%3/3 runs |
Percentage of available rubric weight earned in a run, averaged across completed and fully judged runs. High-importance criteria carry 3 points, medium-importance criteria 2 points and low-importance criteria 1 point. A passed criterion earns its full weight; an unmet criterion earns zero.
Per-task ranges show observed differences across runs. With at most three repetitions, they indicate consistency but are not statistical confidence intervals.
Three runs were requested per model and task. Blank cells mean no completed work product was available to score.
What the twelve tasks indicate
Claude Opus 5 achieved the strongest quality but used 93.4 mean model turns and 111.7 mean tool calls. Among the remaining models averaging no more than 20 turns per task, GPT-6 Astra provided the strongest quality-efficiency balance: 72.5% with 11.4 turns and 50.1 tool calls.
Scores depended strongly on the task. Across the fully covered models, 003S had the highest cross-model mean (84.0%), while 007M was the most difficult (41.0%). Model ranking should therefore be read task by task, not as one universal capability score.
Twelve-task comparison
This table puts overall quality beside the amount of agent activity required to produce it. The score is the equal-weight mean of the twelve task-level weighted scores; turns and tool calls are averaged across those same tasks.
| Model | Twelve-task mean | Mean turns | Mean tool calls |
|---|---|---|---|
| Claude Opus 5 | 80.1% | 93.4 | 111.7 |
| Claude Opus 5.5 | 77.7% | 22.9 | 42.8 |
| Gemini 3.8 Flash | 73.3% | 32.7 | 31.7 |
| GPT-6 Astra | 72.5% | 11.4 | 50.1 |
| GPT-5.6 Sol | 69.4% | 6.2 | 30.8 |
| Claude Sonnet 5 | 68.2% | 93.2 | 117.9 |
| Grok 4.6 | 67.2% | 7.8 | 18.0 |
| Gemini 3.6 Flash | 65.5% | 25.7 | 24.7 |
| GPT-5.6 Luna | 61.5% | 5.0 | 22.5 |
- This is an independently implemented agentic legal evaluation. Results are specific to this harness, task package and judge configuration.
- The default judge was Claude Sonnet 5. Its assessment of Sonnet candidate outputs creates a potential same-family evaluation effect.
- Three requested runs per task provide an initial consistency signal, not a statistically robust estimate. Reported scores use completed, fully judged runs only.
- Candidate models used the same task materials, agent tools and output contract, but provider-specific reasoning and runtime settings were not identical.
- Rubric categories summarize criterion outcomes; they do not replace expert review of the generated legal work products.
How the agent harness and evaluation worked
The tasks were derived from real-world international arbitration matters and organized into controlled matter rooms. Client-facing case descriptions and downloadable work products in this report are anonymized.
Under this agentic legal evaluation structure, each task combined a client-style instruction, a controlled matter room and an expert rubric. The twelve tasks covered two deliverables in Case 001, three in Case 002, one each in Cases 003 and 005, and two in Case 006, and three in Case 007. Where a reference answer was available, it supported task and rubric design but was not shown to the candidate agent.
The labels describe the scope of the work rather than the quality expected. Small tasks are focused assignments with a compact deliverable; medium tasks require broader analysis or evidence review across more materials; long tasks require the most extensive source synthesis and a fuller legal work product. The labels are relative within this evaluation and do not impose one universal page or token limit.
For each run, the harness created an isolated workspace and gave the candidate model the task, matter files and a defined DOCX output contract. Six active tools supported source inventory, bounded source reading, matter-room search, spreadsheet inspection, calculation and workspace notes. A seventh registered tool for external legal research was disabled for these evaluations. The reference answer and rubric were withheld from the candidate.
Case 007 used locally validated OCR for scanned pages. The 007S Opus 5.5 and 007L Grok follow-up cohorts added two precise line/page reading tools (eight active tools, nine registered), task-derived evidence-coverage guidance and a final-answer completion check. Other displayed groups retain their recorded harness version; these execution differences should be considered when comparing results. The default judge and expert rubrics were unchanged.
This report includes three completed independent runs for every candidate model and task. Each run started from the same instructions and matter materials without access to another run's work. The harness retained the submitted DOCX and recorded operational measures including elapsed time, model turns, tool calls, sources read and citation support.
After a candidate submitted its final document, the harness passed that work product—together with the task, expert rubric and reference evidence—to the default judge, Claude Sonnet 5. The judge evaluated every rubric criterion individually, recording whether it was satisfied and grounding the decision in the candidate response and reference materials. For 002M, compound criteria were checked component by component, with automated source-quote validation and an AI-assisted consistency review; all required components had to be satisfied. Equivalent wording counted unless the rubric expressly required an exact phrase. The judge assessed the completed output only; it did not participate in or alter the candidate run.
Deterministic checks confirmed that the required artifact existed and opened successfully. A run entered the results only after both the candidate artifact and criterion-level judgment were complete. The headline percentage is the rubric weight earned divided by the total available weight: high-importance criteria are worth 3 points, medium-importance criteria 2 points and low-importance criteria 1 point. Task results average that weighted score across completed runs; category results group the underlying criterion decisions by rubric theme.
Evaluation releases
The current Version 1.2 contains twelve tasks, nine candidate models and three completed, fully judged runs per model and task (324 displayed runs). Earlier attempts and superseded cohorts remain retained internally.
- Extended Cases 001–003 with 005S, 006M/006L and the updated 007S/007M/007L task packages.
- Added Claude Opus 5.5 and Gemini 3.8 Flash, bringing coverage to nine candidate models.
- Refreshed 001M/001L and 002S/002M/002L using the September 30 task packages; added a full evaluation of 006M.
- Reviewed 002M rubric decisions component by component and validated supporting source quotes. Its approved latest three-run cohorts remain unchanged.
- Added Case 007 (007S, 007M and 007L), evaluated across all nine models with three runs per model/task.
- Recalculated overall results across all twelve tasks with equal task weighting; task scores retain the rubric’s 3/2/1 importance weights.
- Expanded the preliminary release to Cases 001–003.
- Added GPT-6 Astra and Gemini 3.6 Flash.
- Included controlled-harness reruns following task and source-configuration corrections.
- Open archived v1.1 report →
- Initial preliminary single-task evaluation, retained as the original published snapshot.
- Its preliminary results were superseded by subsequent validated evaluations.
- Open archived v1.0 report →
Contract dispute strategy
A multilingual telecommunications dispute requiring a respondent pleading and a client strategy memorandum. Matter languages: English and French.
Respondent pleading
Draft a structured respondent pleading addressing the contractual history, termination, waiver, quantum, counterclaims, procedure and relief sought.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 83.5% | 786.8 s | 79.7 | 102.7 | 48.7 | 100.0% |
| Gemini 3.8 Flash | 79.7% | 168.4 s | 38.3 | 37.3 | 29.7 | 100.0% |
| Claude Opus 5.5 | 78.5% | 401.9 s | 18.0 | 47.7 | 33.3 | 100.0% |
| Claude Sonnet 5 | 76.2% | 274.2 s | 51.7 | 79.0 | 47.0 | 100.0% |
| GPT-6 Astra | 75.9% | 481.7 s | 13.0 | 60.0 | 39.3 | 100.0% |
| Gemini 3.6 Flash | 71.6% | 128.5 s | 26.0 | 25.0 | 16.0 | 100.0% |
| GPT-5.6 Sol | 68.6% | 150.6 s | 6.7 | 30.0 | 22.3 | 100.0% |
| Grok 4.6 | 67.0% | 85.9 s | 6.3 | 14.3 | 7.3 | 100.0% |
| GPT-5.6 Luna | 62.5% | 69.8 s | 4.3 | 17.7 | 15.0 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Claude Opus 5
Gemini 3.8 Flash
Claude Opus 5.5
Claude Sonnet 5
GPT-6 Astra
Gemini 3.6 Flash
GPT-5.6 Sol
Grok 4.6
GPT-5.6 Luna
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Client memorandum
Prepare a confidential and privileged client memorandum on the contractual dispute, legal and financial risks, and recommended strategy, using verified facts and contractual wording.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 88.9% | 243.0 s | 15.0 | 31.7 | 20.3 | 98.8% |
| Gemini 3.8 Flash | 88.1% | 216.4 s | 32.7 | 31.7 | 19.0 | 97.5% |
| Claude Opus 5 | 81.7% | 498.9 s | 80.3 | 102.0 | 52.0 | 100.0% |
| Gemini 3.6 Flash | 77.4% | 157.9 s | 35.3 | 34.3 | 12.3 | 94.3% |
| GPT-5.6 Sol | 76.2% | 146.9 s | 7.7 | 32.3 | 14.7 | 98.6% |
| GPT-6 Astra | 72.6% | 358.8 s | 11.3 | 50.7 | 29.3 | 95.3% |
| Claude Sonnet 5 | 71.0% | 183.7 s | 40.7 | 59.0 | 33.3 | 98.4% |
| GPT-5.6 Luna | 65.1% | 52.3 s | 6.3 | 25.7 | 11.3 | 98.5% |
| Grok 4.6 | 57.1% | 84.2 s | 11.3 | 26.0 | 10.3 | 94.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Claude Opus 5.5
Gemini 3.8 Flash
Claude Opus 5
Gemini 3.6 Flash
GPT-5.6 Sol
GPT-6 Astra
Claude Sonnet 5
GPT-5.6 Luna
Grok 4.6
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
CAIP arbitration defence
A cross-border sale-of-goods arbitration involving force majeure, exhibit assessment and a substantive reply. Matter languages: English, French, Spanish and Chinese.
Force majeure memorandum
Prepare a five-to-six-page English memorandum on contractual force majeure under French law for a respondent in a CAIP arbitration.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Gemini 3.8 Flash | 91.1% | 175.9 s | 34.3 | 33.3 | 9.0 | 100.0% |
| Claude Opus 5 | 88.2% | 920.4 s | 164.7 | 183.3 | 69.0 | 100.0% |
| Claude Sonnet 5 | 83.1% | 478.8 s | 108.7 | 148.0 | 59.0 | 100.0% |
| Grok 4.6 | 81.0% | 168.9 s | 7.3 | 15.0 | 4.0 | 100.0% |
| Claude Opus 5.5 | 77.2% | 308.4 s | 24.3 | 43.7 | 24.0 | 100.0% |
| GPT-6 Astra | 76.4% | 351.6 s | 13.0 | 56.7 | 35.7 | 100.0% |
| GPT-5.6 Sol | 74.3% | 131.6 s | 6.3 | 40.0 | 16.7 | 100.0% |
| Gemini 3.6 Flash | 72.2% | 133.4 s | 20.7 | 19.7 | 11.0 | 93.3% |
| GPT-5.6 Luna | 55.7% | 102.1 s | 5.0 | 28.7 | 22.3 | 89.6% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Gemini 3.8 Flash
Claude Opus 5
Claude Sonnet 5
Grok 4.6
Claude Opus 5.5
GPT-6 Astra
GPT-5.6 Sol
Gemini 3.6 Flash
GPT-5.6 Luna
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Exhibit review
Assess the evidential weight of each exhibit MA001–MA017 submitted with the claimant’s case and present the assessment as a usable table.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 78.8% | 457.3 s | 26.7 | 40.3 | 17.3 | 100.0% |
| Gemini 3.8 Flash | 67.7% | 181.9 s | 26.7 | 25.7 | 20.7 | 100.0% |
| Claude Opus 5 | 66.7% | 658.2 s | 79.3 | 108.3 | 57.0 | 100.0% |
| GPT-6 Astra | 51.9% | 423.1 s | 13.0 | 60.7 | 37.3 | 100.0% |
| Gemini 3.6 Flash | 48.7% | 133.7 s | 15.0 | 14.0 | 12.0 | 100.0% |
| Grok 4.6 | 47.1% | 63.1 s | 6.0 | 10.3 | 5.7 | 97.2% |
| Claude Sonnet 5 | 46.6% | 298.2 s | 86.0 | 105.3 | 14.3 | 97.0% |
| GPT-5.6 Sol | 37.6% | 107.0 s | 4.7 | 16.3 | 9.7 | 100.0% |
| GPT-5.6 Luna | 27.0% | 74.1 s | 4.0 | 9.7 | 7.3 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Claude Opus 5.5
Gemini 3.8 Flash
Claude Opus 5
GPT-6 Astra
Gemini 3.6 Flash
Grok 4.6
Claude Sonnet 5
GPT-5.6 Sol
GPT-5.6 Luna
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Arbitration reply
Draft an approximately 20-page English reply using the supplied legal authorities RL-1–RL-23 and client factual materials R-1–R-24.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Gemini 3.6 Flash | 86.4% | 249.9 s | 44.7 | 43.7 | 41.7 | 100.0% |
| Claude Opus 5 | 81.5% | 1,142.9 s | 120.7 | 160.7 | 79.0 | 100.0% |
| Gemini 3.8 Flash | 80.9% | 250.4 s | 57.7 | 56.7 | 51.0 | 100.0% |
| GPT-5.6 Sol | 73.5% | 241.9 s | 8.3 | 63.0 | 57.7 | 100.0% |
| GPT-6 Astra | 72.8% | 658.9 s | 18.0 | 112.7 | 61.3 | 100.0% |
| Claude Opus 5.5 | 70.4% | 674.1 s | 41.0 | 88.0 | 55.3 | 100.0% |
| Grok 4.6 | 67.3% | 107.0 s | 5.3 | 12.7 | 9.0 | 97.4% |
| GPT-5.6 Luna | 66.0% | 271.1 s | 5.3 | 47.7 | 46.0 | 100.0% |
| Claude Sonnet 5 | 59.9% | 298.1 s | 31.7 | 77.7 | 60.0 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Gemini 3.6 Flash
Claude Opus 5
Gemini 3.8 Flash
GPT-5.6 Sol
GPT-6 Astra
Claude Opus 5.5
Grok 4.6
GPT-5.6 Luna
Claude Sonnet 5
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Arbitration clause drafting
A clause-drafting exercise for an international telecommunications agreement, applying client instructions to governing-law and dispute-resolution provisions. Matter languages: English and French.
- Client instructions
- Pseudonymized roaming framework agreement
- ICC Arbitration Rules 2026 / Mediation Rules 2014
- SCC Arbitration Rules 2023
Clause drafting
Revise governing-law and arbitration clauses in a roaming framework agreement, including French law, Paris arbitration, SCC procedure, a sole arbitrator and emergency relief.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Grok 4.6 | 91.3% | 57.6 s | 11.3 | 17.3 | 3.0 | 100.0% |
| Claude Opus 5 | 88.9% | 406.2 s | 44.3 | 48.7 | 13.0 | 100.0% |
| GPT-5.6 Sol | 84.9% | 72.5 s | 5.0 | 9.7 | 3.3 | 100.0% |
| Claude Sonnet 5 | 84.1% | 499.2 s | 176.7 | 182.0 | 3.0 | 100.0% |
| GPT-6 Astra | 84.1% | 259.7 s | 7.7 | 16.0 | 5.3 | 100.0% |
| Gemini 3.6 Flash | 81.0% | 49.5 s | 12.0 | 11.0 | 3.0 | 100.0% |
| Gemini 3.8 Flash | 81.0% | 105.3 s | 21.0 | 20.0 | 3.7 | 100.0% |
| GPT-5.6 Luna | 80.2% | 44.2 s | 6.3 | 12.0 | 4.3 | 100.0% |
| Claude Opus 5.5 | 80.2% | 224.5 s | 12.3 | 15.3 | 4.0 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Grok 4.6
Claude Opus 5
GPT-5.6 Sol
Claude Sonnet 5
GPT-6 Astra
Gemini 3.6 Flash
Gemini 3.8 Flash
GPT-5.6 Luna
Claude Opus 5.5
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Arbitrator fee determination
An Italian ad hoc arbitration fee dispute requiring analysis of non-acceptance, Article 814(2), the available fee evidence and the documented procedural posture. Matter language: Italian.
Arbitrator fee memorandum
Assess whether an arbitrator's self-liquidated fee binds the parties under Article 814(2) of the Italian Code of Civil Procedure, apply the available fee-scale evidence and describe the documented procedural status.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 98.2% | 368.3 s | 58.7 | 72.7 | 13.0 | 100.0% |
| Claude Opus 5.5 | 91.9% | 206.4 s | 15.7 | 29.3 | 10.3 | 100.0% |
| GPT-5.6 Sol | 82.9% | 109.7 s | 5.3 | 28.0 | 11.0 | 100.0% |
| GPT-6 Astra | 81.1% | 244.4 s | 11.0 | 44.0 | 13.3 | 100.0% |
| Claude Sonnet 5 | 79.3% | 420.6 s | 173.0 | 183.0 | 10.7 | 100.0% |
| GPT-5.6 Luna | 77.5% | 53.8 s | 5.3 | 18.0 | 11.3 | 100.0% |
| Grok 4.6 | 74.8% | 77.7 s | 8.7 | 25.7 | 8.0 | 100.0% |
| Gemini 3.8 Flash | 68.5% | 132.0 s | 27.0 | 26.0 | 11.3 | 100.0% |
| Gemini 3.6 Flash | 65.8% | 101.6 s | 15.3 | 14.3 | 11.0 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Claude Opus 5
Claude Opus 5.5
GPT-5.6 Sol
GPT-6 Astra
Claude Sonnet 5
GPT-5.6 Luna
Grok 4.6
Gemini 3.8 Flash
Gemini 3.6 Flash
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Jurisdiction and merits exposure
An Italian distribution dispute requiring analysis of the tribunal's jurisdiction ruling and a separate opinion on unresolved merits exposure. Matter languages: Italian and English.
Jurisdiction ruling opinion
Assess what the tribunal's jurisdiction ruling establishes and leaves undecided, and evaluate its grounds using the supplied Italian-language record.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Claude Opus 5.5 | 76.8% | 193.3 s | 10.0 | 21.7 | 16.7 | 100.0% |
| GPT-6 Astra | 76.8% | 357.0 s | 9.3 | 22.0 | 17.0 | 100.0% |
| Claude Opus 5 | 75.8% | 371.4 s | 44.0 | 53.7 | 20.7 | 100.0% |
| Claude Sonnet 5 | 70.7% | 181.7 s | 64.0 | 75.7 | 15.0 | 100.0% |
| GPT-5.6 Sol | 70.2% | 180.7 s | 4.3 | 18.3 | 15.0 | 100.0% |
| Gemini 3.8 Flash | 68.7% | 126.6 s | 21.0 | 20.0 | 15.0 | 100.0% |
| GPT-5.6 Luna | 66.2% | 57.7 s | 3.0 | 16.0 | 15.0 | 100.0% |
| Grok 4.6 | 63.1% | 53.7 s | 5.0 | 11.7 | 10.7 | 100.0% |
| Gemini 3.6 Flash | 61.1% | 101.1 s | 19.3 | 18.3 | 15.3 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Claude Opus 5.5
GPT-6 Astra
Claude Opus 5
Claude Sonnet 5
GPT-5.6 Sol
Gemini 3.8 Flash
GPT-5.6 Luna
Grok 4.6
Gemini 3.6 Flash
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Merits exposure opinion
Assess the respondent's exposure to contractual pecuniary-loss and reputational-damage claims after an arbitral tribunal declined jurisdiction, using only the supplied Italian-language record and without predicting a future merits outcome.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| GPT-6 Astra | 81.6% | 196.3 s | 7.7 | 23.3 | 17.0 | 100.0% |
| Claude Opus 5.5 | 80.5% | 227.2 s | 10.7 | 24.3 | 15.7 | 100.0% |
| Claude Opus 5 | 75.7% | 229.4 s | 32.3 | 40.0 | 18.0 | 100.0% |
| GPT-5.6 Sol | 74.5% | 97.9 s | 4.3 | 16.0 | 14.0 | 100.0% |
| GPT-5.6 Luna | 67.4% | 38.3 s | 4.0 | 17.0 | 15.0 | 100.0% |
| Grok 4.6 | 59.6% | 56.7 s | 5.7 | 10.7 | 7.7 | 100.0% |
| Claude Sonnet 5 | 59.2% | 58.2 s | 5.3 | 15.0 | 13.7 | 97.6% |
| Gemini 3.8 Flash | 56.9% | 124.4 s | 18.7 | 17.7 | 15.0 | 97.6% |
| Gemini 3.6 Flash | 55.8% | 103.8 s | 18.0 | 17.0 | 14.3 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
GPT-6 Astra
Claude Opus 5.5
Claude Opus 5
GPT-5.6 Sol
GPT-5.6 Luna
Grok 4.6
Claude Sonnet 5
Gemini 3.8 Flash
Gemini 3.6 Flash
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Algerian-seated arbitration response
A dispute over outstanding invoices requiring a matter summary, reply strategy and complete response to the request for arbitration. Matter languages: English and French.
Matter summary and information request
Summarize the dispute and identify the client documents and information needed to prepare the response.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Grok 4.6 | 85.9% | 66.7 s | 7.7 | 14.3 | 5.3 | 100.0% |
| GPT-6 Astra | 77.1% | 280.2 s | 11.7 | 48.3 | 22.3 | 100.0% |
| Gemini 3.8 Flash | 72.7% | 119.4 s | 31.7 | 30.7 | 20.0 | 100.0% |
| Claude Sonnet 5 | 71.9% | 316.0 s | 133.0 | 160.7 | 20.0 | 100.0% |
| Claude Opus 5 | 71.1% | 569.1 s | 90.0 | 106.0 | 27.0 | 100.0% |
| Claude Opus 5.5 | 68.7% | 283.0 s | 32.7 | 55.0 | 18.7 | 100.0% |
| GPT-5.6 Sol | 66.3% | 135.7 s | 6.0 | 32.7 | 19.0 | 100.0% |
| GPT-5.6 Luna | 63.9% | 68.5 s | 5.0 | 21.0 | 17.7 | 100.0% |
| Gemini 3.6 Flash | 60.2% | 114.7 s | 26.7 | 25.7 | 18.0 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Grok 4.6
GPT-6 Astra
Gemini 3.8 Flash
Claude Sonnet 5
Claude Opus 5
Claude Opus 5.5
GPT-5.6 Sol
GPT-5.6 Luna
Gemini 3.6 Flash
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Reply strategy
Develop the factual and legal response strategy to the request for arbitration concerning outstanding invoices.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 59.3% | 889.4 s | 164.0 | 183.3 | 39.0 | 100.0% |
| Claude Opus 5.5 | 52.7% | 322.7 s | 34.7 | 58.7 | 17.0 | 100.0% |
| GPT-6 Astra | 45.7% | 272.7 s | 10.7 | 56.3 | 22.0 | 100.0% |
| GPT-5.6 Sol | 42.8% | 127.1 s | 7.3 | 42.0 | 19.0 | 100.0% |
| Claude Sonnet 5 | 39.9% | 311.7 s | 137.3 | 182.7 | 20.0 | 97.6% |
| Gemini 3.8 Flash | 34.6% | 120.4 s | 38.7 | 37.7 | 19.7 | 100.0% |
| GPT-5.6 Luna | 33.7% | 64.5 s | 6.3 | 34.3 | 18.7 | 100.0% |
| Grok 4.6 | 30.5% | 78.1 s | 10.7 | 28.0 | 8.7 | 93.9% |
| Gemini 3.6 Flash | 30.0% | 146.0 s | 45.0 | 44.0 | 19.3 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Claude Opus 5
Claude Opus 5.5
GPT-6 Astra
GPT-5.6 Sol
Claude Sonnet 5
Gemini 3.8 Flash
GPT-5.6 Luna
Grok 4.6
Gemini 3.6 Flash
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.
Reply to the request for arbitration
Prepare a complete reply using the pleadings, contract, correspondence, exhibits, procedural rules and available templates.
Bars show mean scores. Capped lines show the observed minimum–maximum across runs, not confidence intervals.
Agent performance
These operational measures explain how the agent reached its answer. Latency is elapsed candidate-run time; turns count model-response cycles; tool calls count actions taken in the matter-room workspace; sources read counts distinct source documents accessed; and citation support measures how often the answer's cited propositions were supported by the cited material. For 002M, citation-support figures retain the original citation assessment of the unchanged answers; only rubric decisions were regraded.
| Model | Score | Latency | Turns | Tool calls | Sources read | Citation support |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 90.9% | 943.4 s | 162.3 | 179.3 | 39.7 | 100.0% |
| Gemini 3.8 Flash | 89.6% | 195.0 s | 45.0 | 44.0 | 20.0 | 100.0% |
| Claude Opus 5.5 | 87.4% | 393.5 s | 34.3 | 58.0 | 18.7 | 100.0% |
| GPT-5.6 Sol | 81.4% | 137.4 s | 8.3 | 40.7 | 19.7 | 100.0% |
| Grok 4.6 | 81.4% | 90.6 s | 8.3 | 29.7 | 15.7 | 100.0% |
| Claude Sonnet 5 | 76.6% | 279.9 s | 110.3 | 146.7 | 20.0 | 96.7% |
| Gemini 3.6 Flash | 75.3% | 139.6 s | 30.0 | 29.0 | 20.0 | 100.0% |
| GPT-6 Astra | 74.5% | 333.7 s | 11.0 | 50.7 | 22.3 | 100.0% |
| GPT-5.6 Luna | 72.7% | 66.4 s | 4.7 | 22.3 | 19.0 | 100.0% |
Rubric-category performance
The category bars below show the criterion pass rate within each rubric category; they are diagnostic breakdowns and are not separately reweighted. Models appear in the same weighted-score order shown above. Categories are ordered from lowest to highest pass rate within each model.
Claude Opus 5
Gemini 3.8 Flash
Claude Opus 5.5
GPT-5.6 Sol
Grok 4.6
Claude Sonnet 5
Gemini 3.6 Flash
GPT-6 Astra
GPT-5.6 Luna
Generated work products
Download the evaluated DOCX submitted by each completed run. Evaluation traces and internal JSON files are intentionally omitted.