AGENT EVALUATION · INTERNATIONAL ARBITRATION

Under this agentic legal evaluation structure, each task combined a client-style instruction, a controlled matter room, a reference answer and an expert rubric. The current release covers deliverables across Cases 001–003; later versions can extend the task set without changing the report structure.

The labels describe the scope of the work rather than the quality expected. Small tasks are focused assignments with a compact deliverable; medium tasks require broader analysis or evidence review across more materials; long tasks require the most extensive source synthesis and a fuller legal work product. The labels are relative within this evaluation and do not impose one universal page or token limit.

For each run, the harness created an isolated workspace and gave the candidate model the task, matter files and a defined DOCX output contract. Six active tools supported source inventory, bounded source reading, matter-room search, spreadsheet inspection, calculation and workspace notes. A seventh registered tool for external legal research was disabled for these evaluations. The reference answer and rubric were withheld from the candidate.

Three separate runs were requested for every candidate model and task. Each run started from the same instructions and matter materials without access to another run's work. The harness retained the submitted DOCX and recorded operational measures including elapsed time, model turns, tool calls, sources read and citation support.

After a candidate submitted its final document, the harness passed that work product—together with the task, expert rubric and reference evidence—to the default judge, Claude Sonnet 5. The judge evaluated every rubric criterion individually, recording whether it was satisfied and grounding the decision in the candidate response and reference materials. The judge assessed the completed output only; it did not participate in or alter the candidate run.

Deterministic checks confirmed that the required artifact existed and opened successfully. A run entered the results only after both the candidate artifact and criterion-level judgment were complete. The headline percentage is the rubric weight earned divided by the total available weight: high-importance criteria are worth 3 points, medium-importance criteria 2 points and low-importance criteria 1 point. Task results average that weighted score across completed runs; category results group the underlying criterion decisions by rubric theme.