Artificial Analysis and Harvey updated their legal-agent benchmark on October 8 to check submitted work against its source documents. The new version only awards its headline credit when an answer clears the rubric and avoids material hallucinations. For model developers, the important change is what the score now refuses to overlook.
One answer can have two different weaknesses
Imagine an assistant asked to summarize several documents and list their unresolved issues. It might faithfully repeat the evidence but omit one requested section. Another answer might cover every section while inventing a date to make the narrative feel complete. Both are flawed, but they require different diagnoses. Adding more detailed formatting instructions is unlikely to repair invented evidence.
The model-development implication is to keep coverage and grounding visible separately. A combined score is useful for acceptance, but separate error categories help decide whether to improve data, retrieval, instruction following or post-training. Otherwise a team can spend its next training run addressing a weakness that was not responsible for the failed outputs.
The new headline is a stricter instrument
The evaluation uses 120 private tasks and a three-model rubric panel. Its headline averages the share of judges that pass every criterion, setting a task to zero when a material hallucination is found. It is therefore not simply a human-certified percentage of completed jobs. The published methodology supplies the operational definition.
That distinction matters when comparing releases. A stricter scoring rule can lower a reported number without any change in the underlying model. Results from different benchmark versions should not be treated as a time series of capability unless the tasks, settings and grading are made comparable. A new instrument can reveal a previously hidden weakness.
Do not teach to the visible answer format
A tempting response to evaluation failures is to add more examples resembling the benchmark. That may improve performance, but it can also teach superficial patterns. The stronger test is whether the model preserves grounding on unfamiliar document layouts, different terminology and questions that cannot be answered fully from the supplied evidence.
An illustrative training contrast would pair a polished but unsupported answer with a less complete answer that accurately identifies a missing fact. The reward should reflect the task’s actual requirements. In some settings, asking for the missing information is the useful behavior. In others, a partial answer with explicit limits is appropriate. There is no universal preference for verbosity or refusal.
The judge also needs a test
Automated evaluation creates another learned component between an output and its score. A checker can miss an invention, reject a supported claim or confuse an immaterial wording difference with a substantive error. Agreement among models is useful evidence, but it is not a guarantee that they have independent failure modes.
A model team could audit a stratified sample of accepted and rejected outputs with domain experts, keeping the model identity hidden. Include difficult supported statements as well as obvious fabrications. Record disagreements and their causes. This proposed audit would measure the reliability of the evaluation instrument rather than assuming it from the reputation of the models used as judges.
A joint test changes what progress looks like
Our assessment is that grounded completeness should be treated as a capability in its own right. A development team needs evidence that a model can both find the necessary material and resist inventing the material it cannot find. Gains on only one axis may leave the accepted-output rate disappointingly low.
The business consequence follows from that technical distinction. An answer that is easy to check can reduce review effort; one that mixes accurate detail with subtle invention can make review harder. Measuring the time needed to verify outputs would add useful evidence, but this benchmark alone does not establish customer productivity gains or performance across every professional domain.
The evidence to watch
Look for paired results on fresh tasks, expert audits of automated judgments, and separate trends for omissions and unsupported claims. Improvement should survive a new set of source documents, not merely a familiar rubric or answer template.
References
Questions, news tips or corrections? Contact [email protected].
