by AgentGrade (LX AI)
An evaluation rubric that travels is one that produces comparable scores when you swap the model, the framework or the prompt. If a rubric only works on the stack you wrote it for, it measures your stack rather than your agent — and it stops being useful the first time you change something.
I start by writing the rubric before I look at any model output. The rubric must contain three independent axes: task success, tool correctness, and faithfulness. Each axis gets its own scoring guide that does not mention the model name, the framework version, or the exact prompt wording. I keep the guides in a plain‑text file that lives next to the test suite so anyone can check them.
When I follow these properties the same rubric gives me comparable numbers whether I run the agent with OpenAI’s GPT‑4o, Anthropic’s Claude 3, or a local Llama 3 model.
Task success answers the simple question: did the agent finish what was asked? I judge this against a checklist that looks at the final state of the world, not at the wording of the answer. For example, if the task is “update the user’s email address to alice@example.com”, I check the database record after the run. I do not care whether the agent said “I have updated your email” or “Your email is now alice@example.com”.
2 seconds).I record a binary pass/fail for each trajectory. If the agent fails the post‑condition, the task‑success score is zero, regardless of how fluent the prose looks.
Tool correctness looks at the calls the agent makes to external APIs, functions, or databases. I treat each call as a mini‑experiment: did the agent pick the right tool, did it pass the right arguments, and did it call it the right number of times? I follow the guidance from the OWASP cheat sheets on safe tool use, for instance the LLM Prompt Injection Prevention Cheat Sheet and the SQL Injection Prevention Cheat Sheet to avoid accidental injection.
The axis score is the sum of points divided by the maximum possible. I keep this axis separate so a mistake in tool use does not hide a success in task completion.
Faithfulness asks whether every statement in the agent’s output can be traced back to the tool results or the context that was given to the model. I treat the output as a set of claims and check each one against the evidence. If a claim cannot be supported, I mark it as a hallucination. This axis matters most when the agent summarises documents, cites sources, or answers questions that require external data.
I record the raw counts so I can see whether a drop in faithfulness comes from a few big hallucinations or many small slips.
I want to be clear about the limits of a travel‑ready rubric. It does not guarantee that the agent is safe, cheap, or fast. It does not promise 100 % accuracy on every possible input. It does not replace a dedicated security review, a cost analysis, or a latency benchmark. The rubric only measures the three axes I defined above. If you need to know whether the agent leaks private keys, you must run a separate secrets‑management check, for example using the Secrets Management Cheat Sheet. If you need to know whether the agent stays within a budget, you must track token usage or API call costs outside of this rubric.
In short, the rubric is a tool for comparing agent quality across stacks, not a silver bullet for all production concerns.
A trajectory is the full log of what happened during one run: the initial prompt, every tool call with its arguments and results, and the final model output. I store each trajectory as a JSON object with a fixed schema so I can diff runs automatically. Below is an example of the schema I use.
| Field | Type | Description |
|---|---|---|
| prompt | string | The exact user or system prompt sent to the model. |
| calls | array | Ordered list of tool executions. |
| calls[].tool | string | Name of the tool or API. |
| calls[].args | object | Arguments passed to the tool. |
| calls[].result | any | Raw output from the tool. |
| output | string | Final text returned by the model. |
| metadata | object | Optional tags such as model name, framework version, timestamp. |
Because the schema never changes, I can compute the three axis scores for each trajectory using the same scripts. I keep a master set of trajectories that covers the core tasks I ship and the common failure modes I have seen. When I change the model, I re‑run the same trajectories and compare the new scores to the baseline.
I integrate the evaluation step into my continuous‑integration pipeline. The pipeline does the following on each commit that can affect behaviour:
This practice catches regressions early. For example, when I switched the underlying retrieval library, the tool‑correctness axis fell from 0.96 to 0.82 because the new library returned results in a different order, causing the agent to make extra calls. The drop appeared in the CI report before any user noticed a problem.
I also keep a human‑scored hold‑out set of twenty trajectories. Every month I score those with a person and compare the average to the automatic scores. If the automatic judge drifts, I recalibrate the rubric or retrain the judge model.
I keep a written checklist next to every release that touches this topic. The list is boring on purpose: name the owner, name the evidence artifact, and name the rollback. If a step cannot produce a file, a ticket, or a log line, it is not a step — it is a wish.
| Signal | What to record | Stop-ship? |
|---|---|---|
| Unexpected write or send | Tool name, args, identity | Yes until gated |
| Missing confirmation on destructive action | Call path and bypass | Yes |
| Untrusted text treated as instruction | Source surface and resulting action | Yes for agent reach |
| Scanner clean, judgment open | Deferred items + owner | Only if core path safe |
agenteval helps with the measurable first pass. Architecture choices, legal sign-off, and customer-facing claims still need a human owner. Treat every checklist row as decision support, not a certificate.
When a prospect asks whether the product "covers everything," I answer with the boundary: automated checks catch known patterns; novel chains and policy judgment still need review. That honesty keeps support load down and keeps the blog aligned with what the product actually does.
If you need a second opinion, pair this write-up with the references below and with your own staging reproduction. Do not skip the reproduction step just because a model wrote a confident summary.
Enough to cover the tasks you actually ship and the ways they fail. A handful of well-chosen trajectories beats a large set of easy ones.
You can, but record that the judge is a model and keep a fixed sample scored by humans, so you can tell whether the judge itself drifted.
Whether each claim in the output is supported by the tool results or context the agent was given, rather than produced from the model's own priors.
No. Score the trajectory too — the tool calls are where most agent failures originate.
On every change that can affect behaviour: prompt, model, tool list, or retrieval configuration.
No. It means it performed well on the tasks you tested. Security, cost and failure handling are separate evaluations.
Try the product: AgentGrade