by AgentGrade (LX AI)
Offline evaluation scores an agent against a fixed set of trajectories before release. Online monitoring watches what the agent actually does in production. The first catches regressions you can reproduce; the second catches the failures you never wrote a test for.
A trajectory is a ordered list of steps that the agent is expected to follow when it receives a specific input. Each step contains:
When you collect trajectories you usually start from a small set of representative user stories, then you expand them by adding edge cases that you have seen in logs or that you can imagine from requirement documents.
Offline evaluation compares the agent’s actual trajectory to the golden trajectory using a rubric that can contain several independent dimensions:
Each dimension receives a score between 0 and 1. The overall offline score is often a weighted average, but you can also treat each dimension as a separate gate.
Because the set of trajectories is static and the rubric is deterministic, you can run the evaluation as a step in your continuous integration pipeline. The step typically:
This gives you fast feedback on regressions that you can reproduce.
Online monitoring does not compare against a pre‑written script. Instead it streams live signals from the running agent and looks for deviations from established baselines. Typical signals include:
Because production traffic is unbounded, you collect these signals over time windows (e.g., five‑minute sliding windows) and compare them to historical quantiles.
To get the data you need to instrument the agent at a few key places:
These logs are shipped to a observability backend (Prometheus, Loki, or a custom timeseries) where you can create dashboards and alerts.
One useful rule is to alert when the proportion of calls to a particular external API that return a 5xx error exceeds the 99th percentile of the last two weeks for three consecutive windows. This catches a sudden degradation in a dependency that the offline set never covered because the failure mode only appears under real‑world load.
Below is a concrete example of how you might structure the gate for a customer‑support agent that uses a language model, a retrieval tool, and a ticket‑creation API.
| Step | Tool or Script | Output | Frequency |
|---|---|---|---|
| Checkout code and fixtures | git clone + git checkout | clean workspace | once per pipeline run |
| Install dependencies | pip install -r requirements.txt | isolated environment | once per pipeline run |
| Run agent on each trajectory | python run_agent.py --trajectories data/trajectories.jsonl | JSONL file with actual traces | once per pipeline run |
| Apply rubric | python score_traces.py --rubric config/rubric.yaml | scores per dimension and overall | once per pipeline run |
| Check thresholds | python gate_check.py --min-overall 0.92 --min-faithfulness 0.88 | pass/fail status | once per pipeline run |
| Publish report | curl -X POST -d @report.json https://metrics.example.com/ingest | record in timeseries | once per pipeline run |
Each step is deliberately kept idempotent so that re‑running the pipeline yields the same result. If any step fails, the pipeline stops and the merge request is blocked.
Effective online monitoring focuses on signals that change quickly when something goes wrong, rather than on metrics that only drift slowly.
Measure the ratio of failed tool calls to total tool calls for each external service. A sudden spike often indicates a downstream outage or a change in the service contract that the agent has not adapted to.
Some agents retry a tool call when they receive an ambiguous response. Track the average number of retries per user request. An increase can mean the model is producing more uncertain outputs or that the tool is returning vague errors.
Compute a sentence embedding for each final agent output and compare its distance to the median embedding of the last stable week. If the 95th percentile distance exceeds a preset bound, trigger an investigation.
If you run a prompt‑injection detector or a profanity filter on the agent output, log the fraction of requests that are flagged. Use the OWASP LLM Prompt Injection Prevention Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html) to decide which patterns to watch. A rise in flagged requests may indicate that the model is becoming more susceptible to jailbreak attempts.
All of these alerts should be routed to an on‑call rotation with a clear runbook that describes how to roll back the agent version, how to hot‑patch a tool wrapper, or how to collect additional trajectories for offline testing.
The real power of having both layers appears when you use production observations to improve your test suite.
When an alert fires, the first step is to extract the exact request‑response sequence that caused the problem. Most observability systems let you download a trace ID that includes:
Save this trace as a new trajectory file.
Work with the product owner or the safety lead to decide what the correct behavior should be. If the failure was due to a bug in the tool, you may decide to mock the tool response to simulate the fixed version. If the failure was due to the model’s reasoning, you may write a chain‑of‑thought that leads to the correct answer.
Place the new trajectory in the version‑controlled trajectories directory. Update the documentation to note the source (e.g., “added from production incident INC‑2024‑09‑12‑07”).
Sometimes a new trajectory reveals a dimension that was previously ignored. For example, you might discover that the agent sometimes produces a correct answer but includes an unnecessary apology phrase that annoys users. You can add a “conciseness” dimension to the rubric and give it a weight.
Run the CI pipeline again. If the new trajectory causes the overall score to dip below the threshold, you know that the current code would have failed the gate – which is exactly what you wanted to catch before release.
Repeating this cycle creates a feedback loop where production teaches the test suite what to test, and the test suite prevents known regressions from reaching users.
This discussion does not promise that offline evaluation will catch every possible bug before release. It does not claim that online monitoring will never miss a failure or that the two layers together provide a 100 % guarantee of correctness. It does not state that using these methods will make your agent flawless or that you can skip safety reviews, red‑team exercises, or formal verification. It does not suggest that the signals described are sufficient for regulatory compliance in high‑risk domains. It does not assert that the workflow will eliminate the need for human oversight or that you can rely solely on automated scores for release decisions. It does not guarantee that any particular threshold will work for every team or every model architecture. It does not assert that the table above is the only way to set up a CI gate or that the alert rules listed are universally optimal. It does not claim that the OWASP cheat sheets will prevent all prompt‑injection attempts or that they replace a dedicated security review. It does not state that the references listed are exhaustive or that they endorse any specific vendor.
Yes, if the agent touches anything a user or a system cares about. They detect different classes of failure.
No. Your test set cannot contain inputs you have not thought of yet.
No. Monitoring tells you something changed after the fact; a gate tells you before release.
Rate of change in failure signals, tool-error rates and unusual loop or retry patterns — not just volume.
Pin the rubric and the trajectories, and record changes to model, prompt and tools alongside the scores.
No. Both are engineering signals. Safety review is a separate exercise.
Try the product: AgentGrade