Offline eval vs online monitoring for AI agents

Defining offline evaluation

Trajectory composition

A trajectory is a ordered list of steps that the agent is expected to follow when it receives a specific input. Each step contains:

- the user utterance or system event that triggers the agent - the thought or reasoning the agent should produce (if you log chain‑of‑thought) - the tool call or API request the agent should make - the expected tool response or environment feedback - the final agent output that should be shown to the user or downstream system

When you collect trajectories you usually start from a small set of representative user stories, then you expand them by adding edge cases that you have seen in logs or that you can imagine from requirement documents.

Scoring rubric

Offline evaluation compares the agent’s actual trajectory to the golden trajectory using a rubric that can contain several independent dimensions:

- Exact match – does the agent produce the same sequence of tool calls and responses? - Semantic similarity – are the utterances or outputs meaningfully equivalent even if wording differs? - Faithfulness – does the agent’s reasoning stay grounded in the provided tool outputs? - Task success – does the final output satisfy the success criteria defined for the story?

Each dimension receives a score between 0 and 1. The overall offline score is often a weighted average, but you can also treat each dimension as a separate gate.

Running offline eval in CI

Because the set of trajectories is static and the rubric is deterministic, you can run the evaluation as a step in your continuous integration pipeline. The step typically:

- checks out the agent code and the trajectory fixtures - installs any required dependencies - executes the agent against each trajectory, capturing the actual trace - applies the rubric and writes a JSON report - fails the build if any score falls below a pre‑defined threshold

This gives you fast feedback on regressions that you can reproduce.

Defining online monitoring

What you observe in production

Online monitoring does not compare against a pre‑written script. Instead it streams live signals from the running agent and looks for deviations from established baselines. Typical signals include:

- distribution of user utterance length or token count - frequency of each tool call - error rates returned by external APIs - latency of each step in the agent loop - rate of fallback or retry behaviors - semantic drift in the final output (measured with embedding distance to a recent reference set)

Because production traffic is unbounded, you collect these signals over time windows (e.g., five‑minute sliding windows) and compare them to historical quantiles.

Instrumentation points

To get the data you need to instrument the agent at a few key places:

- at the entry point – capture the raw user request and a timestamp - before each tool call – log the intended call and the arguments - after each tool response – log the success flag, error code, and payload size - after the agent produces its final output – log the output text and any safety scores - at regular intervals – emit a heartbeat that includes memory usage and CPU load

These logs are shipped to a observability backend (Prometheus, Loki, or a custom timeseries) where you can create dashboards and alerts.

Example alert rule

One useful rule is to alert when the proportion of calls to a particular external API that return a 5xx error exceeds the 99th percentile of the last two weeks for three consecutive windows. This catches a sudden degradation in a dependency that the offline set never covered because the failure mode only appears under real‑world load.

Common pitfalls when relying on just one layer

- Shipping with only offline scores – you may pass the gate but still release an agent that fails on rare user phrasing or on a tool behavior that only shows up under high concurrency. - Shipping with only dashboards – you might notice a problem days after the offending change has been merged, making rollback harder and increasing user impact. - Comparing production numbers across a period when the model, prompt, or tool list also changed – the observed drift could be due to the intentional upgrade rather than a regression, leading to false alarms. - Using the same threshold for offline and online signals – offline scores are usually tight (you expect near‑perfect matches) while online metrics are noisy; applying a single cut‑off can cause either too many false positives or missed detections. - Neglecting to version the trajectory set – if you update the golden trajectories without recording the change, historic offline scores become incomparable.

Building a regression gate in CI

Below is a concrete example of how you might structure the gate for a customer‑support agent that uses a language model, a retrieval tool, and a ticket‑creation API.

Step | Tool or Script | Output | Frequency Checkout code and fixtures | git clone + git checkout | clean workspace | once per pipeline run Install dependencies | pip install -r requirements.txt | isolated environment | once per pipeline run Run agent on each trajectory | python run_agent.py --trajectories data/trajectories.jsonl | JSONL file with actual traces | once per pipeline run Apply rubric | python score_traces.py --rubric config/rubric.yaml | scores per dimension and overall | once per pipeline run Check thresholds | python gate_check.py --min-overall 0.92 --min-faithfulness 0.88 | pass/fail status | once per pipeline run Publish report | curl -X POST -d @report.json https://metrics.example.com/ingest | record in timeseries | once per pipeline run

Each step is deliberately kept idempotent so that re‑running the pipeline yields the same result. If any step fails, the pipeline stops and the merge request is blocked.

Setting up production alerts that matter

Effective online monitoring focuses on signals that change quickly when something goes wrong, rather than on metrics that only drift slowly.

Tool‑error rate

Measure the ratio of failed tool calls to total tool calls for each external service. A sudden spike often indicates a downstream outage or a change in the service contract that the agent has not adapted to.

Loop‑retry count

Some agents retry a tool call when they receive an ambiguous response. Track the average number of retries per user request. An increase can mean the model is producing more uncertain outputs or that the tool is returning vague errors.

Output embedding drift

Compute a sentence embedding for each final agent output and compare its distance to the median embedding of the last stable week. If the 95th percentile distance exceeds a preset bound, trigger an investigation.

Safety‑signal monitor

If you run a prompt‑injection detector or a profanity filter on the agent output, log the fraction of requests that are flagged. Use the OWASP LLM Prompt Injection Prevention Cheat Sheet (https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html) to decide which patterns to watch. A rise in flagged requests may indicate that the model is becoming more susceptible to jailbreak attempts.

All of these alerts should be routed to an on‑call rotation with a clear runbook that describes how to roll back the agent version, how to hot‑patch a tool wrapper, or how to collect additional trajectories for offline testing.

Closing the loop: turning monitoring findings into offline tests

The real power of having both layers appears when you use production observations to improve your test suite.

Capture a novel failure

When an alert fires, the first step is to extract the exact request‑response sequence that caused the problem. Most observability systems let you download a trace ID that includes:

- the original user utterance - every tool call and its arguments - every tool response - the final agent output

Save this trace as a new trajectory file.

Label the expected behavior

Work with the product owner or the safety lead to decide what the correct behavior should be. If the failure was due to a bug in the tool, you may decide to mock the tool response to simulate the fixed version. If the failure was due to the model’s reasoning, you may write a chain‑of‑thought that leads to the correct answer.

Add to the offline set

Place the new trajectory in the version‑controlled trajectories directory. Update the documentation to note the source (e.g., “added from production incident INC‑2024‑09‑12‑07”).

Adjust the rubric if needed

Sometimes a new trajectory reveals a dimension that was previously ignored. For example, you might discover that the agent sometimes produces a correct answer but includes an unnecessary apology phrase that annoys users. You can add a “conciseness” dimension to the rubric and give it a weight.

Re‑run the gate

Run the CI pipeline again. If the new trajectory causes the overall score to dip below the threshold, you know that the current code would have failed the gate – which is exactly what you wanted to catch before release.

Repeating this cycle creates a feedback loop where production teaches the test suite what to test, and the test suite prevents known regressions from reaching users.

What this is NOT

This discussion does not promise that offline evaluation will catch every possible bug before release. It does not claim that online monitoring will never miss a failure or that the two layers together provide a 100 % guarantee of correctness. It does not state that using these methods will make your agent flawless or that you can skip safety reviews, red‑team exercises, or formal verification. It does not suggest that the signals described are sufficient for regulatory compliance in high‑risk domains. It does not assert that the workflow will eliminate the need for human oversight or that you can rely solely on automated scores for release decisions. It does not guarantee that any particular threshold will work for every team or every model architecture. It does not assert that the table above is the only way to set up a CI gate or that the alert rules listed are universally optimal. It does not claim that the OWASP cheat sheets will prevent all prompt‑injection attempts or that they replace a dedicated security review. It does not state that the references listed are exhaustive or that they endorse any specific vendor.

References

- https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html - https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html - https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html - https://docs.agents-squads.com/guides/config-files

Frequently asked questions

Do we need both?

Yes, if the agent touches anything a user or a system cares about. They detect different classes of failure.

Can offline tests replace monitoring?

No. Your test set cannot contain inputs you have not thought of yet.

Can monitoring replace offline tests?

No. Monitoring tells you something changed after the fact; a gate tells you before release.

What should we alert on in production?

Rate of change in failure signals, tool-error rates and unusual loop or retry patterns — not just volume.

How do we keep scores comparable over time?

Pin the rubric and the trajectories, and record changes to model, prompt and tools alongside the scores.

Is either layer a safety certification?

No. Both are engineering signals. Safety review is a separate exercise.

References

- NIST — AI Risk Management Framework - Stanford CRFM — HELM (Holistic Evaluation of Language Models)

Try the product: AgentGrade

2026-09-20 · primary sources only · no fabricated traffic metrics · decision-support only

Related articles

- Agent eval rubrics that travel across stacks - Explore the full AI micro-SaaS topic matrix →