Improve
Ship a version because it scored, not because the demo went well.
An evaluation replays conversations that already happened against a candidate version of Scout, scores each run against the outcome your team approved, and lists what got worse. Then every run in production keeps a trace, so a bad reply is a thing you read rather than a thing you argue about.
Illustrative demo with fictional data.
Why this page exists
Every vendor in this category publishes a resolution rate. None of them let you check one.
A rate printed on a pricing page is the vendor's number, about the vendor's product, measured on somebody else's conversations. It tells you nothing about what an agent would do with the message a customer sent you last Tuesday. The useful question is not “how good is this AI” but “how good is it on my cases, and did my last change make it worse”.
- Your conversations are the only test set that predicts anything.
- The outcome your team already approved is the only grading key that is not circular.
- The number that matters is the delta against the version currently live, not an absolute.
How a run works
Five steps, repeatable, no sandbox data.
Nothing here is synthetic. The input is conversations your team handled, and the grading key is what your team decided was right at the time.
- 1 — Build the set
- Pick real conversations: a week of traffic, everything tagged returns, every conversation Scout handed over, the twelve that went wrong. Sets are saved and reused so runs are comparable.
- 2 — Freeze the expected outcome
- For each conversation, the outcome your team approved: the answer that was correct, the tool that should have run, or “this should have gone to a person”. Written once, reviewed like anything else.
- 3 — Replay the candidate
- The candidate version runs the conversation from the start — scenario matching, knowledge retrieval, tool calls against a sandbox, guardrail checks, draft. No customer is messaged.
- 4 — Score each run
- Did it reach the approved outcome? Did it cite a real source? Did it call the right tool with the right arguments? Did it hand over when it should have? Each run passes or fails on the criteria you set.
- 5 — Compare to live
- The candidate's score sits next to the version currently serving customers. A candidate that scores below live does not ship, and the run says exactly where it lost.
The output
The result you want is the list of things that got worse.
An aggregate score tells you whether to ship. The regression list tells you what to fix. A change that lifts the average while breaking refunds is a change you need to see, not a green tick.
| Run | Live v11 | Candidate v12 | Verdict |
|---|---|---|---|
| #4790 · damaged lamp | Handed over to Support | “We'll refund you today.” | Regression |
| #4663 · where is my order | Quoted the returned window | Invented a Thursday delivery | Regression |
| #4821 · delivery to Alexandria | Cited delivery-zones | Cited delivery-zones | Unchanged |
| #4755 · stock question | Handed over | Answered and cited | Improved |
Illustrative run names and outcomes from the fictional demo store.
- Blocking regressions
- Mark a criterion as blocking — never promise a refund, never quote an unreturned date — and any regression against it stops the version from being published at all.
- Diffed, not described
- Each regression shows the live version's output next to the candidate's. You are reading the two replies, not a summary of them.
- Straight back into a scenario
- A regression points at the clause that failed. Usually the fix is one line in a “Never”, and the next run proves it.
Traces
Every run — evaluated or live — keeps the same record.
The trace is the single artefact that makes all of this checkable. It is produced for the replay and for the real conversation, in the same shape, so a disagreement about what happened is a two-minute read.
- What it read
- Every knowledge passage considered, with its relevance score, including the ones dropped below the threshold.
- What it called
- The tool, the arguments, the latency and whether it was read-only or state-changing.
- What stopped it
- Every guardrail evaluated, and the one that blocked, with the proposed action it blocked.
- Who decided
- The approver's name on anything held, and the final text that was actually sent.
- Scenario matchedMatchedOrder status · 94% · runner-up: Delivery questions 61%
- Knowledge search2 useddelivery-zones (94%), payment-methods (91%) · 3 below threshold
- Tool callExecutedorders.get · order_number=4821 · 240ms · read-only
- Guardrail checks3 passedTool allowlist, refund ceiling, no unreturned dates
- Reply sentSentQuoted the returned status and window, cited 2 sources
- OutcomeResolvedResolved without a person · customer did not write back
Illustrative demo with fictional data.
In practice
A run before every change, and a run every week whether or not you changed anything.
Agent quality drifts because the world moves: a policy changes, a product goes out of stock, a supplier's delivery times slip. A weekly replay on last week's conversations catches that without anyone having to suspect it first.
- Before publishing
- Any candidate version, any edited scenario, any new tool. The run is the gate.
- On a schedule
- Replay the standing set weekly against the live version and watch the score, not the anecdotes.
- After an incident
- Add the conversation that went wrong to the set with the outcome it should have had. It is now a regression test forever.
- Exportable
- Runs, scores and traces can be exported, so an internal review or a customer's audit is a file rather than a screenshot.
Analytics or evaluations
Analytics tells you what happened. Evaluations tell you whether a change made it better.
They answer different questions and you need both. A handoff rate that drops is not automatically good news; it could be the week Scout started promising things it should not have.
Analytics
Volume, first response, resolution time, channel mix, who handled what, how often Scout handed over. Describes production.
Evaluations
One version against another, on a fixed set, graded against an approved outcome. Compares candidates.
Questions
About evaluations and traces
Do evaluations message real customers?
No. A replay reconstructs the conversation from the record and runs the candidate version against it. Tool calls go to a sandbox, and nothing is delivered on any channel.
How big does an evaluation set need to be?
Small and real beats large and synthetic. Thirty conversations that represent your actual cases will catch more regressions than a thousand generated ones. Grow the set by adding every conversation that went wrong.
Who decides what the right answer was?
Your team. The expected outcome is written by the people who own the case, which is also why the grading is worth anything — it encodes your policy, not a model's opinion of it.
What is in a trace for a live conversation?
The same rows as a replay: the scenario that matched and its score, every knowledge passage considered with its relevance, the tools called with their arguments and latency, every guardrail evaluated, any approval and who gave it, and the text that was finally sent.
Can we see a trace for a conversation a customer is complaining about?
Yes. Open the conversation and the run is attached to it. That is the point of keeping the record in the same shape for production and for replays.
Bring your customer conversations together.
Tell us a little about your team. We'll show you how BlendDesk could fit the way you already talk to customers.