Measuring conversational AI — resolution rate, not deflection
Deflection counts the contacts that didn't reach an agent. Resolution counts the customers who actually got an answer. The gap between the two is where AI programs quietly fail — here's what to instrument instead.
On this page
The metric that flatters and the metric that matters
Every conversational AI vendor will quote you a deflection rate. It's an easy number to make look good: any contact that doesn't reach a live agent counts, including the customer who gave up after three failed turns and the one who hung up and called back an hour later. Deflection measures what your queue didn't see. It says nothing about what your customer experienced.
Resolution rate asks the harder question: of the customers who engaged with the AI, how many actually got their issue handled — verified against what happened next? That's the number that predicts CSAT, repeat-contact volume, and whether the automation is genuinely cheaper. This guide covers how to measure it.
Why deflection alone misleads
Deflection-only reporting has three failure modes, and they compound:
- Abandonment counts as success. A customer who quits the bot in frustration never reached an agent, so the deflection number goes up while the experience goes down.
- Channel-switching hides in the gap. The customer the chatbot "deflected" calls your phone line twenty minutes later. If your AI reporting and your voice reporting live in separate tools, that callback never connects to the failed bot session, and you double-count the work while reporting a win.
- It rewards walls, not answers. Optimizing deflection pushes teams toward making the agent harder to reach — burying the escape hatch, adding confirmation loops. That raises the number and lowers loyalty.
A useful rule: deflection is a capacity metric, not a quality metric. Use it for staffing math, never for judging whether the AI works.
The three-bucket model: resolved, escalated, abandoned
Every AI session should land in exactly one of three buckets, with nothing falling through:
- Resolved — the customer's intent was identified, the AI completed the task or delivered the answer, and the session ended on the customer's terms (confirmation, goodbye, or a completed action like a payment or reschedule).
- Escalated — the session handed off to a live agent, by customer request or by design. Escalation is not failure; a well-designed flow escalates complex issues fast, with the transcript and intent attached so the agent doesn't start over.
- Abandoned — the customer left mid-flow: hung up, closed the chat, or stopped responding before the intent was handled. This is the bucket deflection reporting hides, and it's the one that tells you where the flow is broken.
Resolution rate is resolved sessions divided by all sessions. Track the escalation and abandonment shares alongside it — a rising resolution rate driven by falling escalation is great; one driven by customers being trapped in the flow is not.
Measure per intent, not in aggregate
A blended resolution rate across all traffic is nearly useless, because it averages your best flow with your worst. The actionable view is per intent:
- "Check order status" might resolve at a high rate — it's a data lookup.
- "Dispute a charge" might resolve rarely, and probably should escalate by design.
- "Reschedule appointment" sitting in the middle with high abandonment tells you exactly where to look: a broken step in that one flow.
Per-intent tracking also exposes misclassification. If sessions tagged "billing question" abandon at twice the rate of everything else, the intent model may be catching contacts that belong somewhere else. Review the transcripts for your worst intent every week; fix the single biggest leak; repeat.
Verify resolution against what happens next
"The session ended without escalation" is a claim, not a verification. Close the loop with downstream signals:
- Repeat-contact rate — did the same customer contact you again on any channel within a defined window (24-72 hours is common)? A "resolved" session followed by a phone call is an unresolved session. This only works if voice, chat, and SMS share one customer history — on a platform with a single data model across channels, the callback links to the bot session automatically; with siloed per-channel tools, it usually can't.
- Task completion — for transactional intents, check the system of record. Did the payment post? Did the appointment actually move?
- Post-interaction surveys — a one-question CSAT at the end of AI sessions, scored the same way as agent sessions so the comparison is honest.
What to instrument from day one
Retrofitting measurement is painful; build it into the flows:
- Tag every session with a detected intent, even low-confidence ones (tag those "unclassified" — its volume is its own signal).
- Log an explicit terminal state on every session: resolved, escalated, abandoned. No session ends without one.
- On escalation, attach transcript and intent to the handoff, and record where in the flow it happened.
- Stamp sessions with a customer identifier so repeat contacts can be joined across channels.
- Keep AI sessions in the same reporting surface as agent interactions, broken out by intent, channel, and flow version — so when you change a flow, you can see whether resolution moved.
The short version
Deflection tells you what your agents didn't handle; resolution tells you what your customers actually got. Classify every AI session as resolved, escalated, or abandoned, measure each bucket per intent rather than in aggregate, and verify "resolved" against repeat contacts and completed tasks rather than taking the bot's word for it. Teams that report resolution rate honestly tend to ship better flows, because the metric punishes the same things customers do.
Back to
Platform
Return to the main platform page to see the full product family.
Related guides
Guide
Supervisor self-serve reporting — getting your metrics out of the IT ticket queue
What a no-code report builder actually needs, how scheduling and sharing should work, why KPI definitions belong to teams, and the pitfalls that turn self-serve into spreadsheet chaos.
Guide
PCI compliance for phone payments — keeping agents, recordings, and your audit out of scope
DTMF masking, tokenized pay links, why descoping beats securing, what Level 1 certification actually covers, and the evidence your QSA will ask for. How to take card payments over the phone without turning the contact center into a cardholder data environment.
Guide
Omnichannel readiness checklist — what to settle before you merge the queues
Consolidating voice, chat, SMS, and email into one queue fails when the groundwork is skipped. The data model, customer history, routing rules, reporting parity, and rollout order to settle first.