JournalAI Automation

How to test an AI support pilot against your actual returns policy

Build a small, auditable evaluation set, score policy errors separately from editing time, and decide which support questions to pilot first.

In this article

“Can I return this?” looks like one question in a support dashboard. It can mean three very different things: a general policy question, a request for an order-specific decision, or an exception the customer wants someone to approve. A polished answer to the wrong one can be expensive. Before piloting an AI drafting tool, give a reviewer enough information to say exactly what a good answer should do.

Write the answer key before generating replies

Start with a dated policy for one market and a small set of anonymized or invented tickets. Record what facts the agent is allowed to use, what is missing, the expected next action and any commitment the reply must avoid. Include easy questions alongside the boundary cases. Keep some tickets out of the set used to tune the prompt.

Consider a fictional store that allows standard returns within 30 days of delivery. Clearance items go to a reviewer. The following checks exercise different skills even though the customer starts each message the same way.

Scroll horizontally to read the table.

Evidence available to the assistantWhat a reviewer should expect
Standard item, delivered 12 days ago, all other policy conditions metExplain the return steps for this market and distinguish an initiated return from an issued refund.
Purchase date present, delivery date missingAsk for the missing evidence or route the ticket to someone who can retrieve it.
Clearance item, delivered yesterdayRoute the exception with the order reference. Leave the decision with the designated reviewer.
Customer only asks how long the return window isAnswer the general question from the current policy without requesting an order number.

These are test fixtures. Replace the dates, exceptions and evidence requirements with the store's own terms before scoring a tool.

Read the misses before averaging the scores

Imagine that reviewers accept 18 of 20 drafts without edits. Of the two misses, one is wordy and one promises a refund on a clearance item. A 90% acceptance rate hides the difference. Record unauthorized promises, incorrect policy, unnecessary handoffs and editorial edits in separate columns. Track review time over the same categories against the existing manual process. An apparent time saving can disappear if difficult cases are quietly excluded.

In this example, a reasonable next pilot would cover general policy questions. Keep individual refund decisions in agent review, then add fresh exception cases before expanding scope. Twenty tickets can reveal a failure worth fixing. They are too few to establish how often a rare error will occur in production. NIST's AI Risk Management Framework gives teams a broader basis for selecting evaluation measures and comparing system outputs with reference data. NIST guidance on measuring AI risks

What the 90% acceptance rate hides

Worked example · 20 draft replies reviewed against a fictional returns policy.

Accepted without edits
18
Wording needed an edit
1
Unauthorized refund promise
1
How should the two misses change the pilot?

Wordy reply. Edit for clarity and record the time it takes.

Refund promise. Keep order-specific decisions with an agent. Retest clearance-item exceptions before expanding automation.

Give the policy owner a repeatable check

Save the exact policy version, test inputs, accepted actions and reviewer decisions. When shipping windows change, rerun the affected market's cases plus a few from an unchanged market. If the tool starts applying the new window everywhere, the policy owner should be able to point to a specific failed ticket and take that category back into review. Read the klaps guide to localizing customer support

AI evaluationcustomer supportreturns policy
Author

klaps

Marketing, data, CRM, and commerce operations

KLAPS helps brands plan and implement digital marketing, measurement, CRM, conversion improvements, and commerce operations, including Shopify projects.

Marketing executionMeasurement and CRMCommerce operations
Where klaps fits

Plan your next marketing and data improvements.

Share your advertising, measurement, CRM, or commerce operations question. Choose the contact form or KakaoTalk to discuss the scope.