AI Customer Service Quality Management

Target Audience: Customer service managers, quality management personnel, customer service trainers

1. Quick Start: Three Quality Metrics for AI Customer Service

How to View Evaluation Report Scores

Path: AgentOps (sidebar) → AI Assistant Monitoring

In the table, you can directly view the three major scoring metrics for each conversation. Click "View" to see full details.

Why Is Evaluation Needed?

Just like reviewing call recordings of customer service agents, we also need to check the quality of AI responses. The system automatically scores each conversation, helping you quickly identify issues.

---

Three Core Metrics

Metric	Plain Language Explanation	Scoring Criteria
Faithfulness Score	Is the information provided by the AI correct? Does it make things up or hallucinate?	85+ ✅ 60-84 ⚠️ Below 60 ❌
Answer Relevancy Score	Did the AI answer the customer's actual question?	85+ ✅ 60-84 ⚠️ Below 60 ❌
Context Precision Score	Did the AI find the correct reference materials and respond precisely to the context?	85+ ✅ 60-84 ⚠️ Below 60 ❌

---

Quick Assessment Method

All three metrics > 80  → ✅ This response is good
Any metric < 60         → ❌ Immediate improvement needed
Two or more < 70        → ⚠️ Systemic issue, comprehensive review needed

---

2. How to Read Evaluation Reports

Report Example

Conversation ID: #20240120-001
Customer Question: "Do you still have the black trench coat in XL?"
AI Response: "The black trench coat is currently in stock. XL is available for order."

Evaluation Results:
├─ Faithfulness Score (faithfulness_score): 45 ❌ (Said it was in stock, but it actually wasn't)
├─ Answer Relevancy Score (answer_relevancy_score): 90 ✅ (Did answer the stock question)
└─ Context Precision Score (context_precision_score): 70 ⚠️ (Found trench coat data, but size information was not precise enough)

Diagnosis: AI provided incorrect inventory information

---

Three Common Issue Types

Issue A: Low Faithfulness Score (< 60)

Symptoms: The information from the AI is incorrect or fabricated

Common Causes:

• Reference data is outdated (prices, inventory, policies have been updated)

• Contradictory data (different documents say different things)

• AI "guesses" the answer instead of relying on database content

Impact: Customers may receive incorrect information, leading to complaints

---

Issue B: Low Answer Relevancy Score (< 60)

Symptoms: The AI did not answer what the customer actually asked

Common Causes:

• AI provided a lot of information but missed the key point

• Off-topic response with irrelevant content

• Only provided background without giving an actual answer

Impact: Customers need to ask again, reducing satisfaction

---

Issue C: Low Context Precision Score (< 60)

Symptoms: The AI referenced the wrong materials or was not precise enough

Common Causes:

• Inaccurate keyword search

• Confused data from different products/categories

• Unclear titles or categories in the reference materials

Impact: Even if the AI intends to answer well, using the wrong data leads to wrong answers

---

3. Real-World Cases: Common Issues in the Apparel Industry

Case 1: Wrong Price (Low Faithfulness Score)

Problem Identified

Customer Question: "How much is this down jacket?"

AI Response: "This down jacket is priced at NT$ 3,990."

Evaluation Results:

• Faithfulness Score (faithfulness_score): 38 ❌

• Answer Relevancy Score (answer_relevancy_score): 95 ✅

• Context Precision Score (context_precision_score): 85 ✅

Actual Situation: The product had been repriced to NT$ 2,990 (after discount), but the AI still quoted the original price.

---

Improvement Plan

Step 1: Check the Data

• Open the "Product Price List" in the Knowledge Base

• Found that the price had not been updated

Step 2: Update the Data

❌ Problematic Data

Down Jacket Series
- Classic Down Jacket: NT$ 3,990
- Long Down Jacket: NT$ 4,990

✅ Corrected Data

Down Jacket Series (2024 Winter Prices)

Product Name: Classic Down Jacket
- Original Price: NT$ 3,990
- Discounted Price: NT$ 2,990 (effective 2024/1/1)
- Promotion Period: 2024/1/1 - 2024/2/28

Product Name: Long Down Jacket
- Original Price: NT$ 4,990
- Discounted Price: NT$ 3,990 (effective 2024/1/1)
- Promotion Period: 2024/1/1 - 2024/2/28

Last Updated: 2024/1/15

Step 3: Instruct the AI

Add the following to "AI Response Guidelines":

"When answering price questions, always confirm that the price is the latest one and state the applicable period. If there is a promotional discount, provide both the original price and the discounted price."

---

Case 2: Off-Topic Response (Low Answer Relevancy Score)

Problem Identified

Customer Question: "I bought something that doesn't fit. Can I return it?"

AI Response: "We offer a variety of sizes, including S, M, L, and XL. Each garment has a size chart on the product page. We recommend checking the size chart before purchasing."

Evaluation Results:

• Faithfulness Score (faithfulness_score): 90 ✅

• Answer Relevancy Score (answer_relevancy_score): 35 ❌

• Context Precision Score (context_precision_score): 60 ⚠️

Diagnosis: The customer asked "Can I return it?" but the AI talked about "How to choose a size," completely missing the core question.

---

Improvement Plan

Step 1: Adjust Response Guidelines

Add the following to "AI Response Guidelines":

Core Principle: Answer the customer's core question first

Bad example:

• Customer asks "Can I return it?"

• AI answers "We have comprehensive after-sales service..." ❌ (Didn't say yes or no)

Good example:

• Customer asks "Can I return it?"

• AI answers "Yes. Within 7 days of delivery, if the item is unused, you can apply for a return..." ✅ (Direct answer)

Step 2: Restructure the Data

❌ Problematic Data (too unfocused)

After-Sales Service Description

Our company values customer rights and provides comprehensive after-sales service.
Please review the size chart carefully when purchasing to choose the right size.
If you have any questions, feel free to contact customer service...

✅ Corrected Data

Return and Exchange Policy

Q: Can I return an item?
A: Yes. Within 7 days of delivery, if the item is unused and tags are intact, you can apply for a return.

Return Conditions:
✅ Item is unused, unwashed, and not tried on for more than 10 minutes
✅ Tags and packaging are intact
✅ Return request submitted within 7 days of delivery
❌ Sale items and undergarments are non-returnable

Return Process:
1. Call customer service at 0800-XXX-XXX or fill out the online form
2. Customer service will provide the return address and tracking number
3. Refund will be issued to the original payment method within 7-14 days

---

Q: Can I exchange an item?
A: Yes. Within 7 days of delivery, you can exchange for a different size or color (same product only) at no charge.

Exchange Process:
1. Contact customer service to confirm stock availability
2. Ship the item back (same conditions as returns)
3. New item will be shipped within 3 days of receiving the returned item

Key Improvements:

• ✅ Starts with a direct "Yes" or "No" answer

• ✅ Uses Q&A format for clarity at a glance

• ✅ Lists clear conditions to avoid disputes

---

Case 3: Wrong Product Referenced (Low Context Precision Score)

Problem Identified

Customer Question: "What material is the black knit top made of?"

AI Response: "This knit top is made of 100% pure cotton, soft and comfortable, suitable for all seasons."

Evaluation Results:

• Faithfulness Score (faithfulness_score): 88 ✅

• Answer Relevancy Score (answer_relevancy_score): 90 ✅

• Context Precision Score (context_precision_score): 48 ❌

Actual Situation: The black knit top is 70% wool + 30% polyester. The AI referenced the "white knit top" data (100% pure cotton) instead.

---

Improvement Plan

Step 1: Check Data Labels

Problematic file name:

knit-top.pdf

Issue: All knit tops are in the same document, making it difficult for the AI to distinguish between them.

Step 2: Improve Data Structure

✅ Solution A: Separate Files

Product Data/
├─ knit-top_black_model-A001.pdf
├─ knit-top_white_model-A002.pdf
├─ knit-top_gray_model-A003.pdf

✅ Solution B: Clear Headings

# Knit Top Product Information

## Black Knit Top (Model: A001)
- Color: Black
- Material: 70% Wool + 30% Polyester
- Suitable Season: Fall/Winter
- Care Instructions: Hand wash, do not tumble dry

## White Knit Top (Model: A002)
- Color: White
- Material: 100% Pure Cotton
- Suitable Season: All seasons
- Care Instructions: Machine washable, tumble dry on low

## Gray Knit Top (Model: A003)
- Color: Gray
- Material: 50% Wool + 50% Acrylic
- Suitable Season: Fall/Winter
- Care Instructions: Dry clean only

Step 3: Instruct the AI

Add the following to "AI Response Guidelines":

"When a customer mentions a product's color or model number, always verify that the reference data corresponds to the correct color and model. Different colors of the same product may have different materials and specifications."

---

4. Three-Step Improvement Plan

When you identify issues, follow this process:

Low score detected
    ↓
Step 1: Update data content (most important)
    ↓
Step 2: Adjust AI response guidelines
    ↓
Step 3: Escalate to the technical team (if necessary)

---

Step 1: Update Data Content

When to use:

• ✅ Low Faithfulness Score (data is incorrect or outdated)

• ✅ Low Context Precision Score (data is disorganized or poorly labeled)

Checklist:

• Is the data up to date?

• Are prices, inventory, and policies correct?

• Is data for different products clearly separated?

• Are headings clear? (So the AI can find them easily)

• Is the content presented in bullet points or tables? (Rather than large blocks of text)

Data Quality Examples:

❌ Poor Data

Return Policy

Some products can be returned, but certain conditions must be met.
Some special products cannot be returned. Please check before purchasing.
Contact customer service if you need to return an item.

✅ Good Data

Return Policy

Returnable Products:
✅ General apparel (tops, pants, outerwear)
✅ Accessories (bags, hats, scarves)

Non-Returnable Products:
❌ Undergarments, swimwear
❌ Sale items (50% off or more)
❌ Customized items

Return Conditions (all must be met):
1. Within 7 days of delivery
2. Item is unused (tags intact, no signs of wear)
3. Packaging is intact

Return Process:
1. Call customer service at 0800-XXX-XXX
2. Provide your order number
3. Customer service will provide the return address
4. Ship the item back (registered mail recommended)
5. Refund issued within 7-14 days of receiving the item

Contact Information:
- Customer Service Hotline: 0800-XXX-XXX (09:00-21:00)
- Online Chat: Bottom right corner of the website
- Email: [e-mail]

---

Step 2: Adjust AI Response Guidelines

When to use:

• ✅ Low Answer Relevancy Score (off-topic responses)

• ✅ Low Faithfulness Score (AI guessing or hallucinating)

AI Response Guidelines Template:

# AI Customer Service Response Guidelines

## Core Rules

1. **Answer the core question first**
   - Customer asks "Can I/Is it possible" → Answer "Yes" or "No" first
   - Customer asks "How much" → State the price first
   - Customer asks "How to" → Provide the steps first

2. **Only state what you are certain about**
   - All information must come from reference materials
   - If unsure, say "This requires assistance from a human agent"
   - Never guess or assume

3. **Pay attention to details**
   - Do not confuse colors, sizes, or model numbers
   - Verify prices are current
   - Clearly state promotion periods

## Response Format

### Policy Questions (returns, membership, promotions)
Paragraph 1: Directly answer "Yes" or "No"
Paragraph 2: State the conditions (in bullet points)
Paragraph 3: Tell the customer what to do (process or contact info)

### Product Questions (price, material, stock)
Paragraph 1: Directly answer the question (price/material/availability)
Paragraph 2: Supplementary product info (specs, sizes, colors)
Paragraph 3: Purchase link or next steps

### Process Questions (how to buy, return, exchange)
Paragraph 1: Summarize the process (3-5 steps)
Paragraph 2: Explain each step in detail
Paragraph 3: Notes or contact information

## Prohibited Behaviors

❌ Do not say "usually," "generally," or "approximately" (be specific)
❌ Do not confuse information from different products
❌ Do not omit important details (price, size, deadline)
❌ Do not speculate about information the customer hasn't provided

## Examples

✅ Good Response:
Customer: "Can I return this jacket?"
AI: "Yes. Within 7 days of delivery, you can apply for a return if the following conditions are met:
- Item is unused, tags are intact
- Packaging is undamaged
- Not a sale item
Return process: Please call customer service at 0800-XXX-XXX, and we will provide the return address and instructions. Refunds are processed within 7-14 business days to the original payment method."

❌ Bad Response:
Customer: "Can I return this jacket?"
AI: "Our company values customer rights and provides comprehensive after-sales service. We recommend reviewing the product description carefully and choosing the right size before purchasing. If you have any questions, feel free to contact customer service..." (Did not directly answer whether a return is possible)

---

Step 3: Escalate to the Technical Team

When to use:

• Context Precision Score is consistently low

• The same issue occurs repeatedly

• No improvement after adjusting data and guidelines

Escalation Content:

Issue Type: Low Context Precision Score

Issue Description:
When customers ask about "black" products, the AI frequently retrieves data for "white" or other colors.

Scope of Impact:
Approximately 15% of product-related queries are affected

Attempted Improvements:
✅ Separated product data by color into individual files
✅ Explicitly labeled colors in headings
⚠️ Issue is not fully resolved

Suggested Technical Adjustment:
Request the system to more accurately recognize "color" keywords

Attachments:
- test_cases_color_queries.csv (100 test questions)
- current_results.csv (current system retrieval results)
- expected_results.csv (expected correct results)

---

5. Daily Management Checklist

Daily Checks

When issues are found:

If the same type of issue occurs ≥ 3 times
→ Handle immediately (update data or adjust guidelines)

If it involves price or policy errors
→ Urgent fix, complete within the same day

If it's an occasional issue
→ Log and observe, add to discussion agenda

---

Response Quality Tracking

1. Data Review

This Week's Statistics:
- Total conversations: ___ sessions
- Average Faithfulness Score (faithfulness_score): ___ points
- Average Answer Relevancy Score (answer_relevancy_score): ___ points
- Average Context Precision Score (context_precision_score): ___ points
- Anomalous conversations: ___ sessions (____%)

2. Issue Analysis

Top 3 High-Frequency Issues:
1. ________ (__ occurrences) - Which metric is low?
2. ________ (__ occurrences) - Which metric is low?
3. ________ (__ occurrences) - Which metric is low?

3. Improvement Actions

This week's tasks:
□ Update ___ documents (Owner: ___)
□ Adjust ___ response guidelines (Owner: ___)
□ Escalate ___ technical issues (Owner: ___)

Next week's goals:
- Reduce anomalous conversation rate to < ____%
- All metrics averaging > ___ points

---

Appendix A: Issue Diagnosis Quick Reference

Score Status	Possible Cause	Improvement Method
Low Faithfulness Score	Outdated or incorrect data, AI hallucination	Step 1: Update data content
Low Answer Relevancy Score	AI gives off-topic responses	Step 2: Adjust response guidelines
Low Context Precision Score	AI references wrong data or lacks precision	Step 1: Improve data labeling
Multiple metrics are low	Systemic issue	Steps 1+2, Step 3 if necessary

---

Improvement Priority Order

First Priority: Faithfulness Score < 60
→ May give customers incorrect information or hallucinated content, leading to complaints

Second Priority: Answer Relevancy Score < 60
→ Poor customer experience, requires repeated inquiries

Third Priority: Context Precision Score < 60
→ Although the issue is less obvious, it will affect quality over time

---

Appendix B: System Evaluation Metrics Reference

Primary Metrics (No Ground Truth Required)

These three metrics are the core of this guide and can be directly applied to daily customer service conversation evaluations:

Name	English Full Name	Description
Faithfulness Score	Faithfulness	Evaluates whether the AI response aligns with database content, or if it hallucinates or fabricates information
Answer Relevancy Score	Answer Relevancy	Evaluates whether the AI response is relevant to the customer's question, or if it's off-topic
Context Precision Score	Context Precision	Evaluates whether the AI response precisely addresses the context and references the correct materials

Advanced Metrics (Ground Truth Required)

The following metrics require pre-prepared "ground truth" answers and are suitable for test case evaluations:

Name	English Full Name	Description
Answer Correctness	Answer Correctness	Compares AI response against the ground truth to evaluate correctness
Answer Similarity	Answer Similarity	Evaluates semantic similarity between the AI response and the ground truth
Context Recall	Context Recall	Evaluates whether the system retrieved all necessary reference materials

Other Available Metrics (DeepEval)

The system also supports the following additional evaluation metrics for more comprehensive quality checks:

Name	English Name	Description
Bias Detection	Bias	Detects whether responses contain biased or discriminatory content
Toxicity Detection	Toxicity	Detects whether responses contain inappropriate or offensive content
Hallucination Detection	Hallucination	Detects whether the AI generates content that contradicts facts
Contextual Relevancy	Contextual Relevancy	Evaluates whether retrieved reference materials are relevant to the question

Usage Recommendations

1. Daily Monitoring: Use the three primary metrics (Faithfulness Score, Answer Relevancy Score, Context Precision Score)

2. Test Evaluations: Use advanced metrics with prepared ground truth answers for systematic evaluations

3. Quality Assurance: Enable bias and toxicity detection to ensure responses comply with corporate standards

---

FAQ

Q1: I'm not technical — can I still manage AI customer service? A: Absolutely! Just like managing customer service staff, you only need to:

• Review evaluation reports daily to identify problematic conversations

• Check whether data is correct and complete

• Adjust the AI's "response guidelines" (just like training customer service scripts)

---

Q2: How are the scores calculated? Does the AI grade itself? A: No. The scoring is performed automatically by a dedicated "evaluation system," like having another AI serve as "quality control" to check the first AI's responses.

---

Q3: Are all three metrics equally important? Can I just look at one? A: We recommend looking at all three, as they reflect different issues:

• Faithfulness Score (faithfulness_score): Whether the AI aligns with database content, whether it hallucinated

• Answer Relevancy Score (answer_relevancy_score): Whether the AI understood the question and gave a relevant response

• Context Precision Score (context_precision_score): Whether the AI precisely addressed the context and found the right reference materials

Looking at only one may cause you to miss important issues.

---

Q4: How long until I see improvement after making changes? A:

• Data updates: Take effect immediately (improvements visible the same day)

• Response guideline adjustments: Take effect immediately

• Technical adjustments: Require 2-4 weeks (depending on complexity)

---

Conclusion

Managing AI customer service is just like managing a real customer service team:

✅ Regularly check quality (review evaluation reports) ✅ Continuously update knowledge (update data content) ✅ Optimize response scripts (adjust response guidelines) ✅ Track improvement results (monitor score changes)

By following this guide, you can continuously improve your AI customer service — even without technical expertise!