> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/maiagent-user-guide/en/serve/ai-quality-guide.md).
# AI Customer Service Quality Management
AI customer service quality management guide: understand faithfulness, answer relevancy, and context precision, identify issues in evaluation reports, and improve role instructions and your knowledge
> **Audience**: Customer service managers, quality management staff, and customer service trainers
On this page: [1. Quick Start: Three Quality Metrics for AI Customer Service](#quick-start-three-quality-metrics)｜[2. How to Read Evaluation Reports](#how-to-read-evaluation-report)｜[3. Practical Examples: Common Issues in Apparel](#case-study-common-issues-in-apparel)｜[4. Three-Step Improvement Plan](#three-step-improvement-plan)｜[5. Daily Management Checklist](#daily-management-checklist)｜[Appendix A: Issue Diagnosis Reference](#appendix-a-issue-diagnosis-reference)｜[Appendix B: System Evaluation Metrics](#appendix-b-system-evaluation-metrics)｜[Frequently Asked Questions Q\&A](#faq)｜[Closing](#closing)
## 1. Quick Start: Three Quality Metrics for AI Customer Service
#### **How Do I View Evaluation Report Scores?**
**Path**: AgentOps (sidebar) → AI Assistant Monitoring → Conversation Records tab
The Quality Score column in the table uses faithfulness as the main representative score. Hover over a score to view individual metrics (including faithfulness, answer relevancy, and context precision) in Quality Score Details. Click the Details icon in the actions column to open the details for that conversation.
Quality Score column and Details button in Agent Monitoring → Conversation Records
{% hint style="info" %}
Quality scores are generated only after you enable automatic evaluation for the Agent: go to AI Assistants → select an Agent → Settings → Advanced Settings, turn on Enable Automatic Evaluation, and save. Once enabled, the system uses an LLM to calculate quality scores for each conversation, consuming your organization's usage quota. When disabled, the Quality Score column displays “-”.
{% endhint %}
Enable automatic evaluation in the Agent's advanced settings
#### Why Is Evaluation Necessary?
Just as you review recordings of customer service calls, you also need to check the quality of AI responses.\
The system automatically scores each conversation to help you identify issues quickly.
***
#### Three Core Metrics
| Metric | Plain-Language Explanation | Scoring Criteria |
| --------------------------- | -------------------------------------------------------------------------------- | ---------------------------------------------- |
| **Faithfulness Score** | Is the AI's information correct, without fabrications or hallucinations? |
85 or above ✅ 60-84 ⚠️ Below 60 ❌
|
| **Answer Relevancy Score** | Did the AI answer what the customer actually wanted to know? |
85 or above ✅ 60-84 ⚠️ Below 60 ❌
|
| **Context Precision Score** | Did the AI find the right reference material and focus precisely on the context? |
85 or above ✅ 60-84 ⚠️ Below 60 ❌
|
***
#### Quick Assessment
```
All three metrics > 80 → ✅ This was a good response
Any metric < 60 → ❌ Immediate improvement needed
Two or more < 70 → ⚠️ Systemic issue; comprehensive review needed
```
***
## 2. How to Read Evaluation Reports
#### Sample Report
```
Conversation ID: #20240120-001
Customer question: “Is the black trench coat in size XL still in stock?”
AI response: “The black trench coat is currently in stock. Size XL is available to order.”
Evaluation results:
├─ Faithfulness score (faithfulness_score): 45 ❌ (Claims it is in stock, but it is actually out of stock)
├─ Answer relevancy score (answer_relevancy_score): 90 ✅ (Does answer the inventory question)
└─ Context precision score (context_precision_score): 70 ⚠️ (Found trench coat information, but size information is not precise enough)
Issue diagnosis: The AI provided incorrect inventory information
```
***
#### Three Common Types of Issues
**Issue A: Low Faithfulness Score (< 60)**
**Symptom**: The AI provides incorrect or fabricated information
**Common causes**:
* Outdated reference material (prices, inventory, or policies have changed)
* Conflicting information (different documents give different answers)
* The AI “guesses” answers instead of relying on database content
**Impact**: Customers may receive incorrect information, leading to complaints
***
**Issue B: Low Answer Relevancy Score (< 60)**
**Symptom**: The AI does not answer what the customer actually wants to know
**Common causes**:
* The AI says a lot but misses the point
* The response is off-topic and includes irrelevant content
* The AI provides background information without an actual answer
**Impact**: Customers need to ask again, reducing satisfaction
***
**Issue C: Low Context Precision Score (< 60)**
**Symptom**: The AI finds incorrect reference material or material that is not precise enough
**Common causes**:
* Inaccurate keyword searches
* Confusion between information for different products/categories
* Unclear titles or categories in reference material
**Impact**: Even if the AI tries to answer well, using the wrong material leads to incorrect answers
***
## 3. Practical Examples: Common Issues in Apparel
#### Case 1: Incorrect Price (Low Faithfulness Score)
**📊 Issue Identified**
**Customer question**: “How much is this down jacket?”
**AI response**: “This down jacket costs NT$ 3,990.”
**Evaluation results**:
* Faithfulness score (faithfulness\_score): **38** ❌
* Answer relevancy score (answer\_relevancy\_score): 95 ✅
* Context precision score (context\_precision\_score): 85 ✅
**Actual situation**:\
The price has been reduced to NT$ 2,990 (after discount), but the AI still quotes the original price.
***
**💡 Improvement Plan**
**Step 1: Check the Data**
* Open the “Product Price List” in the knowledge base
* Find that the prices have not been updated
**Step 2: Update the Data**
❌ **Problematic Data**
```
Down Jacket Collection
- Classic down jacket: NT$ 3,990
- Long down jacket: NT$ 4,990
```
✅ **Corrected Data**
```
Down Jacket Collection (Winter 2024 Prices)
Product name: Classic down jacket
- Original price: NT$ 3,990
- Discounted price: NT$ 2,990 (from 2024/1/1)
- Promotion period: 2024/1/1 - 2024/2/28
Product name: Long down jacket
- Original price: NT$ 4,990
- Discounted price: NT$ 3,990 (from 2024/1/1)
- Promotion period: 2024/1/1 - 2024/2/28
Last updated: 2024/1/15
```
**Step 3: Remind the AI**
Add the following to “AI Response Principles”:
> “When answering price questions, always confirm that the price is current and specify its validity period. If a discount promotion is available, provide both the original and discounted prices.”
***
#### Case 2: Off-Topic Response (Low Answer Relevancy Score)
**📊 Issue Identified**
**Customer question**: “Can I return it if it doesn't fit?”
**AI response**: “We offer a range of sizes, including S, M, L, and XL. Each item's size chart is listed on its product page. We recommend checking the size chart and taking your measurements before purchasing.”
**Evaluation results**:
* Faithfulness score (faithfulness\_score): 90 ✅
* Answer relevancy score (answer\_relevancy\_score): **35** ❌
* Context precision score (context\_precision\_score): 60 ⚠️
**Issue diagnosis**:\
The customer asks “Can I return it?”, but the AI explains “How to choose a size” without answering the core question at all.
***
**💡 Improvement Plan**
**Step 1: Adjust Response Principles**
Add the following to “AI Response Principles”:
> **Core principle: Answer the customer's core question first**
>
> Incorrect example:
>
> * Customer asks “Can I return it?”
> * AI answers “We provide comprehensive after-sales service...” ❌ (Does not say whether a return is allowed)
>
> Correct example:
>
> * Customer asks “Can I return it?”
> * AI answers “Yes. You can request a return within 7 days of delivery if the item is unused...” ✅ (Direct answer)
**Step 2: Restructure the Data**
❌ **Problematic Data (Unfocused)**
```
After-Sales Service Information
Our company values customer rights and provides comprehensive after-sales service.
Please read the size chart carefully when purchasing and choose a suitable size.
If you have any questions, please contact customer service...
```
✅ **Corrected Data**
```
Return and Exchange Policy
Q: Can I return an item?
A: Yes. You can request a return within 7 days of delivery if the item is unused and its tags are intact.
Return conditions:
✅ Item is unused, unwashed, and has not been tried on for more than 10 minutes
✅ Tags and packaging are intact
✅ Request is submitted within 7 days of delivery
❌ Sale items and underwear cannot be returned
Return process:
1. Call customer service at 0800-XXX-XXX or complete the online form
2. Customer service provides the return address and reference number
3. Refund is issued to the original payment method within 7-14 days
---
Q: Can I exchange an item?
A: Yes. Within 7 days of delivery, you can exchange it for another size or color free of charge (same product only).
Exchange process:
1. Contact customer service to confirm availability
2. Send the item back (same conditions as returns)
3. The replacement is shipped within 3 days of receiving the item
```
**Key improvements**:
* ✅ Start with a direct “Yes” or “No”
* ✅ Use a Q\&A format for clarity
* ✅ List explicit conditions to avoid disputes
***
#### Case 3: Wrong Product Retrieved (Low Context Precision Score)
**📊 Issue Identified**
**Customer question**: “What material is the black knit top made of?”
**AI response**: “This knit top is made of 100% pure cotton. It is soft, comfortable, and suitable for all seasons.”
**Evaluation results**:
* Faithfulness score (faithfulness\_score): 88 ✅
* Answer relevancy score (answer\_relevancy\_score): 90 ✅
* Context precision score (context\_precision\_score): **48** ❌
**Actual situation**:\
The black knit top is 70% wool + 30% polyester. The AI retrieved information for the “white knit top” (100% pure cotton).
***
**💡 Improvement Plan**
**Step 1: Check Data Labels**
Problematic data filename:
```
針織上衣.pdf
```
Issue: All knit tops are in the same document, making them difficult for the AI to distinguish.
**Step 2: Improve the Data Structure**
✅ **Solution A: Separate Documents**
```
商品資料/
├─ 針織上衣_黑色_型號A001.pdf
├─ 針織上衣_白色_型號A002.pdf
├─ 針織上衣_灰色_型號A003.pdf
```
✅ **Solution B: Clear Headings**
```markdown
# Knit Top Product Information
## Black Knit Top (Model: A001)
- Color: Black
- Material: 70% wool + 30% polyester
- Suitable seasons: Fall and winter
- Care: Hand wash; do not tumble dry
## White Knit Top (Model: A002)
- Color: White
- Material: 100% pure cotton
- Suitable seasons: Spring, summer, fall, and winter
- Care: Machine washable; tumble dry on low
## Gray Knit Top (Model: A003)
- Color: Gray
- Material: 50% wool + 50% acrylic
- Suitable seasons: Fall and winter
- Care: Dry clean
```
**Step 3: Remind the AI**
Add the following to “AI Response Principles”:
> “When a customer mentions a product's color or model, always verify that the reference material is for that color and model. Different colors of the same product may have different materials and specifications.”
***
## 4. Three-Step Improvement Plan
When you identify an issue, follow this process:
```
Identify a low score
↓
Step 1: Update the data content (most important)
↓
Step 2: Adjust AI response principles
↓
Step 3: Escalate to the technical team (if needed)
```
***
#### Step 1: Update the Data Content
**When to use**:
* ✅ Low faithfulness score (incorrect or outdated data)
* ✅ Low context precision score (disorganized or poorly labeled data)
**Checklist**:
* [ ] Is the data the latest version?
* [ ] Are prices, inventory, and policies correct?
* [ ] Is information for different products clearly separated?
* [ ] Are headings clear? (So the AI can find the information easily)
* [ ] Is information presented in lists or tables? (Rather than long paragraphs)
**Data quality examples**:
❌ **Poor Data**
```
Return Information
Some products can be returned, but certain conditions apply.
Some special products cannot be returned; please check before purchasing.
Contact customer service if you need to return an item.
```
✅ **Good Data**
```
Return Policy
Returnable products:
✅ General apparel (tops, pants, jackets)
✅ Accessories (bags, hats, scarves)
Non-returnable products:
❌ Underwear and swimwear
❌ Sale items (50% off or more)
❌ Customized products
Return conditions (all must be met):
1. Within 7 days of delivery
2. Item is unused (tags intact; no signs of wear from trying it on)
3. Packaging is intact
Return process:
1. Call customer service at 0800-XXX-XXX
2. Provide the order number
3. Customer service provides the return address
4. Send the item back (registered mail recommended)
5. Refund is issued 7-14 days after receiving the item
Contact information:
- Customer service hotline: 0800-XXX-XXX (09:00-21:00)
- Online customer service: Chat box at the bottom right of the official website
- Email: service@example.com
```
***
#### Step 2: Adjust AI Response Principles
In this guide, “AI Response Principles” refers to the Agent's Role Instructions (AI Assistants → select an Agent → Settings → Response Mode Settings → Role Instructions).
**When to use**:
* ✅ Low answer relevancy score (off-topic answers)
* ✅ Low faithfulness score (AI guessing or fabricating information)
**AI Response Principles Template**:
```markdown
# AI Customer Service Response Principles
## Core Rules
1. **Answer the core question first**
- Customer asks “Can I?” → Start with “Yes” or “No”
- Customer asks “How much?” → Start with the price
- Customer asks “How do I?” → Start with the steps
2. **Only state what you know**
- All information must come from reference material
- If unsure, say “A human customer service representative needs to assist with this”
- Never guess or make assumptions
3. **Pay attention to details**
- Do not confuse colors, sizes, or models
- Confirm that prices are current
- Clearly state promotion periods
## Response Format
### Policy Questions (Returns, Exchanges, Membership, Promotions)
Paragraph 1: Answer directly with “Yes” or “No”
Paragraph 2: Explain the conditions (using a list)
Paragraph 3: Tell the customer what to do (process or contact information)
### Product Questions (Price, Material, Inventory)
Paragraph 1: Answer the question directly (price/material/availability)
Paragraph 2: Add product information (specifications, sizes, colors)
Paragraph 3: Provide a purchase link or next step
### Process Questions (How to Buy, Return, or Exchange)
Paragraph 1: Summarize the process (3-5 steps)
Paragraph 2: Explain each step in detail
Paragraph 3: Provide notes or contact information
## Prohibited Actions
❌ Do not say “usually,” “generally,” or “probably” (be specific)
❌ Do not confuse information for different products
❌ Do not omit important conditions (prices, sizes, deadlines)
❌ Do not infer information the customer has not provided
## Examples
✅ Good response:
Customer: “Can I return this jacket?” AI: “Yes. You can request a return within 7 days of delivery if the following conditions are met:
The item is unused and the tags are intact
The packaging is undamaged
It is not a sale item
Return process: Call customer service at 0800-XXX-XXX. We will provide the return address and instructions. The refund will be issued to the original payment method in about 7-14 business days.”
❌ Poor response:
Customer: “Can I return this jacket?” AI: “Our company values customer rights and provides comprehensive after-sales service. We recommend reading the product description carefully before purchasing and choosing a suitable size. If you have any questions, please contact customer service...” (Does not directly answer whether a return is allowed)
```
***
#### Step 3: Escalate to the Technical Team
**When to use**:
* Context precision scores remain low
* The same issue recurs
* Updating data and principles has not improved the situation
**Report content**:
```
Issue type: Low context precision score
Issue description:
When customers ask about “black” products, the AI often retrieves information for “white” or other colors.
Scope of impact:
This occurs in about 15% of product inquiries
Improvements attempted:
✅ Separated product information for different colors into separate files
✅ Clearly labeled colors in headings
⚠️ The issue is still not fully resolved
Suggested technical adjustment:
Improve the system's ability to identify “color” keywords more precisely
Attachments:
- test_cases_color_queries.csv (100 test questions)
- current_results.csv (current system retrieval results)
- expected_results.csv (expected correct results)
```
***
## 5. Daily Management Checklist
#### Daily Checks
**When you find an issue:**
```
If the same type of issue occurs ≥ 3 times
→ Address it immediately (update data or adjust principles)
If it involves incorrect prices or policies
→ Make an urgent correction the same day
If it is an isolated issue
→ Record it, monitor it, and add it to the discussion
```
***
#### Response Quality Tracking
**1. Data Review**
```
This week's statistics:
- Total conversations: ___
- Average faithfulness score (faithfulness_score): ___
- Average answer relevancy score (answer_relevancy_score): ___
- Average context precision score (context_precision_score): ___
- Problematic conversations: ___ (____%)
```
**2. Issue Analysis**
```
Top 3 recurring issues:
1. ________ (__ occurrences) - Which metric is low?
2. ________ (__ occurrences) - Which metric is low?
3. ________ (__ occurrences) - Which metric is low?
```
**3. Improvement Actions**
```
This week's tasks:
□ Update ___ documents (Owner: ___)
□ Adjust ___ response principles (Owner: ___)
□ Report ___ technical issues (Owner: ___)
Next week's goals:
- Reduce the rate of problematic conversations to < ____%
- Average score for every metric > ___
```
***
## Appendix A: Issue Diagnosis Reference
| Score Pattern | Possible Cause | Improvement |
| ------------------------------- | ----------------------------------------------------- | ---------------------------------- |
| **Low faithfulness score** | Outdated or incorrect data; AI fabricates information | Step 1: Update the data content |
| **Low answer relevancy score** | AI gives off-topic answers | Step 2: Adjust response principles |
| **Low context precision score** | AI finds incorrect or imprecise material | Step 1: Improve data labeling |
| **Multiple low metrics** | Systemic issue | Steps 1+2, and Step 3 if needed |
***
#### Improvement Priorities
```
First priority: Faithfulness score < 60
→ May give customers incorrect or fabricated information, leading to complaints
Second priority: Answer relevancy score < 60
→ Poor customer experience; customers must ask again
Third priority: Context precision score < 60
→ The issue may be less obvious, but it affects quality over time
```
***
## Appendix B: System Evaluation Metrics
#### Primary Metrics (No Reference Answer Required)
These three metrics are the core of this guide and can be applied directly to daily customer service conversation evaluations:
| Chinese Name | Full English Name | Description |
| --------------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------- |
| **Faithfulness Score** | Faithfulness | Evaluates whether the AI response matches database content and whether it hallucinates or fabricates information |
| **Answer Relevancy Score** | Answer Relevancy | Evaluates whether the AI response is relevant to the customer's question or is off-topic |
| **Context Precision Score** | Context Precision | Evaluates whether the AI response focuses precisely on the context and uses the right reference material |
#### Advanced Metrics (Reference Answer Required)
The following metrics require a “reference answer” (ground truth) prepared in advance and are suitable for test case evaluations:
| Chinese Name | Full English Name | Description |
| ---------------------- | ------------------ | ------------------------------------------------------------------------------ |
| **Answer Correctness** | Answer Correctness | Compares the AI response with the reference answer to evaluate correctness |
| **Answer Similarity** | Answer Similarity | Evaluates semantic similarity between the AI response and the reference answer |
| **Context Recall** | Context Recall | Evaluates whether the system retrieves all necessary reference material |
#### Other Available Metrics (DeepEval)
In the evaluation results under AgentOps → Automated Testing, the system also provides the following additional metrics for more comprehensive quality checks:
| Chinese Name | English Name | Description |
| ------------------------ | -------------------- | -------------------------------------------------------------------------- |
| **Bias** | Bias | Detects bias or discriminatory content in responses |
| **Toxicity** | Toxicity | Detects inappropriate or offensive content in responses |
| **Hallucination** | Hallucination | Detects AI-generated content that contradicts facts |
| **Contextual Relevancy** | Contextual Relevancy | Evaluates whether retrieved reference material is relevant to the question |
#### Usage Recommendations
1. **Daily monitoring**: Use the three primary metrics (faithfulness, answer relevancy, and context precision)
2. **Test evaluations**: Combine advanced metrics with reference answers for systematic evaluations
3. **Quality control**: Monitor bias and toxicity metrics in automated test results to ensure responses meet company standards
***
## Frequently Asked Questions Q\&A
**Q1: Can I manage AI customer service without technical knowledge?**\
A: Yes! Just as you manage customer service staff, you only need to:
* Review evaluation reports daily to identify problematic conversations
* Check that data is correct and complete
* Adjust the AI's “response principles” (like training customer service staff on what to say)
***
**Q2: Where do the scores come from? Does the AI evaluate itself?**\
A: No. A dedicated “evaluation system” scores responses automatically, like another AI acting as “quality control” to check the first AI's responses.
***
**Q3: Are all three metrics important? Can I look at just one?**\
A: We recommend checking all three because they reflect different issues:
* **Faithfulness score** (`faithfulness_score`): Does the AI match database content without fabricating information?
* **Answer relevancy score** (`answer_relevancy_score`): Does the AI understand the question and provide a relevant answer?
* **Context precision score** (`context_precision_score`): Does the AI focus precisely on the context and find the right reference material?
If you check only one metric, you may miss important issues.
***
**Q4: How soon will I see results after making improvements?**\
A:
* Data updates: Take effect immediately (improvements can be seen the same day)
* Response principle adjustments: Take effect immediately
* Technical adjustments: Take 2-4 weeks (depending on the complexity of the issue)
***
## Closing
Managing AI customer service is like managing a human customer service team:
✅ **Check quality regularly** (review evaluation reports)\
✅ **Keep knowledge up to date** (update data content)\
✅ **Improve service responses** (adjust response principles)\
✅ **Record improvement results** (track score changes)
Follow this guide to keep improving your AI customer service, even without technical knowledge!
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.maiagent.ai/maiagent-user-guide/en/serve/ai-quality-guide.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
