> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/maiagent-user-guide/maiagent-user-guide-en/build/ai-quality-guide.md).

# AI Customer Service Quality Management

> **Target Audience**: Customer service managers, quality management personnel, customer service trainers

## 1. Quick Start: Three Quality Metrics for AI Customer Service

#### **How to View Evaluation Report Scores**

**Path**: AgentOps (sidebar) → AI Assistant Monitoring

In the table, you can directly view the three major scoring metrics for each conversation. Click "View" to see full details.

#### Why Is Evaluation Needed?

Just like reviewing call recordings of customer service agents, we also need to check the quality of AI responses.\
The system automatically scores each conversation, helping you quickly identify issues.

***

#### Three Core Metrics

| Metric                      | Plain Language Explanation                                                            | Scoring Criteria                       |
| --------------------------- | ------------------------------------------------------------------------------------- | -------------------------------------- |
| **Faithfulness Score**      | Is the information provided by the AI correct? Does it make things up or hallucinate? | <p>85+ ✅<br>60-84 ⚠️<br>Below 60 ❌</p> |
| **Answer Relevancy Score**  | Did the AI answer the customer's actual question?                                     | <p>85+ ✅<br>60-84 ⚠️<br>Below 60 ❌</p> |
| **Context Precision Score** | Did the AI find the correct reference materials and respond precisely to the context? | <p>85+ ✅<br>60-84 ⚠️<br>Below 60 ❌</p> |

***

#### Quick Assessment Method

```
All three metrics > 80  → ✅ This response is good
Any metric < 60         → ❌ Immediate improvement needed
Two or more < 70        → ⚠️ Systemic issue, comprehensive review needed
```

***

## 2. How to Read Evaluation Reports

#### Report Example

```
Conversation ID: #20240120-001
Customer Question: "Do you still have the black trench coat in XL?"
AI Response: "The black trench coat is currently in stock. XL is available for order."

Evaluation Results:
├─ Faithfulness Score (faithfulness_score): 45 ❌ (Said it was in stock, but it actually wasn't)
├─ Answer Relevancy Score (answer_relevancy_score): 90 ✅ (Did answer the stock question)
└─ Context Precision Score (context_precision_score): 70 ⚠️ (Found trench coat data, but size information was not precise enough)

Diagnosis: AI provided incorrect inventory information
```

***

#### Three Common Issue Types

**Issue A: Low Faithfulness Score (< 60)**

**Symptoms**: The information from the AI is incorrect or fabricated

**Common Causes**:

* Reference data is outdated (prices, inventory, policies have been updated)
* Contradictory data (different documents say different things)
* AI "guesses" the answer instead of relying on database content

**Impact**: Customers may receive incorrect information, leading to complaints

***

**Issue B: Low Answer Relevancy Score (< 60)**

**Symptoms**: The AI did not answer what the customer actually asked

**Common Causes**:

* AI provided a lot of information but missed the key point
* Off-topic response with irrelevant content
* Only provided background without giving an actual answer

**Impact**: Customers need to ask again, reducing satisfaction

***

**Issue C: Low Context Precision Score (< 60)**

**Symptoms**: The AI referenced the wrong materials or was not precise enough

**Common Causes**:

* Inaccurate keyword search
* Confused data from different products/categories
* Unclear titles or categories in the reference materials

**Impact**: Even if the AI intends to answer well, using the wrong data leads to wrong answers

***

## 3. Real-World Cases: Common Issues in the Apparel Industry

#### Case 1: Wrong Price (Low Faithfulness Score)

**Problem Identified**

**Customer Question**: "How much is this down jacket?"

**AI Response**: "This down jacket is priced at NT$ 3,990."

**Evaluation Results**:

* Faithfulness Score (faithfulness\_score): **38** ❌
* Answer Relevancy Score (answer\_relevancy\_score): 95 ✅
* Context Precision Score (context\_precision\_score): 85 ✅

**Actual Situation**:\
The product had been repriced to NT$ 2,990 (after discount), but the AI still quoted the original price.

***

**Improvement Plan**

**Step 1: Check the Data**

* Open the "Product Price List" in the Knowledge Base
* Found that the price had not been updated

**Step 2: Update the Data**

❌ **Problematic Data**

```
Down Jacket Series
- Classic Down Jacket: NT$ 3,990
- Long Down Jacket: NT$ 4,990
```

✅ **Corrected Data**

```
Down Jacket Series (2024 Winter Prices)

Product Name: Classic Down Jacket
- Original Price: NT$ 3,990
- Discounted Price: NT$ 2,990 (effective 2024/1/1)
- Promotion Period: 2024/1/1 - 2024/2/28

Product Name: Long Down Jacket
- Original Price: NT$ 4,990
- Discounted Price: NT$ 3,990 (effective 2024/1/1)
- Promotion Period: 2024/1/1 - 2024/2/28

Last Updated: 2024/1/15
```

**Step 3: Instruct the AI**

Add the following to "AI Response Guidelines":

> "When answering price questions, always confirm that the price is the latest one and state the applicable period. If there is a promotional discount, provide both the original price and the discounted price."

***

#### Case 2: Off-Topic Response (Low Answer Relevancy Score)

**Problem Identified**

**Customer Question**: "I bought something that doesn't fit. Can I return it?"

**AI Response**: "We offer a variety of sizes, including S, M, L, and XL. Each garment has a size chart on the product page. We recommend checking the size chart before purchasing."

**Evaluation Results**:

* Faithfulness Score (faithfulness\_score): 90 ✅
* Answer Relevancy Score (answer\_relevancy\_score): **35** ❌
* Context Precision Score (context\_precision\_score): 60 ⚠️

**Diagnosis**:\
The customer asked "Can I return it?" but the AI talked about "How to choose a size," completely missing the core question.

***

**Improvement Plan**

**Step 1: Adjust Response Guidelines**

Add the following to "AI Response Guidelines":

> **Core Principle: Answer the customer's core question first**
>
> Bad example:
>
> * Customer asks "Can I return it?"
> * AI answers "We have comprehensive after-sales service..." ❌ (Didn't say yes or no)
>
> Good example:
>
> * Customer asks "Can I return it?"
> * AI answers "Yes. Within 7 days of delivery, if the item is unused, you can apply for a return..." ✅ (Direct answer)

**Step 2: Restructure the Data**

❌ **Problematic Data (too unfocused)**

```
After-Sales Service Description

Our company values customer rights and provides comprehensive after-sales service.
Please review the size chart carefully when purchasing to choose the right size.
If you have any questions, feel free to contact customer service...
```

✅ **Corrected Data**

```
Return and Exchange Policy

Q: Can I return an item?
A: Yes. Within 7 days of delivery, if the item is unused and tags are intact, you can apply for a return.

Return Conditions:
✅ Item is unused, unwashed, and not tried on for more than 10 minutes
✅ Tags and packaging are intact
✅ Return request submitted within 7 days of delivery
❌ Sale items and undergarments are non-returnable

Return Process:
1. Call customer service at 0800-XXX-XXX or fill out the online form
2. Customer service will provide the return address and tracking number
3. Refund will be issued to the original payment method within 7-14 days

---

Q: Can I exchange an item?
A: Yes. Within 7 days of delivery, you can exchange for a different size or color (same product only) at no charge.

Exchange Process:
1. Contact customer service to confirm stock availability
2. Ship the item back (same conditions as returns)
3. New item will be shipped within 3 days of receiving the returned item
```

**Key Improvements**:

* ✅ Starts with a direct "Yes" or "No" answer
* ✅ Uses Q\&A format for clarity at a glance
* ✅ Lists clear conditions to avoid disputes

***

#### Case 3: Wrong Product Referenced (Low Context Precision Score)

**Problem Identified**

**Customer Question**: "What material is the black knit top made of?"

**AI Response**: "This knit top is made of 100% pure cotton, soft and comfortable, suitable for all seasons."

**Evaluation Results**:

* Faithfulness Score (faithfulness\_score): 88 ✅
* Answer Relevancy Score (answer\_relevancy\_score): 90 ✅
* Context Precision Score (context\_precision\_score): **48** ❌

**Actual Situation**:\
The black knit top is 70% wool + 30% polyester. The AI referenced the "white knit top" data (100% pure cotton) instead.

***

**Improvement Plan**

**Step 1: Check Data Labels**

Problematic file name:

```
knit-top.pdf
```

Issue: All knit tops are in the same document, making it difficult for the AI to distinguish between them.

**Step 2: Improve Data Structure**

✅ **Solution A: Separate Files**

```
Product Data/
├─ knit-top_black_model-A001.pdf
├─ knit-top_white_model-A002.pdf
├─ knit-top_gray_model-A003.pdf
```

✅ **Solution B: Clear Headings**

```markdown
# Knit Top Product Information

## Black Knit Top (Model: A001)
- Color: Black
- Material: 70% Wool + 30% Polyester
- Suitable Season: Fall/Winter
- Care Instructions: Hand wash, do not tumble dry

## White Knit Top (Model: A002)
- Color: White
- Material: 100% Pure Cotton
- Suitable Season: All seasons
- Care Instructions: Machine washable, tumble dry on low

## Gray Knit Top (Model: A003)
- Color: Gray
- Material: 50% Wool + 50% Acrylic
- Suitable Season: Fall/Winter
- Care Instructions: Dry clean only
```

**Step 3: Instruct the AI**

Add the following to "AI Response Guidelines":

> "When a customer mentions a product's color or model number, always verify that the reference data corresponds to the correct color and model. Different colors of the same product may have different materials and specifications."

***

## 4. Three-Step Improvement Plan

When you identify issues, follow this process:

```
Low score detected
    ↓
Step 1: Update data content (most important)
    ↓
Step 2: Adjust AI response guidelines
    ↓
Step 3: Escalate to the technical team (if necessary)
```

***

#### Step 1: Update Data Content

**When to use**:

* ✅ Low Faithfulness Score (data is incorrect or outdated)
* ✅ Low Context Precision Score (data is disorganized or poorly labeled)

**Checklist**:

* [ ] Is the data up to date?
* [ ] Are prices, inventory, and policies correct?
* [ ] Is data for different products clearly separated?
* [ ] Are headings clear? (So the AI can find them easily)
* [ ] Is the content presented in bullet points or tables? (Rather than large blocks of text)

**Data Quality Examples**:

❌ **Poor Data**

```
Return Policy

Some products can be returned, but certain conditions must be met.
Some special products cannot be returned. Please check before purchasing.
Contact customer service if you need to return an item.
```

✅ **Good Data**

```
Return Policy

Returnable Products:
✅ General apparel (tops, pants, outerwear)
✅ Accessories (bags, hats, scarves)

Non-Returnable Products:
❌ Undergarments, swimwear
❌ Sale items (50% off or more)
❌ Customized items

Return Conditions (all must be met):
1. Within 7 days of delivery
2. Item is unused (tags intact, no signs of wear)
3. Packaging is intact

Return Process:
1. Call customer service at 0800-XXX-XXX
2. Provide your order number
3. Customer service will provide the return address
4. Ship the item back (registered mail recommended)
5. Refund issued within 7-14 days of receiving the item

Contact Information:
- Customer Service Hotline: 0800-XXX-XXX (09:00-21:00)
- Online Chat: Bottom right corner of the website
- Email: service@example.com
```

***

#### Step 2: Adjust AI Response Guidelines

**When to use**:

* ✅ Low Answer Relevancy Score (off-topic responses)
* ✅ Low Faithfulness Score (AI guessing or hallucinating)

**AI Response Guidelines Template**:

```markdown
# AI Customer Service Response Guidelines

## Core Rules

1. **Answer the core question first**
   - Customer asks "Can I/Is it possible" → Answer "Yes" or "No" first
   - Customer asks "How much" → State the price first
   - Customer asks "How to" → Provide the steps first

2. **Only state what you are certain about**
   - All information must come from reference materials
   - If unsure, say "This requires assistance from a human agent"
   - Never guess or assume

3. **Pay attention to details**
   - Do not confuse colors, sizes, or model numbers
   - Verify prices are current
   - Clearly state promotion periods

## Response Format

### Policy Questions (returns, membership, promotions)
Paragraph 1: Directly answer "Yes" or "No"
Paragraph 2: State the conditions (in bullet points)
Paragraph 3: Tell the customer what to do (process or contact info)

### Product Questions (price, material, stock)
Paragraph 1: Directly answer the question (price/material/availability)
Paragraph 2: Supplementary product info (specs, sizes, colors)
Paragraph 3: Purchase link or next steps

### Process Questions (how to buy, return, exchange)
Paragraph 1: Summarize the process (3-5 steps)
Paragraph 2: Explain each step in detail
Paragraph 3: Notes or contact information

## Prohibited Behaviors

❌ Do not say "usually," "generally," or "approximately" (be specific)
❌ Do not confuse information from different products
❌ Do not omit important details (price, size, deadline)
❌ Do not speculate about information the customer hasn't provided

## Examples

✅ Good Response:
Customer: "Can I return this jacket?"
AI: "Yes. Within 7 days of delivery, you can apply for a return if the following conditions are met:
- Item is unused, tags are intact
- Packaging is undamaged
- Not a sale item
Return process: Please call customer service at 0800-XXX-XXX, and we will provide the return address and instructions. Refunds are processed within 7-14 business days to the original payment method."

❌ Bad Response:
Customer: "Can I return this jacket?"
AI: "Our company values customer rights and provides comprehensive after-sales service. We recommend reviewing the product description carefully and choosing the right size before purchasing. If you have any questions, feel free to contact customer service..." (Did not directly answer whether a return is possible)
```

***

#### Step 3: Escalate to the Technical Team

**When to use**:

* Context Precision Score is consistently low
* The same issue occurs repeatedly
* No improvement after adjusting data and guidelines

**Escalation Content**:

```
Issue Type: Low Context Precision Score

Issue Description:
When customers ask about "black" products, the AI frequently retrieves data for "white" or other colors.

Scope of Impact:
Approximately 15% of product-related queries are affected

Attempted Improvements:
✅ Separated product data by color into individual files
✅ Explicitly labeled colors in headings
⚠️ Issue is not fully resolved

Suggested Technical Adjustment:
Request the system to more accurately recognize "color" keywords

Attachments:
- test_cases_color_queries.csv (100 test questions)
- current_results.csv (current system retrieval results)
- expected_results.csv (expected correct results)
```

***

## 5. Daily Management Checklist

#### Daily Checks

**When issues are found:**

```
If the same type of issue occurs ≥ 3 times
→ Handle immediately (update data or adjust guidelines)

If it involves price or policy errors
→ Urgent fix, complete within the same day

If it's an occasional issue
→ Log and observe, add to discussion agenda
```

***

#### Response Quality Tracking

**1. Data Review**

```
This Week's Statistics:
- Total conversations: ___ sessions
- Average Faithfulness Score (faithfulness_score): ___ points
- Average Answer Relevancy Score (answer_relevancy_score): ___ points
- Average Context Precision Score (context_precision_score): ___ points
- Anomalous conversations: ___ sessions (____%)
```

**2. Issue Analysis**

```
Top 3 High-Frequency Issues:
1. ________ (__ occurrences) - Which metric is low?
2. ________ (__ occurrences) - Which metric is low?
3. ________ (__ occurrences) - Which metric is low?
```

**3. Improvement Actions**

```
This week's tasks:
□ Update ___ documents (Owner: ___)
□ Adjust ___ response guidelines (Owner: ___)
□ Escalate ___ technical issues (Owner: ___)

Next week's goals:
- Reduce anomalous conversation rate to < ____%
- All metrics averaging > ___ points
```

***

## Appendix A: Issue Diagnosis Quick Reference

| Score Status                    | Possible Cause                               | Improvement Method                 |
| ------------------------------- | -------------------------------------------- | ---------------------------------- |
| **Low Faithfulness Score**      | Outdated or incorrect data, AI hallucination | Step 1: Update data content        |
| **Low Answer Relevancy Score**  | AI gives off-topic responses                 | Step 2: Adjust response guidelines |
| **Low Context Precision Score** | AI references wrong data or lacks precision  | Step 1: Improve data labeling      |
| **Multiple metrics are low**    | Systemic issue                               | Steps 1+2, Step 3 if necessary     |

***

#### Improvement Priority Order

```
First Priority: Faithfulness Score < 60
→ May give customers incorrect information or hallucinated content, leading to complaints

Second Priority: Answer Relevancy Score < 60
→ Poor customer experience, requires repeated inquiries

Third Priority: Context Precision Score < 60
→ Although the issue is less obvious, it will affect quality over time
```

***

## Appendix B: System Evaluation Metrics Reference

#### Primary Metrics (No Ground Truth Required)

These three metrics are the core of this guide and can be directly applied to daily customer service conversation evaluations:

| Name                        | English Full Name | Description                                                                                                     |
| --------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------- |
| **Faithfulness Score**      | Faithfulness      | Evaluates whether the AI response aligns with database content, or if it hallucinates or fabricates information |
| **Answer Relevancy Score**  | Answer Relevancy  | Evaluates whether the AI response is relevant to the customer's question, or if it's off-topic                  |
| **Context Precision Score** | Context Precision | Evaluates whether the AI response precisely addresses the context and references the correct materials          |

#### Advanced Metrics (Ground Truth Required)

The following metrics require pre-prepared "ground truth" answers and are suitable for test case evaluations:

| Name                   | English Full Name  | Description                                                                |
| ---------------------- | ------------------ | -------------------------------------------------------------------------- |
| **Answer Correctness** | Answer Correctness | Compares AI response against the ground truth to evaluate correctness      |
| **Answer Similarity**  | Answer Similarity  | Evaluates semantic similarity between the AI response and the ground truth |
| **Context Recall**     | Context Recall     | Evaluates whether the system retrieved all necessary reference materials   |

#### Other Available Metrics (DeepEval)

The system also supports the following additional evaluation metrics for more comprehensive quality checks:

| Name                        | English Name         | Description                                                                  |
| --------------------------- | -------------------- | ---------------------------------------------------------------------------- |
| **Bias Detection**          | Bias                 | Detects whether responses contain biased or discriminatory content           |
| **Toxicity Detection**      | Toxicity             | Detects whether responses contain inappropriate or offensive content         |
| **Hallucination Detection** | Hallucination        | Detects whether the AI generates content that contradicts facts              |
| **Contextual Relevancy**    | Contextual Relevancy | Evaluates whether retrieved reference materials are relevant to the question |

#### Usage Recommendations

1. **Daily Monitoring**: Use the three primary metrics (Faithfulness Score, Answer Relevancy Score, Context Precision Score)
2. **Test Evaluations**: Use advanced metrics with prepared ground truth answers for systematic evaluations
3. **Quality Assurance**: Enable bias and toxicity detection to ensure responses comply with corporate standards

***

## FAQ

**Q1: I'm not technical — can I still manage AI customer service?**\
A: Absolutely! Just like managing customer service staff, you only need to:

* Review evaluation reports daily to identify problematic conversations
* Check whether data is correct and complete
* Adjust the AI's "response guidelines" (just like training customer service scripts)

***

**Q2: How are the scores calculated? Does the AI grade itself?**\
A: No. The scoring is performed automatically by a dedicated "evaluation system," like having another AI serve as "quality control" to check the first AI's responses.

***

**Q3: Are all three metrics equally important? Can I just look at one?**\
A: We recommend looking at all three, as they reflect different issues:

* **Faithfulness Score** (`faithfulness_score`): Whether the AI aligns with database content, whether it hallucinated
* **Answer Relevancy Score** (`answer_relevancy_score`): Whether the AI understood the question and gave a relevant response
* **Context Precision Score** (`context_precision_score`): Whether the AI precisely addressed the context and found the right reference materials

Looking at only one may cause you to miss important issues.

***

**Q4: How long until I see improvement after making changes?**\
A:

* Data updates: Take effect immediately (improvements visible the same day)
* Response guideline adjustments: Take effect immediately
* Technical adjustments: Require 2-4 weeks (depending on complexity)

***

## Conclusion

Managing AI customer service is just like managing a real customer service team:

✅ **Regularly check quality** (review evaluation reports)\
✅ **Continuously update knowledge** (update data content)\
✅ **Optimize response scripts** (adjust response guidelines)\
✅ **Track improvement results** (monitor score changes)

By following this guide, you can continuously improve your AI customer service — even without technical expertise!


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.maiagent.ai/maiagent-user-guide/maiagent-user-guide-en/build/ai-quality-guide.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.