> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/maiagent-user-guide/maiagent-user-guide-en/agent-ops/evaluations.md).

# Automated Evaluation & AI Assistant Monitoring

## Automated Evaluation <a href="#auto-evaluation" id="auto-evaluation"></a>

The automated evaluation feature allows you to use pre-built test datasets to automatically test AI Assistant response quality. The system sends test questions to the AI Assistant, compares actual responses with expected responses, and generates detailed evaluation reports.

### Access Automated Evaluation <a href="#access-auto-evaluation" id="access-auto-evaluation"></a>

Go to "<mark style="color:blue;">AgentOps</mark>" in the left sidebar, then click "<mark style="color:blue;">Automated Testing</mark>".

The page displays all evaluation records, including evaluation name, test dataset, AI Assistant, success rate, average response time, and creation time.

### Create and Run a Test <a href="#create-and-run-test" id="create-and-run-test"></a>

1. Ensure you have created a test dataset (refer to [Test Dataset Management](/maiagent-user-guide/maiagent-user-guide-en/agent-ops/test-datasets.md))
2. Click the "<mark style="color:blue;">Create Test</mark>" button
3. Fill in the evaluation name and description, then select the test dataset and AI Assistant
4. Click "<mark style="color:blue;">Start Evaluation</mark>" and the system will automatically execute all test cases

{% hint style="info" %}
Evaluation execution time depends on the number of test cases. Typically, 50 test cases take approximately 2-3 minutes.
{% endhint %}

### View Evaluation Results <a href="#view-evaluation-results" id="view-evaluation-results"></a>

Click on an evaluation record to view the detailed report:

The detailed report includes:

* **Success Rate**: Percentage of test cases that passed
* **AI Insights**: Automatically generated evaluation summary and improvement suggestions
* **Metric Statistics**: Quality scores, response relevance, and other metrics
* **Test Case Details**: Expected response, actual response, score, and status for each question

**Success Rate Reference Benchmarks:**

| AI Assistant Type              | Recommended Success Rate |
| ------------------------------ | ------------------------ |
| Product Inquiry Assistant      | ≥ 95%                    |
| Customer Support Assistant     | ≥ 90%                    |
| General Conversation Assistant | ≥ 80%                    |

{% hint style="info" %}
For more information about insight reports, refer to: [Evaluation Insight Reports](https://github.com/Playma-Co-Ltd/maiagent-user-guide-gitbook/tree/main/en/track/evaluation-insights.md)
{% endhint %}

### Manage Evaluation Records <a href="#manage-evaluation-records" id="manage-evaluation-records"></a>

* **Search and Filter**: Filter records by test dataset, AI Assistant, or keywords
* **Re-run**: After modifying the Knowledge Base or AI settings, re-run tests to verify improvements
* **Export**: Export evaluation results in Excel format

***

## AI Assistant Monitoring <a href="#ai-agent-monitoring" id="ai-agent-monitoring"></a>

AI Assistant Monitoring provides real-time conversation operational data, allowing you to gain deep insight into the processing details, performance metrics, and quality scores of each conversation.

### Access AI Assistant Monitoring <a href="#access-ai-agent-monitoring" id="access-ai-agent-monitoring"></a>

Go to "<mark style="color:blue;">AgentOps</mark>" in the left sidebar, then click "<mark style="color:blue;">AI Assistant Monitoring</mark>".

### Monitoring Field Descriptions <a href="#monitoring-field-descriptions" id="monitoring-field-descriptions"></a>

| Field                         | Description                                                               |
| ----------------------------- | ------------------------------------------------------------------------- |
| User Input Message            | The question sent by the user to the AI                                   |
| Output Message                | The AI Assistant's response content                                       |
| AI Assistant                  | The name of the AI Assistant that handled the conversation                |
| User Feedback                 | Thumbs up 👍 or thumbs down 👎                                            |
| Faithfulness Score            | Whether the response is faithful to the Knowledge Base content            |
| Response Relevance Score      | The degree of relevance between the response and the question             |
| Response Time                 | The total time for the AI to generate a response                          |
| LLM Processing Reasoning Time | The time spent on LLM reasoning and response generation                   |
| Total Token Count             | Total tokens consumed in the conversation (including question and answer) |
| LLM                           | The name of the language model used                                       |
| User                          | The user who initiated the conversation                                   |

### Search and Filter <a href="#search-and-filter" id="search-and-filter"></a>

* **Keyword Search**: Search input/output messages or usernames
* **LLM Filter**: Select a specific language model to compare performance across different models
* **AI Assistant Filter**: Select a specific assistant to track its operational status
* **Time Range**: Select last 7 days, 30 days, 90 days, or a custom date range
* **Export**: Export monitoring data in Excel or CSV format

### Monitoring Best Practices <a href="#monitoring-best-practices" id="monitoring-best-practices"></a>

**Daily Review**: Check conversations from the last 24 hours to identify abnormal response times or errors

**Identify Performance Bottlenecks**:

* Response time > 10 seconds: Check Knowledge Base retrieval efficiency or consider a faster LLM
* High token usage: Evaluate whether the System Prompt or conversation history can be shortened

**Quality Issue Tracking**:

1. Use keyword search to find problematic conversations
2. Analyze root causes (insufficient Knowledge Base / AI misunderstanding / model limitations)
3. Add problem cases to the test dataset and run automated evaluations to verify fixes

***

## FAQ <a href="#faq" id="faq"></a>

### Q: What is the difference between Automated Evaluation and AI Assistant Monitoring? <a href="#faq-evaluation-vs-monitoring" id="faq-evaluation-vs-monitoring"></a>

|             | Automated Evaluation                     | AI Assistant Monitoring           |
| ----------- | ---------------------------------------- | --------------------------------- |
| Purpose     | Periodic quality testing                 | Real-time operational monitoring  |
| Data Source | Predefined test datasets                 | Actual user conversations         |
| Key Metrics | Success rate, response time              | Performance, cost, quality scores |
| Best For    | Quality verification, regression testing | Daily monitoring, troubleshooting |

### Q: How often should evaluations be run? <a href="#faq-how-often-to-evaluate" id="faq-how-often-to-evaluate"></a>

Recommendation: Core features weekly, full tests monthly, and immediately after major updates.

### Q: Do evaluations affect actual users? <a href="#faq-evaluation-impact-on-users" id="faq-evaluation-impact-on-users"></a>

No. Automated evaluations run in an isolated environment and do not interfere with actual user conversations.

### Q: How long is monitoring data retained? <a href="#faq-monitoring-data-retention" id="faq-monitoring-data-retention"></a>

The default retention period is 90 days. You can periodically export important data for long-term storage.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.maiagent.ai/maiagent-user-guide/maiagent-user-guide-en/agent-ops/evaluations.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.