> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/maiagent-user-guide/en/build/agent-ops/evaluation-insights.md).

# Evaluation Insights Reports

Evaluation insight reports: automatically analyze metric scores, identify issue types, and generate improvement suggestions after batch tests. Learn how insights are calculated and where to apply them

This guide explains how to use evaluation insights to quickly understand your Agent's test results and get automatically generated improvement suggestions.

## What Are Evaluation Insights? <a href="#what-is-evaluation-insight" id="what-is-evaluation-insight"></a>

Evaluation insights extend AgentOps automated testing. After you complete a single-turn batch test, the system displays metric scores and automatically analyzes the test results to generate an "<mark style="color:blue;">AI Insights</mark>" section on the evaluation details page (called an insight report in this guide). Multi-turn conversation evaluations currently do not generate insight reports.

This report tells you:

* **How the test results performed overall**
* **Which metrics have low pass rates**
* **What specific improvements to make**
* **Which items to address first**

It is like having an experienced consultant review your test report and provide a set of improvement recommendations.

## Key Benefits of Evaluation Insights <a href="#core-advantages" id="core-advantages"></a>

### 📊 Automated Analysis <a href="#automated-analysis" id="automated-analysis"></a>

You do not need to interpret complex scoring data manually. The system automatically identifies patterns in issues and organizes them into an easy-to-understand report.

### 🎯 Priority Recommendations <a href="#priority-recommendations" id="priority-recommendations"></a>

The system recommends which items to address first based on the severity and scope of impact of each issue, helping you make improvements more efficiently.

### 💡 Specific Improvement Guidance <a href="#concrete-improvement-directions" id="concrete-improvement-directions"></a>

The report goes beyond identifying issues to offer specific suggestions, such as which settings to adjust and what content to add.

### 🌐 Multilingual Support <a href="#multilingual-support" id="multilingual-support"></a>

Insight reports are generated in the language of the admin interface at the time the evaluation is created. For example, an evaluation created in the Traditional Chinese interface produces a Traditional Chinese report.

## How to Use Evaluation Insights <a href="#how-to-use-evaluation-insight" id="how-to-use-evaluation-insight"></a>

### Step 1: Complete a Batch Test <a href="#step-complete-batch-test" id="step-complete-batch-test"></a>

1. Go to "<mark style="color:blue;">AgentOps</mark>" → "<mark style="color:blue;">Automated Testing</mark>"
2. Run a batch test (see [Automated Evaluation](/maiagent-user-guide/en/build/agent-ops/evaluations.md#auto-evaluation))
3. Wait for the test to finish

### Step 2: View Evaluation Results <a href="#step-view-evaluation-results" id="step-view-evaluation-results"></a>

1. In the automated testing list, click the first (document) icon in the "<mark style="color:blue;">Actions</mark>" column of the corresponding row to open the evaluation details page
2. Review the evaluation metrics
3. Find the "<mark style="color:blue;">AI Insights</mark>" section

### Step 3: Report Language <a href="#step-select-report-language" id="step-select-report-language"></a>

The AI Insights section does not have a language selector. Insight reports are generated in the language of the admin interface at the time the evaluation is created. To get a report in a specific language, switch the admin interface to that language before creating the evaluation.

### Step 4: Read the Insight Report <a href="#step-read-insight-report" id="step-read-insight-report"></a>

The AI Insights section includes the following:

#### 1. Overall Evaluation Summary <a href="#overall-evaluation-summary" id="overall-evaluation-summary"></a>

"<mark style="color:blue;">Evaluation Summary</mark>" describes the evaluation's overall performance, main strengths, and issues in two or three sentences.

#### 2. Issue Analysis <a href="#issue-analysis" id="issue-analysis"></a>

"<mark style="color:blue;">Metric Statistics</mark>" lists the following for each evaluation metric (such as answer relevancy, context precision, bias, offensiveness, and hallucination):

* Pass rate, labeled <mark style="color:blue;">Good</mark> (80% or above), <mark style="color:blue;">Warning</mark> (50% or above but below 80%), or <mark style="color:blue;">Critical</mark> (below 50%) based on the pass rate
* Number of failed cases / number of completed cases
* Average score

These statistics cover all completed test cases (including those marked as unstable), so their basis differs from the success rate shown above.

#### 3. Improvement Suggestions <a href="#improvement-suggestions" id="improvement-suggestions"></a>

"<mark style="color:blue;">Improvement Suggestions</mark>" lists recommended actions individually. Each includes:

* <mark style="color:blue;">Priority</mark>: High, Medium, or Low
* <mark style="color:blue;">Reason Analysis</mark>: Why this action is recommended
* <mark style="color:blue;">Affected Metrics</mark>: Which metrics the suggestion addresses

### Step 5: Make Improvements Based on the Suggestions <a href="#step-apply-improvements" id="step-apply-improvements"></a>

1. Address issues one by one in the order of priority given in the report
2. Adjust Agent settings using the specific improvement steps as a guide
3. Run the test again after making adjustments to verify their effectiveness

## How Evaluation Insights Are Calculated <a href="#how-insights-are-calculated" id="how-insights-are-calculated"></a>

When an evaluation ends, the system first aggregates each metric's pass rate, failed case count, and average score. AI then uses these statistics to determine whether failures are systematic or occasional, prioritize issues by severity, and write a summary and improvement suggestions. AI relies on metric statistics; it does not read each case's response individually. To understand why a particular case failed, check the case list at the bottom of the details page.

{% hint style="info" %}
Insight reports are generated as the evaluation wraps up and appear on the details page only after the evaluation ends. If an error occurs while generating insights, evaluation scores are still retained, but that evaluation will not have an AI Insights section.
{% endhint %}

## Practical Use Cases <a href="#use-cases" id="use-cases"></a>

### 🏥 Customer Service for a Medical Clinic <a href="#use-case-medical-clinic" id="use-case-medical-clinic"></a>

**Metric Statistics**:

* Hallucination: 40% pass rate (<mark style="color:blue;">Critical</mark>)
* Answer relevancy: 90% pass rate (<mark style="color:blue;">Good</mark>)

**Improvement Suggestions** (illustrative):

* Recommended action: Limit the response scope in the role instructions, require answers to be based only on knowledge base content, and direct users to contact the clinic when no information is found
* <mark style="color:blue;">Priority</mark>: High
* <mark style="color:blue;">Reason Analysis</mark>: A hallucination pass rate below 50% indicates that most responses contain content outside the knowledge base
* <mark style="color:blue;">Affected Metrics</mark>: Hallucination

AI derives suggestions from metric statistics. To see which responses contain content outside the knowledge base, review the cases individually in the case list at the bottom of the details page.

### 🏢 Enterprise Knowledge Management <a href="#use-case-enterprise-knowledge-management" id="use-case-enterprise-knowledge-management"></a>

**Metric Statistics**:

* Context recall: 45% pass rate (<mark style="color:blue;">Critical</mark>)
* Answer relevancy: 70% pass rate (<mark style="color:blue;">Warning</mark>)

**Improvement Suggestions** (illustrative):

* Recommended action: Check whether the knowledge base covers the information needed for the test questions, and adjust retrieval settings so that relevant passages can be found
* <mark style="color:blue;">Priority</mark>: High
* <mark style="color:blue;">Reason Analysis</mark>: Low context recall indicates that retrieved passages often lack the information needed to answer the question, which also affects answer relevancy
* <mark style="color:blue;">Affected Metrics</mark>: Context recall, answer relevancy

## Frequently Asked Questions <a href="#faq" id="faq"></a>

### Q1: Can I Download an Insight Report? <a href="#faq-download-insight-report" id="faq-download-insight-report"></a>

Yes, but only evaluations with the "<mark style="color:blue;">Completed</mark>" status can be exported. "Partially Completed" evaluations cannot be exported. Click "<mark style="color:blue;">Export Report</mark>" on the evaluation details page. The system generates an Excel file in the background. Once it is ready, the button changes to "<mark style="color:blue;">Download Report</mark>"; click it again to download. See [Cancel, Resume, and Export](/maiagent-user-guide/en/build/agent-ops/evaluations.md#cancel-resume-export) for the workflow. The "Suggestions and Insights" worksheet in the Excel file includes the summary, recommended actions, and metric statistics.

### Q2: Are Insight Reports Automatically Updated with Test Results? <a href="#faq-auto-update-insight-report" id="faq-auto-update-insight-report"></a>

Each new batch test generates an insight report for that evaluation; reports for previous evaluations are not overwritten. When you click "<mark style="color:blue;">Resume Evaluation</mark>" for a partially completed evaluation, the system regenerates its insight report based on the latest results.

### Q3: Can I Change the Language of an Insight Report? <a href="#faq-switch-report-language" id="faq-switch-report-language"></a>

Changing the report language is currently not supported. Reports are generated in the language of the admin interface at the time the evaluation is created. Switching the interface language later does not change the language of the report content.

### Q4: Do I Have to Follow Every Suggestion in the Insight Report? <a href="#faq-must-follow-all-suggestions" id="faq-must-follow-all-suggestions"></a>

No. You can prioritize items with a "<mark style="color:blue;">High Priority</mark>" based on your needs and available resources. The system's suggestions serve as guidance, and you can adapt them to your business needs.

### Q5: Why Did My Test Not Generate an Insight Report? <a href="#faq-no-insight-generated" id="faq-no-insight-generated"></a>

Possible reasons:

* This is a multi-turn conversation evaluation: multi-turn evaluations currently do not generate insight reports
* The evaluation has not ended yet
* The evaluation failed: no insight report is generated if no cases completed successfully
* An error occurred while generating insights: scores are still retained, but that evaluation has no insight report


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.maiagent.ai/maiagent-user-guide/en/build/agent-ops/evaluation-insights.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.