Evaluation Insights Report

This guide explains how to use the Evaluation Insights feature to quickly understand your AI assistant's test results and receive system-generated improvement recommendations.

What Are Evaluation Insights?

Evaluation Insights is an extension of AgentOps automated testing. After completing a batch test, the system not only displays metric scores but also automatically analyzes test results and generates an "Insights Report."

This report tells you:

• How the test results performed overall

• Where the main issues are

• How to make specific improvements

• Which items to prioritize

Think of it as having a senior consultant review your test report and provide you with an improvement recommendation document.

Core Advantages of Evaluation Insights

📊 Automated Analysis

No need to manually interpret complex scoring data — the system automatically identifies problem patterns and compiles them into an easy-to-understand report.

🎯 Priority Recommendations

The system recommends which items to address first based on issue severity and scope of impact, making your improvement efforts more efficient.

💡 Specific Improvement Directions

It doesn't just tell you "there's a problem" — it also provides concrete improvement suggestions, such as which settings to adjust and what content to supplement.

🌐 Multi-Language Support

Insights reports support multiple languages including Traditional Chinese, Simplified Chinese, and English. You can choose the language that best suits your team's needs.

How to Use the Evaluation Insights Feature

Step 1: Complete a Batch Test

1. Go to "AgentOps" → "Automated Testing"

2. Run a batch test (refer to AgentOps User Guide)

3. Wait for the test to complete

Step 2: View Evaluation Results

1. Go to the test result detail page

2. Review each scoring metric

3. Find the "Insights Report" section

Step 3: Select the Report Language

In the Insights Report section, you can choose the display language for the report:

• Traditional Chinese: Suitable for Taiwan-based teams

• Simplified Chinese: Suitable for Mainland China-based teams

• English: Suitable for international teams or situations requiring English reports

After selecting a language, the system automatically generates the insights report in the corresponding language.

Step 4: Read the Insights Report

The insights report typically includes the following content:

1. Overall Evaluation Summary

• Test pass rate

• Key strengths

• Areas needing improvement

2. Issue Analysis

• Answer relevance issues

• Safety risks (bias, offensiveness, hallucinations)

• Response speed issues

3. Improvement Suggestions

• Priority items (labeled as "High Priority," "Medium Priority," or "Low Priority")

• Specific improvement steps

• Expected improvement outcomes

Step 5: Make Improvements Based on Suggestions

1. Address issues in order of the priority listed in the report

2. Adjust AI assistant settings based on the specific improvement steps

3. After making adjustments, run the test again to verify the results

How Evaluation Insights Are Calculated

The system considers the following factors to generate the insights report:

• Scoring Metrics: The levels and distribution of each score

• Failed Cases: The number and reasons for failed cases

• Problem Patterns: The frequency of recurring similar issues

• Best Practices: Industry standards and optimization recommendations

Generating the insights report requires some processing time, typically completed within 1-2 minutes after the test finishes.

Practical Use Cases

🏥 Medical Clinic Customer Service

Test Results:

• Pass rate: 72%

• Hallucination score: High

Insights Report Suggestion:

"The test found that some responses contained unverified medical advice. It is recommended to explicitly restrict the role instructions: 'Do not provide diagnostic or treatment advice; only provide appointment scheduling and clinic information.'"

🏢 Enterprise Internal Knowledge Management

Test Results:

• Pass rate: 88%

• Answer relevance: Low

Insights Report Suggestion:

"The test found that some responses were too brief or did not fully answer the question. It is recommended to add more detailed operating steps and examples to the knowledge base."

FAQ

Q1: Can the insights report be downloaded?

Currently, the insights report can only be viewed within the system. You can copy the report content and compile it into an internal document.

Q2: Does the insights report update automatically with new test results?

Yes, each time a new batch test is run, the system generates a new insights report reflecting the latest test results.

Q3: Can I switch the language of the insights report?

Yes, you can switch the report language at any time, and the system will regenerate the report in the selected language.

Q4: Do I have to implement all the suggestions in the insights report?

Not necessarily. You can prioritize "High Priority" items based on actual needs and available resources. The system's suggestions are reference directions — you can adjust them according to your business requirements.

Q5: Why didn't my test generate an insights report?

Possible reasons:

• Too few test cases (at least 10 are recommended)

• The test has not finished yet

• The system is still processing (please wait 1-2 minutes)