Evaluation Insights Report
This guide explains how to use the Evaluation Insights feature to quickly understand your AI assistant's test results and receive system-generated improvement recommendations.
What Are Evaluation Insights?
Evaluation Insights is an extension of AgentOps automated testing. After completing a batch test, the system not only displays metric scores but also automatically analyzes test results and generates an "Insights Report."
This report tells you:
How the test results performed overall
Where the main issues are
How to make specific improvements
Which items to prioritize
Think of it as having a senior consultant review your test report and provide you with an improvement recommendation document.
Core Advantages of Evaluation Insights
📊 Automated Analysis
No need to manually interpret complex scoring data — the system automatically identifies problem patterns and compiles them into an easy-to-understand report.
🎯 Priority Recommendations
The system recommends which items to address first based on issue severity and scope of impact, making your improvement efforts more efficient.
💡 Specific Improvement Directions
It doesn't just tell you "there's a problem" — it also provides concrete improvement suggestions, such as which settings to adjust and what content to supplement.
🌐 Multi-Language Support
Insights reports support multiple languages including Traditional Chinese, Simplified Chinese, and English. You can choose the language that best suits your team's needs.
How to Use the Evaluation Insights Feature
Step 1: Complete a Batch Test
Go to "AgentOps" → "Automated Testing"
Run a batch test (refer to AgentOps User Guide)
Wait for the test to complete
Step 2: View Evaluation Results
Go to the test result detail page
Review each scoring metric
Find the "Insights Report" section
Step 3: Select the Report Language
In the Insights Report section, you can choose the display language for the report:
Traditional Chinese: Suitable for Taiwan-based teams
Simplified Chinese: Suitable for Mainland China-based teams
English: Suitable for international teams or situations requiring English reports
After selecting a language, the system automatically generates the insights report in the corresponding language.
Step 4: Read the Insights Report
The insights report typically includes the following content:
1. Overall Evaluation Summary
Test pass rate
Key strengths
Areas needing improvement
2. Issue Analysis
Answer relevance issues
Safety risks (bias, offensiveness, hallucinations)
Response speed issues
3. Improvement Suggestions
Priority items (labeled as "High Priority," "Medium Priority," or "Low Priority")
Specific improvement steps
Expected improvement outcomes
Step 5: Make Improvements Based on Suggestions
Address issues in order of the priority listed in the report
Adjust AI assistant settings based on the specific improvement steps
After making adjustments, run the test again to verify the results
How Evaluation Insights Are Calculated
The system considers the following factors to generate the insights report:
Scoring Metrics: The levels and distribution of each score
Failed Cases: The number and reasons for failed cases
Problem Patterns: The frequency of recurring similar issues
Best Practices: Industry standards and optimization recommendations
Practical Use Cases
🏥 Medical Clinic Customer Service
Test Results:
Pass rate: 72%
Hallucination score: High
Insights Report Suggestion:
"The test found that some responses contained unverified medical advice. It is recommended to explicitly restrict the role instructions: 'Do not provide diagnostic or treatment advice; only provide appointment scheduling and clinic information.'"
🏢 Enterprise Internal Knowledge Management
Test Results:
Pass rate: 88%
Answer relevance: Low
Insights Report Suggestion:
"The test found that some responses were too brief or did not fully answer the question. It is recommended to add more detailed operating steps and examples to the knowledge base."
FAQ
Q1: Can the insights report be downloaded?
Currently, the insights report can only be viewed within the system. You can copy the report content and compile it into an internal document.
Q2: Does the insights report update automatically with new test results?
Yes, each time a new batch test is run, the system generates a new insights report reflecting the latest test results.
Q3: Can I switch the language of the insights report?
Yes, you can switch the report language at any time, and the system will regenerate the report in the selected language.
Q4: Do I have to implement all the suggestions in the insights report?
Not necessarily. You can prioritize "High Priority" items based on actual needs and available resources. The system's suggestions are reference directions — you can adjust them according to your business requirements.
Q5: Why didn't my test generate an insights report?
Possible reasons:
Too few test cases (at least 10 are recommended)
The test has not finished yet
The system is still processing (please wait 1-2 minutes)
Last updated
Was this helpful?