Automated Evaluation & AI Assistant Monitoring

Automate testing and real-time monitoring of AI Assistant response quality, performance metrics, and costs

Automated Evaluation

The automated evaluation feature allows you to use pre-built test datasets to automatically test AI Assistant response quality. The system sends test questions to the AI Assistant, compares actual responses with expected responses, and generates detailed evaluation reports.

Access Automated Evaluation

Go to "AgentOps" in the left sidebar, then click "Automated Testing".

The page displays all evaluation records, including evaluation name, test dataset, AI Assistant, success rate, average response time, and creation time.

Create and Run a Test

1. Ensure you have created a test dataset (refer to Test Dataset Management)

2. Click the "Create Test" button

3. Fill in the evaluation name and description, then select the test dataset and AI Assistant

4. Click "Start Evaluation" and the system will automatically execute all test cases

Evaluation execution time depends on the number of test cases. Typically, 50 test cases take approximately 2-3 minutes.

View Evaluation Results

Click on an evaluation record to view the detailed report:

The detailed report includes:

• Success Rate: Percentage of test cases that passed

• AI Insights: Automatically generated evaluation summary and improvement suggestions

• Metric Statistics: Quality scores, response relevance, and other metrics

• Test Case Details: Expected response, actual response, score, and status for each question

Success Rate Reference Benchmarks:

AI Assistant Type	Recommended Success Rate
Product Inquiry Assistant	≥ 95%
Customer Support Assistant	≥ 90%
General Conversation Assistant	≥ 80%

For more information about insight reports, refer to: Evaluation Insight Reports

Manage Evaluation Records

• Search and Filter: Filter records by test dataset, AI Assistant, or keywords

• Re-run: After modifying the Knowledge Base or AI settings, re-run tests to verify improvements

• Export: Export evaluation results in Excel format

---

AI Assistant Monitoring

AI Assistant Monitoring provides real-time conversation operational data, allowing you to gain deep insight into the processing details, performance metrics, and quality scores of each conversation.

Access AI Assistant Monitoring

Go to "AgentOps" in the left sidebar, then click "AI Assistant Monitoring".

Monitoring Field Descriptions

Field	Description
User Input Message	The question sent by the user to the AI
Output Message	The AI Assistant's response content
AI Assistant	The name of the AI Assistant that handled the conversation
User Feedback	Thumbs up 👍 or thumbs down 👎
Faithfulness Score	Whether the response is faithful to the Knowledge Base content
Response Relevance Score	The degree of relevance between the response and the question
Response Time	The total time for the AI to generate a response
LLM Processing Reasoning Time	The time spent on LLM reasoning and response generation
Total Token Count	Total tokens consumed in the conversation (including question and answer)
LLM	The name of the language model used
User	The user who initiated the conversation

Search and Filter

• Keyword Search: Search input/output messages or usernames

• LLM Filter: Select a specific language model to compare performance across different models

• AI Assistant Filter: Select a specific assistant to track its operational status

• Time Range: Select last 7 days, 30 days, 90 days, or a custom date range

• Export: Export monitoring data in Excel or CSV format

Monitoring Best Practices

Daily Review: Check conversations from the last 24 hours to identify abnormal response times or errors

Identify Performance Bottlenecks:

• Response time > 10 seconds: Check Knowledge Base retrieval efficiency or consider a faster LLM

• High token usage: Evaluate whether the System Prompt or conversation history can be shortened

Quality Issue Tracking:

1. Use keyword search to find problematic conversations

2. Analyze root causes (insufficient Knowledge Base / AI misunderstanding / model limitations)

3. Add problem cases to the test dataset and run automated evaluations to verify fixes

---

FAQ

Q: What is the difference between Automated Evaluation and AI Assistant Monitoring?

	Automated Evaluation	AI Assistant Monitoring
Purpose	Periodic quality testing	Real-time operational monitoring
Data Source	Predefined test datasets	Actual user conversations
Key Metrics	Success rate, response time	Performance, cost, quality scores
Best For	Quality verification, regression testing	Daily monitoring, troubleshooting

Q: How often should evaluations be run?

Recommendation: Core features weekly, full tests monthly, and immediately after major updates.

Q: Do evaluations affect actual users?

No. Automated evaluations run in an isolated environment and do not interfere with actual user conversations.

Q: How long is monitoring data retained?

The default retention period is 90 days. You can periodically export important data for long-term storage.