Automated Evaluation & AI Assistant Monitoring

Automate testing and real-time monitoring of AI Assistant response quality, performance metrics, and costs

Automated Evaluation

The automated evaluation feature allows you to use pre-built test datasets to automatically test AI Assistant response quality. The system sends test questions to the AI Assistant, compares actual responses with expected responses, and generates detailed evaluation reports.

Access Automated Evaluation

Go to "AgentOps" in the left sidebar, then click "Automated Testing".

The page displays all evaluation records, including evaluation name, test dataset, AI Assistant, success rate, average response time, and creation time.

Create and Run a Test

1. Ensure you have created a test dataset (refer to Test Dataset Management)

2. Click the "Create Test" button

3. Fill in the evaluation name and description, then select the test dataset and AI Assistant

4. Click "Start Evaluation" and the system will automatically execute all test cases

Evaluation execution time depends on the number of test cases. Typically, 50 test cases take approximately 2-3 minutes.

View Evaluation Results

Click on an evaluation record to view the detailed report:

The detailed report includes:

• Success Rate: Percentage of test cases that passed

• AI Insights: Automatically generated evaluation summary and improvement suggestions

• Metric Statistics: Quality scores, response relevance, and other metrics

• Test Case Details: Expected response, actual response, score, and status for each question

Success Rate Reference Benchmarks:

AI Assistant Type	Recommended Success Rate
Product Inquiry Assistant	≥ 95%
Customer Support Assistant	≥ 90%
General Conversation Assistant	≥ 80%

For more information about insight reports, refer to: Evaluation Insight Reports

Manage Evaluation Records

• Search and Filter: Filter records by test dataset, AI Assistant, or keywords

• Re-run: After modifying the Knowledge Base or AI settings, re-run tests to verify improvements

• Export: Export evaluation results in Excel format

Edit the Evaluation Name and Description

After a test is complete, you can still edit its evaluation name and description without deleting or rerunning the test. The test dataset, AI Assistant, evaluation results, and execution status remain unchanged.

For example, after Mai created a weekly customer service quality test, she noticed that the week number in the evaluation name was incorrect. She opened the edit dialog from the automated testing list, corrected the name, and added the test objective. After she saved the changes, the list immediately displayed the updated information while retaining the original test results.

1

Open the Automated Testing List

From the left menu, go to “AgentOps” → “Automated Testing”.

2

Open the Edit Dialog

Find the evaluation you want to update, then click the “Edit” icon in the “Actions” column for that row.

3

Edit and Save

Edit the “Evaluation Name” or “Description”, then click “Confirm”. The evaluation name is required and can contain up to 200 characters; the description may be left blank.

The edit dialog is prefilled with the current evaluation name and description

You can edit the name and description of evaluations that are running, completed, or failed. This action only updates the identifying information and does not rerun the test.

---

AI Assistant Monitoring

AI Assistant Monitoring provides real-time conversation operational data, allowing you to gain deep insight into the processing details, performance metrics, and quality scores of each conversation.

Access AI Assistant Monitoring

Go to "AgentOps" in the left sidebar, then click "AI Assistant Monitoring".

Monitoring Field Descriptions

Field	Description
User Input Message	The question sent by the user to the AI
Output Message	The AI Assistant's response content
AI Assistant	The name of the AI Assistant that handled the conversation
User Feedback	Thumbs up 👍 or thumbs down 👎
Faithfulness Score	Whether the response is faithful to the Knowledge Base content
Response Relevance Score	The degree of relevance between the response and the question
Response Time	The total time for the AI to generate a response
LLM Processing Reasoning Time	The time spent on LLM reasoning and response generation
Total Token Count	Total tokens consumed in the conversation (including question and answer)
LLM	The name of the language model used
User	The user who initiated the conversation

Search and Filter

• Keyword Search: Search input/output messages or usernames

• LLM Filter: Select a specific language model to compare performance across different models

• AI Assistant Filter: Select a specific assistant to track its operational status

• Time Range: Select last 7 days, 30 days, 90 days, or a custom date range

• Export: Export monitoring data in Excel or CSV format

Dashboard Metrics and Error Rate

Switch to the “Dashboard” tab to view overall service metrics and their percentage changes compared with the preceding period of the same length:

Metric	Description
Conversations	Total number of conversations during the selected period
Error Rate	Percentage of responses with system errors (see the calculation below)
Average Response Time	Average time from receiving a message to completing the response
Average TTFT	Average time from when a user sends a message until the first token appears

The dashboard also provides charts for conversation volume, response time, error-rate trends, and LLM model distribution. The “Assistant Service Ranking” compares error rates and response speeds across assistants.

Dashboard metric cards and trend charts

How Is the Error Rate Calculated?

Error rate = Number of responses with system errors ÷ Total number of responses × 100%

Only system-level errors are counted, such as model call failures and timeouts. The following are not counted as errors to avoid overestimating the error rate:

• Responses interrupted by the user (Client Interrupt)

• Normal blocking by content safeguards (Hook)

The error rate measures service stability, not response quality. If the error rate rises abnormally, review the assistant's conversation records to identify the cause, such as model issues or external tool integration failures. To assess response quality, review faithfulness and answer relevance scores together with users' thumbs-up and thumbs-down feedback.

Monitoring Best Practices

Daily Review: Check conversations from the last 24 hours to identify abnormal response times or errors

Identify Performance Bottlenecks:

• Response time > 10 seconds: Check Knowledge Base retrieval efficiency or consider a faster LLM

• High token usage: Evaluate whether the System Prompt or conversation history can be shortened

Quality Issue Tracking:

1. Use keyword search to find problematic conversations

2. Analyze root causes (insufficient Knowledge Base / AI misunderstanding / model limitations)

3. Add problem cases to the test dataset and run automated evaluations to verify fixes

---

FAQ

Q: What is the difference between Automated Evaluation and AI Assistant Monitoring?

	Automated Evaluation	AI Assistant Monitoring
Purpose	Periodic quality testing	Real-time operational monitoring
Data Source	Predefined test datasets	Actual user conversations
Key Metrics	Success rate, response time	Performance, cost, quality scores
Best For	Quality verification, regression testing	Daily monitoring, troubleshooting

Q: How often should evaluations be run?

Recommendation: Core features weekly, full tests monthly, and immediately after major updates.

Q: Do evaluations affect actual users?

No. Automated evaluations run in an isolated environment and do not interfere with actual user conversations.

Q: How long is monitoring data retained?

The default retention period is 90 days. You can periodically export important data for long-term storage.