Automated Evaluation and AI Assistant Monitoring

Automate testing and real-time monitoring of AI Assistant response quality, performance metrics, and costs

Automated Evaluation

The automated evaluation feature allows you to use pre-built test datasets to automatically test AI Assistant response quality. The system sends test questions to the AI Assistant, compares actual responses with expected responses, and generates detailed evaluation reports.

Access Automated Evaluation

Go to "AgentOps" in the left sidebar, then click "Automated Testing".

The page displays all evaluation records, including evaluation name, test dataset, AI Assistant, success rate, average response time, and creation time.

Create and Run a Test

1. Ensure you have created a test dataset (refer to Test Dataset Management)

2. Click the "Create Test" button

3. Fill in the evaluation name and description, then select the test dataset and AI Assistant

4. Click "Start Evaluation" and the system will automatically execute all test cases

Evaluation execution time depends on the number of test cases. Typically, 50 test cases take approximately 2-3 minutes.

Use Custom Evaluation Metrics

In addition to built-in metrics, you can define your team's own scoring criteria using natural language. For example, Mai, a customer service manager, wants to verify whether the customer service assistant remains polite and empathetic. She creates a "Customer Service Tone" metric, enters the evaluation criteria, and sets a passing threshold. After the evaluation is complete, she can view each case's "Customer Service Tone" score in the results and identify responses that need adjustment.

1

When creating an evaluation, complete the basic settings, including the name, test dataset, AI Assistant, and evaluation model.

2

In the "Custom Metrics" section, click "Add Custom Metric", then enter an easily identifiable metric name and evaluation criteria.

3

Adjust the passing threshold according to your team's quality standards, then click "Confirm" to start the evaluation.

After switching to multi-turn conversations, you can set the maximum number of turns, three evaluation metrics, and the simulation language

Set evaluation criteria for customer service tone using natural language

The system compares names without regard to letter case. A custom metric cannot duplicate any of the 11 built-in metrics. If a name conflicts, use a name that expresses your team's evaluation objective.

Evaluate Multi-turn Conversations

Multi-turn evaluation is ideal for testing tasks that require back-and-forth confirmation. For example, Mei, a quality manager, wants the assistant to ask for the order number first, confirm the reason for the return, and then explain the next steps. She creates a multi-turn test case, enters the scenario and expected result, and runs the evaluation. When it is complete, she can expand the conversation transcript to review every response and the Knowledge Base content retrieved during each turn, instead of looking only at the final answer.

1

First, go to "Test Datasets" and open the target test dataset. Add a scenario, expected result, and simulated user persona to the multi-turn test case.

2

Return to "Automated Testing", click "Create Evaluation", and switch to "Multi-turn Conversation".

3

Select the test dataset, AI Assistant, and evaluation model. Set the maximum number of turns, evaluation metrics, and simulation language, then start the evaluation. At least one evaluation metric must be enabled.

4

After the evaluation is complete, open the results and select an individual case to expand its details. Review the metric scores, conversation transcript, and "Retrieved Content" for each turn. If an individual case fails, its error is displayed separately without hiding the results of other cases.

Expand a case to review the assistant's response and the Knowledge Base content used in each turn

If you no longer need an ongoing multi-turn evaluation, click "Cancel Evaluation" in the evaluation details and confirm. The system stops processing subsequent cases after the current case is complete. Scores and transcripts for completed cases are retained, the status is shown as "Partially Completed", and unprocessed cases are shown as "Pending Evaluation".

Completed cases are retained after cancellation, while unprocessed cases remain pending evaluation

View Evaluation Results

Click on an evaluation record to view the detailed report:

The detailed report includes:

• Success Rate: Percentage of test cases that passed

• AI Insights: Automatically generated evaluation summary and improvement suggestions

• Metric Statistics: Quality scores, response relevance, and other metrics

• Test Case Details: Expected response, actual response, score, and status for each question

Success Rate Reference Benchmarks:

AI Assistant Type	Recommended Success Rate
Product Inquiry Assistant	≥ 95%
Customer Support Assistant	≥ 90%
General Conversation Assistant	≥ 80%

For more information about insight reports, refer to: Evaluation Insight Reports

Manage Evaluation Records

• Search and Filter: Filter records by test dataset, AI Assistant, or keywords

• Re-run: After modifying the Knowledge Base or AI settings, re-run tests to verify improvements

• Export: Export evaluation results in Excel format

Edit the Evaluation Name and Description

After a test is complete, you can still edit its evaluation name and description without deleting or rerunning the test. The test dataset, AI Assistant, evaluation results, and execution status remain unchanged.

For example, after Mai created a weekly customer service quality test, she noticed that the week number in the evaluation name was incorrect. She opened the edit dialog from the automated testing list, corrected the name, and added the test objective. After she saved the changes, the list immediately displayed the updated information while retaining the original test results.

1

Open the Automated Testing List

From the left menu, go to “AgentOps” → “Automated Testing”.

2

Open the Edit Dialog

Find the evaluation you want to update, then click the “Edit” icon in the “Actions” column for that row.

3

Edit and Save

Edit the “Evaluation Name” or “Description”, then click “Confirm”. The evaluation name is required and can contain up to 200 characters; the description may be left blank.

The edit dialog is prefilled with the current evaluation name and description

You can edit the name and description of evaluations that are running, completed, or failed. This action only updates the identifying information and does not rerun the test.

---

AI Assistant Monitoring

AI Assistant Monitoring provides real-time conversation operational data, allowing you to gain deep insight into the processing details, performance metrics, and quality scores of each conversation.

Access AI Assistant Monitoring

Go to "AgentOps" in the left sidebar, then click "AI Assistant Monitoring".

Monitoring Field Descriptions

Field	Description
User Input Message	The question sent by the user to the AI
Output Message	The AI Assistant's response content
AI Assistant	The name of the AI Assistant that handled the conversation
User Feedback	Thumbs up 👍 or thumbs down 👎
Faithfulness Score	Whether the response is faithful to the Knowledge Base content
Response Relevance Score	The degree of relevance between the response and the question
Response Time	The total time for the AI to generate a response
LLM Processing Reasoning Time	The time spent on LLM reasoning and response generation
Total Token Count	Total tokens consumed in the conversation (including question and answer)
LLM	The name of the language model used
User	The user who initiated the conversation
Source	Whether the record is an Interactive Conversation (a user asks a question) or a Scheduled Run (automatically triggered by an agent schedule)
Member	The organization member who initiated the conversation; displays “—” when no member is associated with it (for example, a scheduled run)

View Image Search Usage

When an AI Assistant uses image search in the Knowledge Base, the system separately records image embedding usage. You can review an individual conversation record to see which resources were used for that response:

1

Go to AgentOps → AI Assistant Monitoring, then switch to Conversation Records.

2

Find the response you want to review, then click Details on the far right.

3

Scroll down to view the Token and Credit breakdown. Standard text embedding is listed under Embedding. If the response used image search, Image Embedding is also displayed, and the Token and Credit totals include this item.

Usage breakdown for an individual conversation; “Image Embedding” appears only when the response actually used image search

If Image Embedding is not displayed, it does not mean the record is missing. This item appears only when the response actually incurs image embedding usage. When image search is not used, the screen continues to show the standard embedding breakdown.

For example, Mai's customer service manager asks the product assistant to identify the appearance of a component from catalog images. After testing, she opens the conversation details and sees the image embedding usage and corresponding Credits. This allows her to distinguish the cost of image search from standard text retrieval before deciding whether to adjust the image catalog or assistant settings.

Search and Filter

• Keyword Search: Search input/output messages or usernames

• LLM Filter: Select a specific language model to compare performance across different models

• AI Assistant Filter: Select a specific assistant to track its operational status

• Time Range: Select last 7 days, 30 days, 90 days, or a custom date range

• Export: Export monitoring data in Excel or CSV format

• Source Filter: Show only Interactive Conversations or Scheduled Runs

• Member Filter: Select one or more organization members and show only conversations they initiated

Filter Conversation Records by Member and Source

The filter bar in Conversation Records includes All Sources and All Members dropdowns by default. The list also has Source and Member columns, so you can see who asked a question and whether a person or a schedule initiated the run.

For example, a customer service manager wants to understand how a new colleague has used the internal AI Assistant over the past few weeks. She opens Conversation Records, sets Source to Interactive Conversation to exclude scheduled records, and selects the colleague under All Members. The list now shows only that colleague's conversations. After reviewing them, she clicks Export to share the same list with the training team without asking an engineer to retrieve the data.

1. Filter by Member

Go to AgentOps → AI Assistant Monitoring in the left menu, switch to Conversation Records, and click All Members. Select the members you want to view; you can select multiple members or search by keyword.

Select members from the All Members dropdown (member names and email addresses are obscured)

The list then shows only conversations initiated by those members. You can combine this filter with keyword, AI Assistant, error status, date, and other filters. The Member column on the right shows the member associated with each record.

After filtering for one member, the Member column on the right shows the associated member for each record (obscured)

2. Filter Scheduled Runs by Source

Tasks run automatically through Agent Schedules also appear in Conversation Records. Their Source is labeled Scheduled Run, while records from users' questions are labeled Interactive Conversation. Click All Sources to show only one of these types.

When Scheduled Run is selected as the source, the member filter and Export button are disabled; hovering over Export shows the reason

Limitations when “Scheduled Run” is selected

• Scheduled runs have no associated member or user feedback, so the Member and Rating filters are disabled and cleared.

• Export currently supports only interactive conversations. When the source is Scheduled Run, the button is disabled. Clear the source filter before exporting.

• Dashboard statistics include only interactive conversations. View scheduled runs in Conversation Records.

Click Details on a scheduled run record. Basic Information shows the Schedule Name, Source, and Status for that run. Identification Information also shows the Schedule ID, so you can identify which schedule created the record.

Basic information for a scheduled run record shows the schedule name, Source as Scheduled Run, and Status as Success

Export Requests

After you click Export in Conversation Records, the system creates an Excel file in the background using the current filters (including member and source) and displays “Export request created, processing...”. Go to AgentOps → Export Requests in the left menu to view each export's Status, the User who requested it, Record Count, File Size, Processing Time, and creation or completion time. Once the status is Completed, you can download it from the Actions column.

The export requests list shows status, requesting user (obscured), record count, and file size

When multiple administrators share an organization, use the User column to identify who created each export before deciding which file to download or delete.

Dashboard Metrics and Error Rate

Switch to the “Dashboard” tab to view overall service metrics and their percentage changes compared with the preceding period of the same length:

Metric	Description
Conversations	The number of responses from AI Assistants during the selected period. Each AI response in a conversation counts once; responses handled by human agents and automatically scheduled runs are excluded (see the definition below for details)
Error Rate	Percentage of responses with system errors (see the calculation below)
Average Response Time	Average time from receiving a message to completing the response
Average TTFT	Average time from when a user sends a message until the first token appears

The dashboard also provides charts for conversation volume, response time, error-rate trends, and LLM model distribution. The “Assistant Service Ranking” compares error rates and response speeds across assistants.

Dashboard metric cards and trend charts

How Is the Conversation Count Calculated?

The dashboard's "Conversations" metric and conversation volume trend count each response from an AI Assistant, rather than the number of conversations opened:

Scenario	Counted?	Count
A user enters the service but does not send a message	No	0
A user sends 1 message and the AI responds once	Yes	1
In the same conversation, a user asks 5 questions and the AI responds 5 times	Yes	5
A user sends a message, but a human agent takes over and responds	No	0
A user sends a message, but an error occurs while the AI is responding	Yes	1
An automatically scheduled task runs	No	0

To see how many conversations were opened, view "Conversation Count" on the AI Assistant's "Usage Analytics" page instead. It counts 1 when a user sends the first message, regardless of how many turns the conversation contains, and includes conversations handled by human agents. These metrics have different definitions, so the dashboard's "Conversations" count is generally higher than the "Conversation Count" in "Usage Analytics." See Usage Analytics for details.

How Is the Error Rate Calculated?

Error rate = Number of responses with system errors ÷ Total number of responses × 100%

Only system-level errors are counted, such as model call failures and timeouts. The following are not counted as errors to avoid overestimating the error rate:

• Responses interrupted by the user (Client Interrupt)

• Normal blocking by content safeguards (Hook)

The error rate measures service stability, not response quality. If the error rate rises abnormally, review the assistant's conversation records to identify the cause, such as model issues or external tool integration failures. To assess response quality, review faithfulness and answer relevance scores together with users' thumbs-up and thumbs-down feedback.

Monitoring Best Practices

Daily Review: Check conversations from the last 24 hours to identify abnormal response times or errors

Identify Performance Bottlenecks:

• Response time > 10 seconds: Check Knowledge Base retrieval efficiency or consider a faster LLM

• High token usage: Evaluate whether the System Prompt or conversation history can be shortened

Quality Issue Tracking:

1. Use keyword search to find problematic conversations

2. Analyze root causes (insufficient Knowledge Base / AI misunderstanding / model limitations)

3. Add problem cases to the test dataset and run automated evaluations to verify fixes

---

FAQ

Q: What is the difference between Automated Evaluation and AI Assistant Monitoring?

	Automated Evaluation	AI Assistant Monitoring
Purpose	Periodic quality testing	Real-time operational monitoring
Data Source	Predefined test datasets	Actual user conversations
Key Metrics	Success rate, response time	Performance, cost, quality scores
Best For	Quality verification, regression testing	Daily monitoring, troubleshooting

Q: How often should evaluations be run?

Recommendation: Core features weekly, full tests monthly, and immediately after major updates.

Q: Do evaluations affect actual users?

No. Automated evaluations run in an isolated environment and do not interfere with actual user conversations.

Q: How long is monitoring data retained?

The default retention period is 90 days. You can periodically export important data for long-term storage.