AgentOps Overview

AgentOps provides comprehensive AI Assistant operation management and quality control tools

What is AgentOps?

AgentOps (Agent Operations) is the AI Assistant operation management module provided by the MaiAgent platform, focusing on testing, evaluating, and monitoring the actual performance of AI Assistants. With AgentOps, you can systematically manage AI Assistant quality and ensure consistently excellent service experiences.

Why Do You Need AgentOps?

Building an AI Assistant is just the first step — continuous monitoring and optimization are essential for long-term success. AgentOps helps you:

Ensure Consistent Quality

• Establish standardized testing mechanisms

• Track AI Assistant performance trends

• Identify and fix quality issues promptly

Improve Operational Efficiency

• Automate testing processes to save manpower

• Quickly identify performance bottlenecks

• Optimize system resources and costs

Data-Driven Decision Making

• Quantify AI Assistant performance metrics

• Compare the effectiveness of different settings or models

• Develop optimization strategies based on actual data

AgentOps Core Features

1. Test Dataset Management

Create and manage test case collections to verify AI Assistant response quality.

Key Features:

• Create multiple test datasets

• Manage test cases and expected responses

• Support search and categorization

• Team collaboration on test case maintenance

Use Cases:

• Verify response accuracy after Knowledge Base updates

• Establish standard testing workflows

• Collect frequently asked questions as test benchmarks

For details: Test Dataset Management

2. Automated Evaluation

Use test datasets to run automated evaluations and generate detailed quality reports.

Key Features:

• One-click batch testing

• Calculate success rates and response times

• Generate detailed pass/fail reports

• Compare performance across different periods or AI Assistants

Use Cases:

• Regular quality checks

• Regression testing before and after updates

• Compare effectiveness of different AI Assistants or models

For details: Automated Evaluation

3. AI Assistant Monitoring

Monitor AI Assistant operations in real time and analyze every conversation in detail.

Key Features:

• View input/output message content

• Track reply time and processing time

• Monitor token usage and costs

• Analyze quality scores and user feedback

Use Cases:

• Real-time system operation monitoring

• Identify performance issues and anomalies

• Track costs and usage

• In-depth analysis of individual conversations

For details: AI Assistant Monitoring

4. Evaluation Insight Report

After batch testing is complete, the system automatically analyzes the results and generates an improvement recommendations report — no need to manually interpret each scoring data point.

Key Features:

• Automatically identify problem patterns in test results

• Suggest handling priorities based on severity and scope of impact

• Provide concrete improvement directions, not just scores

• Support multilingual reports

Use Cases:

• Success rate falls below expectations and you need to know where the problems are

• A single test run produces many failed cases and you need to decide which to fix first

• You need to explain the current quality status and improvement plan to your team or management

For details: Evaluation Insight Report

5. Tool Execution Records

Track the complete history of AI Assistant external tool invocations and quickly pinpoint the cause of tool execution failures.

Key Features:

• Record input parameters, execution results, duration, and success/failure status for each tool invocation

• Retain detailed error messages to help pinpoint failure causes

• Track usage count and success rate for each tool

• Cover API tools, Text-to-SQL tools, crawler tools, and custom tools

Use Cases:

• AI Assistant cannot answer questions that require querying external systems

• Tools fail intermittently and you need to determine whether it's a timeout, authentication, or parameter issue

• Evaluating whether a tool needs configuration adjustments or replacement

For details: Tool Execution Records

AgentOps Workflow

Sub-feature Relationship Map

AgentOps has five sub-features divided into two tracks: Pre-launch uses test datasets and automated evaluation to establish quality benchmarks, while Post-launch uses monitoring and tool records to observe real operations. The two tracks form a cycle by "feeding issues discovered in production back into the test datasets."

1. Test Dataset Management

Create reusable test case collections — the starting point of the entire testing chain.

2. Automated Evaluation

Batch test AI Assistants using test datasets, producing success rates and average response times.

3. Evaluation Insight Report

Automatically analyze evaluation results into problem patterns and improvement suggestions, ranked by priority.

4. AI Assistant Monitoring

Inspect real conversations one by one after launch — inputs, outputs, duration, tokens, and scores.

5. Tool Execution Records

Track AI Assistant external tool invocation parameters, results, and failure causes.

Sequence and dependencies of each sub-feature:

Sub-feature	Stage	Prerequisites	Output	Next Step
Test Dataset Management	Pre-launch	Common questions and expected responses	Reusable test case collections	Hand off to Automated Evaluation
Automated Evaluation	Pre-launch	Test datasets, AI Assistant to test	Success rate, average response time, per-case pass/fail	Hand off to Evaluation Insight Report for analysis
Evaluation Insight Report	Pre-launch	A completed automated evaluation	Problem patterns, priorities, concrete improvement directions	Adjust Knowledge Base or role instructions based on suggestions, then test again
AI Assistant Monitoring	Post-launch	Real conversations from a live AI Assistant	Input/output, duration, token usage, scores for each conversation	Feed discovered issues back into test datasets
Tool Execution Records	Post-launch	AI Assistant with configured tools	Tool invocation parameters, results, duration, and error messages	Fix tool settings, or add a test case

Two commonly confused points:

• "Automated Evaluation" tests questions from the test datasets, while "AI Assistant Monitoring" examines conversations actual users have had — the former is a pre-launch quality gate, the latter is post-launch observation.

• "Evaluation Insight Report" is not a standalone feature — it is an extension of Automated Evaluation: you must first complete a batch test before an insight report is available.

Standard Quality Management Flow

1. Create Test Datasets
   ↓
   Collect common questions and expected responses
   Build structured test cases

2. Run Automated Evaluations
   ↓
   Run tests regularly or after updates
   Generate success rate and quality reports

3. Analyze Evaluation Results
   ↓
   Identify failed cases
   Find issues that need improvement

4. Optimize AI Assistant
   ↓
   Update Knowledge Base content
   Adjust AI settings and prompts

5. Monitor Actual Operations
   ↓
   Use AI Assistant Monitoring to track real conversations
   Continuously ensure quality stability

6. Continuous Improvement
   ↓
   Discover new issues from monitoring data
   Update test datasets
   Repeat the evaluation and optimization cycle

Quick Start Steps

Week 1: Build Foundational Tests

1. Collect 20-30 frequently asked questions

2. Create your first test dataset

3. Run the initial automated evaluation

4. Record the current success rate as a baseline

Week 2: Continuous Optimization

1. Analyze failed test cases

2. Update the Knowledge Base or adjust AI settings

3. Re-run evaluations to verify improvements

4. Start using AI Assistant Monitoring to observe real conversations

Week 3: Build Habits

1. Run automated evaluations once a week

2. Check monitoring data daily to identify anomalies

3. Continuously add test cases to cover more scenarios

4. Create quality reports to track long-term trends

AgentOps Integration with Other Features

AgentOps + Knowledge Base Management

Integration Use Cases:

• Run automated evaluations immediately after Knowledge Base updates

• Supplement Knowledge Base content when monitoring data reveals knowledge gaps

• Create test cases based on real conversations to optimize the Knowledge Base

Related documentation: Build an Enterprise Knowledge Base

AgentOps + Usage Analytics

Complementary Analysis:

• Usage Analytics: Provides overall trends and statistics (conversation volume, satisfaction, etc.)

• AgentOps: Provides in-depth quality analysis and individual conversation details

Combined use enables you to:

• Detect anomalies from overall trends

• Identify root causes from monitoring details

• Comprehensively evaluate AI Assistant performance

Related documentation: Usage Analytics

AgentOps + Response Quality Control

Integration Use Cases:

• AgentOps provides technical quality monitoring

• Response Quality Control provides manual review and annotation

• Together they form a complete quality assurance system

Related documentation: Response Quality Control

Best Practices

1. Establish a Regular Evaluation Schedule

Daily Check (5 minutes):

• Review AI Assistant Monitoring to identify anomalies

Weekly Evaluation (30 minutes):

• Run core test datasets

• Check whether success rates remain stable

• Update or add test cases

Monthly Analysis (2 hours):

• Run a full evaluation

• Export monitoring data for in-depth analysis

• Produce quality reports

• Plan optimization goals for the next month

2. Set Quality Baselines

Set reasonable quality targets for your AI Assistant:

Metric	Baseline	Target	Description
Automated Evaluation Success Rate	≥ 85%	≥ 95%	Standard test case pass rate
Average Response Time	< 5 seconds	< 3 seconds	User-perceived response speed
User Satisfaction	≥ 80%	≥ 90%	Like / total feedback ratio
Token Usage	-	-	Set limits based on budget

3. Establish an Issue Tracking Workflow

When quality issues are discovered:

Issue Discovered
↓
Record in test dataset (prevent recurrence)
↓
Analyze root cause (Knowledge Base / AI settings / model capability)
↓
Implement improvements
↓
Run evaluations to verify effectiveness
↓
Continue monitoring to confirm stability

4. Team Collaboration

Establish AgentOps responsibility assignments:

AI Manager:

• Overall quality monitoring

• Develop optimization strategies

• Regular evaluations and reporting

Knowledge Base Administrator:

• Update the Knowledge Base based on evaluation results

• Supplement missing knowledge content

Technical Staff:

• Analyze performance issues

• Optimize system settings

• Handle technical anomalies

5. Continuous Improvement Cycle

The core of AgentOps is continuous improvement:

1. Measure: Obtain data through evaluations and monitoring

2. Analyze: Identify issues and improvement opportunities

3. Act: Implement optimization measures

4. Verify: Confirm improvement effectiveness

5. Standardize: Incorporate successful practices into workflows

FAQ

Q1: What is the difference between AgentOps and "Usage Analytics"?

Usage Analytics:

• Overall trends and statistics (conversation volume, word count, satisfaction)

• Suitable for understanding AI Assistant usage overview

• Focuses on business metrics

AgentOps:

• In-depth quality analysis and individual conversation details

• Suitable for technical quality management

• Focuses on technical metrics (success rate, response time, cost)

The two are complementary — we recommend using them together.

Q2: How often should evaluations be run?

Recommended frequency:

• Core features: Once a week

• Full tests: Once a month

• After major updates: Run immediately

• When issues are found: Test and verify at any time

Adjust based on your AI Assistant's importance and update frequency.

Q3: How many test cases should a test dataset contain?

Recommended amounts:

• Minimum: 20 test cases (covering core features)

• Recommended: 50-100 test cases (balancing coverage and efficiency)

• Comprehensive: 200+ test cases (for large or critical systems)

Start with core features and expand gradually.

Q4: What success rate is considered acceptable?

It depends on the AI Assistant's purpose:

• High-risk applications (finance, healthcare): ≥ 95%

• Customer support: ≥ 90%

• General conversation: ≥ 85%

• Experimental features: ≥ 80%

The key is to establish a baseline and continuously improve.

Q5: Does AgentOps affect real users?

No. AgentOps testing and monitoring run in isolated environments or in the background, without interfering with the actual user experience.

Q6: Can I create multiple test sets for different business areas?

Yes. We recommend creating multiple test datasets for different business areas or features:

• Product inquiry test set

• Order processing test set

• Technical support test set

• ...and more

This allows you to track quality performance separately for each area.

---

Get Started with AgentOps

Ready to start using AgentOps? We recommend following this order:

1. Create Test Datasets: Collect common questions and create your first test set

2. Run Automated Evaluations: Test your AI Assistant and review success rates and response times

3. Read the Evaluation Insight Report: Understand where the problems are and what to prioritize

4. Use AI Assistant Monitoring: Gain in-depth understanding of actual operations after launch

5. View Tool Execution Records: If your AI Assistant has configured tools, track whether tool invocations are working properly

With AgentOps, you can establish a systematic quality management workflow to ensure your AI Assistant consistently delivers excellent service.