AgentOps Overview

AgentOps provides comprehensive AI Assistant operation management and quality control tools

What is AgentOps?

AgentOps (Agent Operations) is the AI Assistant operation management module provided by the MaiAgent platform, focusing on testing, evaluating, and monitoring the actual performance of AI Assistants. With AgentOps, you can systematically manage AI Assistant quality and ensure consistently excellent service experiences.

Why Do You Need AgentOps?

Building an AI Assistant is just the first step — continuous monitoring and optimization are essential for long-term success. AgentOps helps you:

Ensure Consistent Quality

• Establish standardized testing mechanisms

• Track AI Assistant performance trends

• Identify and fix quality issues promptly

Improve Operational Efficiency

• Automate testing processes to save manpower

• Quickly identify performance bottlenecks

• Optimize system resources and costs

Data-Driven Decision Making

• Quantify AI Assistant performance metrics

• Compare the effectiveness of different settings or models

• Develop optimization strategies based on actual data

AgentOps Core Features

1. Test Dataset Management

Create and manage test case collections to verify AI Assistant response quality.

Key Features:

• Create multiple test datasets

• Manage test cases and expected responses

• Support search and categorization

• Team collaboration on test case maintenance

Use Cases:

• Verify response accuracy after Knowledge Base updates

• Establish standard testing workflows

• Collect frequently asked questions as test benchmarks

For details: Test Dataset Management

2. Automated Evaluation

Use test datasets to run automated evaluations and generate detailed quality reports.

Key Features:

• One-click batch testing

• Calculate success rates and response times

• Generate detailed pass/fail reports

• Compare performance across different periods or AI Assistants

Use Cases:

• Regular quality checks

• Regression testing before and after updates

• Compare effectiveness of different AI Assistants or models

For details: Automated Evaluation

3. AI Assistant Monitoring

Monitor AI Assistant operations in real time and analyze every conversation in detail.

Key Features:

• View input/output message content

• Track reply time and processing time

• Monitor token usage and costs

• Analyze quality scores and user feedback

Use Cases:

• Real-time system operation monitoring

• Identify performance issues and anomalies

• Track costs and usage

• In-depth analysis of individual conversations

For details: AI Assistant Monitoring

AgentOps Workflow

Standard Quality Management Flow

1. Create Test Datasets
   ↓
   Collect common questions and expected responses
   Build structured test cases

2. Run Automated Evaluations
   ↓
   Run tests regularly or after updates
   Generate success rate and quality reports

3. Analyze Evaluation Results
   ↓
   Identify failed cases
   Find issues that need improvement

4. Optimize AI Assistant
   ↓
   Update Knowledge Base content
   Adjust AI settings and prompts

5. Monitor Actual Operations
   ↓
   Use AI Assistant Monitoring to track real conversations
   Continuously ensure quality stability

6. Continuous Improvement
   ↓
   Discover new issues from monitoring data
   Update test datasets
   Repeat the evaluation and optimization cycle

Quick Start Steps

Week 1: Build Foundational Tests

1. Collect 20-30 frequently asked questions

2. Create your first test dataset

3. Run the initial automated evaluation

4. Record the current success rate as a baseline

Week 2: Continuous Optimization

1. Analyze failed test cases

2. Update the Knowledge Base or adjust AI settings

3. Re-run evaluations to verify improvements

4. Start using AI Assistant Monitoring to observe real conversations

Week 3: Build Habits

1. Run automated evaluations once a week

2. Check monitoring data daily to identify anomalies

3. Continuously add test cases to cover more scenarios

4. Create quality reports to track long-term trends

AgentOps Integration with Other Features

AgentOps + Knowledge Base Management

Integration Use Cases:

• Run automated evaluations immediately after Knowledge Base updates

• Supplement Knowledge Base content when monitoring data reveals knowledge gaps

• Create test cases based on real conversations to optimize the Knowledge Base

Related documentation: Build an Enterprise Knowledge Base

AgentOps + Usage Analytics

Complementary Analysis:

• Usage Analytics: Provides overall trends and statistics (conversation volume, satisfaction, etc.)

• AgentOps: Provides in-depth quality analysis and individual conversation details

Combined use enables you to:

• Detect anomalies from overall trends

• Identify root causes from monitoring details

• Comprehensively evaluate AI Assistant performance

Related documentation: Usage Analytics

AgentOps + Response Quality Control

Integration Use Cases:

• AgentOps provides technical quality monitoring

• Response Quality Control provides manual review and annotation

• Together they form a complete quality assurance system

Related documentation: Response Quality Control

Best Practices

1. Establish a Regular Evaluation Schedule

Daily Check (5 minutes):

• Review AI Assistant Monitoring to identify anomalies

Weekly Evaluation (30 minutes):

• Run core test datasets

• Check whether success rates remain stable

• Update or add test cases

Monthly Analysis (2 hours):

• Run a full evaluation

• Export monitoring data for in-depth analysis

• Produce quality reports

• Plan optimization goals for the next month

2. Set Quality Baselines

Set reasonable quality targets for your AI Assistant:

Metric	Baseline	Target	Description
Automated Evaluation Success Rate	≥ 85%	≥ 95%	Standard test case pass rate
Average Response Time	< 5 seconds	< 3 seconds	User-perceived response speed
User Satisfaction	≥ 80%	≥ 90%	Like / total feedback ratio
Token Usage	-	-	Set limits based on budget

3. Establish an Issue Tracking Workflow

When quality issues are discovered:

Issue Discovered
↓
Record in test dataset (prevent recurrence)
↓
Analyze root cause (Knowledge Base / AI settings / model capability)
↓
Implement improvements
↓
Run evaluations to verify effectiveness
↓
Continue monitoring to confirm stability

4. Team Collaboration

Establish AgentOps responsibility assignments:

AI Manager:

• Overall quality monitoring

• Develop optimization strategies

• Regular evaluations and reporting

Knowledge Base Administrator:

• Update the Knowledge Base based on evaluation results

• Supplement missing knowledge content

Technical Staff:

• Analyze performance issues

• Optimize system settings

• Handle technical anomalies

5. Continuous Improvement Cycle

The core of AgentOps is continuous improvement:

1. Measure: Obtain data through evaluations and monitoring

2. Analyze: Identify issues and improvement opportunities

3. Act: Implement optimization measures

4. Verify: Confirm improvement effectiveness

5. Standardize: Incorporate successful practices into workflows

FAQ

Q1: What is the difference between AgentOps and "Usage Analytics"?

Usage Analytics:

• Overall trends and statistics (conversation volume, word count, satisfaction)

• Suitable for understanding AI Assistant usage overview

• Focuses on business metrics

AgentOps:

• In-depth quality analysis and individual conversation details

• Suitable for technical quality management

• Focuses on technical metrics (success rate, response time, cost)

The two are complementary — we recommend using them together.

Q2: How often should evaluations be run?

Recommended frequency:

• Core features: Once a week

• Full tests: Once a month

• After major updates: Run immediately

• When issues are found: Test and verify at any time

Adjust based on your AI Assistant's importance and update frequency.

Q3: How many test cases should a test dataset contain?

Recommended amounts:

• Minimum: 20 test cases (covering core features)

• Recommended: 50-100 test cases (balancing coverage and efficiency)

• Comprehensive: 200+ test cases (for large or critical systems)

Start with core features and expand gradually.

Q4: What success rate is considered acceptable?

It depends on the AI Assistant's purpose:

• High-risk applications (finance, healthcare): ≥ 95%

• Customer support: ≥ 90%

• General conversation: ≥ 85%

• Experimental features: ≥ 80%

The key is to establish a baseline and continuously improve.

Q5: Does AgentOps affect real users?

No. AgentOps testing and monitoring run in isolated environments or in the background, without interfering with the actual user experience.

Q6: Can I create multiple test sets for different business areas?

Yes. We recommend creating multiple test datasets for different business areas or features:

• Product inquiry test set

• Order processing test set

• Technical support test set

• ...and more

This allows you to track quality performance separately for each area.

---

Get Started with AgentOps

Ready to start using AgentOps? We recommend following this order:

1. Create Test Datasets: Collect common questions and create your first test set

2. Run Automated Evaluations: Test your AI Assistant and review evaluation reports

3. Use AI Assistant Monitoring: Gain in-depth understanding of actual operations

With AgentOps, you can establish a systematic quality management workflow to ensure your AI Assistant consistently delivers excellent service.