> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/maiagent-user-guide/maiagent-user-guide-en/agent-ops/agent-ops.md).

# AgentOps Overview

## What is AgentOps? <a href="#what-is-agent-ops" id="what-is-agent-ops"></a>

AgentOps (Agent Operations) is the AI Assistant operation management module provided by the MaiAgent platform, focusing on testing, evaluating, and monitoring the actual performance of AI Assistants. With AgentOps, you can systematically manage AI Assistant quality and ensure consistently excellent service experiences.

## Why Do You Need AgentOps? <a href="#why-agent-ops" id="why-agent-ops"></a>

Building an AI Assistant is just the first step — continuous monitoring and optimization are essential for long-term success. AgentOps helps you:

**Ensure Consistent Quality**

* Establish standardized testing mechanisms
* Track AI Assistant performance trends
* Identify and fix quality issues promptly

**Improve Operational Efficiency**

* Automate testing processes to save manpower
* Quickly identify performance bottlenecks
* Optimize system resources and costs

**Data-Driven Decision Making**

* Quantify AI Assistant performance metrics
* Compare the effectiveness of different settings or models
* Develop optimization strategies based on actual data

## AgentOps Core Features <a href="#agent-ops-core-features" id="agent-ops-core-features"></a>

### 1. Test Dataset Management <a href="#test-dataset-management" id="test-dataset-management"></a>

Create and manage test case collections to verify AI Assistant response quality.

**Key Features:**

* Create multiple test datasets
* Manage test cases and expected responses
* Support search and categorization
* Team collaboration on test case maintenance

**Use Cases:**

* Verify response accuracy after Knowledge Base updates
* Establish standard testing workflows
* Collect frequently asked questions as test benchmarks

> For details: [Test Dataset Management](/maiagent-user-guide/maiagent-user-guide-en/agent-ops/test-datasets.md)

### 2. Automated Evaluation <a href="#auto-evaluation" id="auto-evaluation"></a>

Use test datasets to run automated evaluations and generate detailed quality reports.

**Key Features:**

* One-click batch testing
* Calculate success rates and response times
* Generate detailed pass/fail reports
* Compare performance across different periods or AI Assistants

**Use Cases:**

* Regular quality checks
* Regression testing before and after updates
* Compare effectiveness of different AI Assistants or models

> For details: [Automated Evaluation](/maiagent-user-guide/maiagent-user-guide-en/agent-ops/evaluations.md)

### 3. AI Assistant Monitoring <a href="#ai-agent-monitoring" id="ai-agent-monitoring"></a>

Monitor AI Assistant operations in real time and analyze every conversation in detail.

**Key Features:**

* View input/output message content
* Track reply time and processing time
* Monitor token usage and costs
* Analyze quality scores and user feedback

**Use Cases:**

* Real-time system operation monitoring
* Identify performance issues and anomalies
* Track costs and usage
* In-depth analysis of individual conversations

> For details: [AI Assistant Monitoring](/maiagent-user-guide/maiagent-user-guide-en/agent-ops/evaluations.md)

## AgentOps Workflow <a href="#agent-ops-workflow" id="agent-ops-workflow"></a>

### Standard Quality Management Flow <a href="#standard-quality-management-flow" id="standard-quality-management-flow"></a>

```
1. Create Test Datasets
   ↓
   Collect common questions and expected responses
   Build structured test cases

2. Run Automated Evaluations
   ↓
   Run tests regularly or after updates
   Generate success rate and quality reports

3. Analyze Evaluation Results
   ↓
   Identify failed cases
   Find issues that need improvement

4. Optimize AI Assistant
   ↓
   Update Knowledge Base content
   Adjust AI settings and prompts

5. Monitor Actual Operations
   ↓
   Use AI Assistant Monitoring to track real conversations
   Continuously ensure quality stability

6. Continuous Improvement
   ↓
   Discover new issues from monitoring data
   Update test datasets
   Repeat the evaluation and optimization cycle
```

### Quick Start Steps <a href="#quick-start-steps" id="quick-start-steps"></a>

**Week 1: Build Foundational Tests**

1. Collect 20-30 frequently asked questions
2. Create your first test dataset
3. Run the initial automated evaluation
4. Record the current success rate as a baseline

**Week 2: Continuous Optimization**

1. Analyze failed test cases
2. Update the Knowledge Base or adjust AI settings
3. Re-run evaluations to verify improvements
4. Start using AI Assistant Monitoring to observe real conversations

**Week 3: Build Habits**

1. Run automated evaluations once a week
2. Check monitoring data daily to identify anomalies
3. Continuously add test cases to cover more scenarios
4. Create quality reports to track long-term trends

## AgentOps Integration with Other Features <a href="#agent-ops-integrations" id="agent-ops-integrations"></a>

### AgentOps + Knowledge Base Management <a href="#agent-ops-plus-knowledge-base" id="agent-ops-plus-knowledge-base"></a>

**Integration Use Cases:**

* Run automated evaluations immediately after Knowledge Base updates
* Supplement Knowledge Base content when monitoring data reveals knowledge gaps
* Create test cases based on real conversations to optimize the Knowledge Base

> Related documentation: [Build an Enterprise Knowledge Base](https://github.com/Playma-Co-Ltd/maiagent-user-guide-gitbook/tree/main/en/km/km.md)

### AgentOps + Usage Analytics <a href="#agent-ops-plus-usage-analytics" id="agent-ops-plus-usage-analytics"></a>

**Complementary Analysis:**

* **Usage Analytics**: Provides overall trends and statistics (conversation volume, satisfaction, etc.)
* **AgentOps**: Provides in-depth quality analysis and individual conversation details

Combined use enables you to:

* Detect anomalies from overall trends
* Identify root causes from monitoring details
* Comprehensively evaluate AI Assistant performance

> Related documentation: [Usage Analytics](https://github.com/Playma-Co-Ltd/maiagent-user-guide-gitbook/tree/main/en/track/usage.md)

### AgentOps + Response Quality Control <a href="#agent-ops-plus-reply-quality-control" id="agent-ops-plus-reply-quality-control"></a>

**Integration Use Cases:**

* AgentOps provides technical quality monitoring
* Response Quality Control provides manual review and annotation
* Together they form a complete quality assurance system

> Related documentation: [Response Quality Control](https://github.com/Playma-Co-Ltd/maiagent-user-guide-gitbook/tree/main/en/track/quality.md)

## Best Practices <a href="#best-practices" id="best-practices"></a>

### 1. Establish a Regular Evaluation Schedule <a href="#establish-regular-evaluation" id="establish-regular-evaluation"></a>

**Daily Check (5 minutes):**

* Review AI Assistant Monitoring to identify anomalies

**Weekly Evaluation (30 minutes):**

* Run core test datasets
* Check whether success rates remain stable
* Update or add test cases

**Monthly Analysis (2 hours):**

* Run a full evaluation
* Export monitoring data for in-depth analysis
* Produce quality reports
* Plan optimization goals for the next month

### 2. Set Quality Baselines <a href="#set-quality-baseline" id="set-quality-baseline"></a>

Set reasonable quality targets for your AI Assistant:

| Metric                            | Baseline    | Target      | Description                   |
| --------------------------------- | ----------- | ----------- | ----------------------------- |
| Automated Evaluation Success Rate | ≥ 85%       | ≥ 95%       | Standard test case pass rate  |
| Average Response Time             | < 5 seconds | < 3 seconds | User-perceived response speed |
| User Satisfaction                 | ≥ 80%       | ≥ 90%       | Like / total feedback ratio   |
| Token Usage                       | -           | -           | Set limits based on budget    |

### 3. Establish an Issue Tracking Workflow <a href="#issue-tracking-workflow" id="issue-tracking-workflow"></a>

When quality issues are discovered:

```
Issue Discovered
↓
Record in test dataset (prevent recurrence)
↓
Analyze root cause (Knowledge Base / AI settings / model capability)
↓
Implement improvements
↓
Run evaluations to verify effectiveness
↓
Continue monitoring to confirm stability
```

### 4. Team Collaboration <a href="#team-collaboration" id="team-collaboration"></a>

Establish AgentOps responsibility assignments:

**AI Manager:**

* Overall quality monitoring
* Develop optimization strategies
* Regular evaluations and reporting

**Knowledge Base Administrator:**

* Update the Knowledge Base based on evaluation results
* Supplement missing knowledge content

**Technical Staff:**

* Analyze performance issues
* Optimize system settings
* Handle technical anomalies

### 5. Continuous Improvement Cycle <a href="#continuous-improvement-cycle" id="continuous-improvement-cycle"></a>

The core of AgentOps is continuous improvement:

1. **Measure**: Obtain data through evaluations and monitoring
2. **Analyze**: Identify issues and improvement opportunities
3. **Act**: Implement optimization measures
4. **Verify**: Confirm improvement effectiveness
5. **Standardize**: Incorporate successful practices into workflows

## FAQ <a href="#faq" id="faq"></a>

### Q1: What is the difference between AgentOps and "Usage Analytics"? <a href="#faq-agent-ops-vs-usage-analytics" id="faq-agent-ops-vs-usage-analytics"></a>

**Usage Analytics:**

* Overall trends and statistics (conversation volume, word count, satisfaction)
* Suitable for understanding AI Assistant usage overview
* Focuses on business metrics

**AgentOps:**

* In-depth quality analysis and individual conversation details
* Suitable for technical quality management
* Focuses on technical metrics (success rate, response time, cost)

The two are complementary — we recommend using them together.

### Q2: How often should evaluations be run? <a href="#faq-evaluation-frequency" id="faq-evaluation-frequency"></a>

Recommended frequency:

* **Core features**: Once a week
* **Full tests**: Once a month
* **After major updates**: Run immediately
* **When issues are found**: Test and verify at any time

Adjust based on your AI Assistant's importance and update frequency.

### Q3: How many test cases should a test dataset contain? <a href="#faq-test-dataset-size" id="faq-test-dataset-size"></a>

Recommended amounts:

* **Minimum**: 20 test cases (covering core features)
* **Recommended**: 50-100 test cases (balancing coverage and efficiency)
* **Comprehensive**: 200+ test cases (for large or critical systems)

Start with core features and expand gradually.

### Q4: What success rate is considered acceptable? <a href="#faq-success-rate-threshold" id="faq-success-rate-threshold"></a>

It depends on the AI Assistant's purpose:

* **High-risk applications** (finance, healthcare): ≥ 95%
* **Customer support**: ≥ 90%
* **General conversation**: ≥ 85%
* **Experimental features**: ≥ 80%

The key is to establish a baseline and continuously improve.

### Q5: Does AgentOps affect real users? <a href="#faq-impact-on-real-users" id="faq-impact-on-real-users"></a>

No. AgentOps testing and monitoring run in isolated environments or in the background, without interfering with the actual user experience.

### Q6: Can I create multiple test sets for different business areas? <a href="#faq-multiple-test-sets-per-agent" id="faq-multiple-test-sets-per-agent"></a>

Yes. We recommend creating multiple test datasets for different business areas or features:

* Product inquiry test set
* Order processing test set
* Technical support test set
* ...and more

This allows you to track quality performance separately for each area.

***

## Get Started with AgentOps <a href="#get-started-with-agent-ops" id="get-started-with-agent-ops"></a>

Ready to start using AgentOps? We recommend following this order:

1. [**Create Test Datasets**](/maiagent-user-guide/maiagent-user-guide-en/agent-ops/test-datasets.md): Collect common questions and create your first test set
2. [**Run Automated Evaluations**](/maiagent-user-guide/maiagent-user-guide-en/agent-ops/evaluations.md): Test your AI Assistant and review evaluation reports
3. [**Use AI Assistant Monitoring**](/maiagent-user-guide/maiagent-user-guide-en/agent-ops/evaluations.md): Gain in-depth understanding of actual operations

With AgentOps, you can establish a systematic quality management workflow to ensure your AI Assistant consistently delivers excellent service.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.maiagent.ai/maiagent-user-guide/maiagent-user-guide-en/agent-ops/agent-ops.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.