> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/maiagent-user-guide/en/agent-ops/agent-ops.md). # AgentOps Overview AgentOps provides comprehensive AI Assistant operation management and quality control tools ## What is AgentOps? AgentOps (Agent Operations) is the AI Assistant operation management module provided by the MaiAgent platform, focusing on testing, evaluating, and monitoring the actual performance of AI Assistants. With AgentOps, you can systematically manage AI Assistant quality and ensure consistently excellent service experiences. ## Why Do You Need AgentOps? Building an AI Assistant is just the first step — continuous monitoring and optimization are essential for long-term success. AgentOps helps you: **Ensure Consistent Quality** * Establish standardized testing mechanisms * Track AI Assistant performance trends * Identify and fix quality issues promptly **Improve Operational Efficiency** * Automate testing processes to save manpower * Quickly identify performance bottlenecks * Optimize system resources and costs **Data-Driven Decision Making** * Quantify AI Assistant performance metrics * Compare the effectiveness of different settings or models * Develop optimization strategies based on actual data ## AgentOps Core Features ### 1. Test Dataset Management Create and manage test case collections to verify AI Assistant response quality. **Key Features:** * Create multiple test datasets * Manage test cases and expected responses * Support search and categorization * Team collaboration on test case maintenance **Use Cases:** * Verify response accuracy after Knowledge Base updates * Establish standard testing workflows * Collect frequently asked questions as test benchmarks > For details: [Test Dataset Management](/maiagent-user-guide/en/agent-ops/test-datasets.md) ### 2. Automated Evaluation Use test datasets to run automated evaluations and generate detailed quality reports. **Key Features:** * One-click batch testing * Calculate success rates and response times * Generate detailed pass/fail reports * Compare performance across different periods or AI Assistants **Use Cases:** * Regular quality checks * Regression testing before and after updates * Compare effectiveness of different AI Assistants or models > For details: [Automated Evaluation](/maiagent-user-guide/en/agent-ops/evaluations.md#auto-evaluation) ### 3. AI Assistant Monitoring Monitor AI Assistant operations in real time and analyze every conversation in detail. **Key Features:** * View input/output message content * Track reply time and processing time * Monitor token usage and costs * Analyze quality scores and user feedback **Use Cases:** * Real-time system operation monitoring * Identify performance issues and anomalies * Track costs and usage * In-depth analysis of individual conversations > For details: [AI Assistant Monitoring](/maiagent-user-guide/en/agent-ops/evaluations.md#ai-agent-monitoring) ### 4. Evaluation Insight Report After batch testing is complete, the system automatically analyzes the results and generates an improvement recommendations report — no need to manually interpret each scoring data point. **Key Features:** * Automatically identify problem patterns in test results * Suggest handling priorities based on severity and scope of impact * Provide concrete improvement directions, not just scores * Support multilingual reports **Use Cases:** * Success rate falls below expectations and you need to know where the problems are * A single test run produces many failed cases and you need to decide which to fix first * You need to explain the current quality status and improvement plan to your team or management > For details: [Evaluation Insight Report](/maiagent-user-guide/en/agent-ops/evaluation-insights.md) ### 5. Tool Execution Records Track the complete history of AI Assistant external tool invocations and quickly pinpoint the cause of tool execution failures. **Key Features:** * Record input parameters, execution results, duration, and success/failure status for each tool invocation * Retain detailed error messages to help pinpoint failure causes * Track usage count and success rate for each tool * Cover API tools, Text-to-SQL tools, crawler tools, and custom tools **Use Cases:** * AI Assistant cannot answer questions that require querying external systems * Tools fail intermittently and you need to determine whether it's a timeout, authentication, or parameter issue * Evaluating whether a tool needs configuration adjustments or replacement > For details: [Tool Execution Records](/maiagent-user-guide/en/agent-ops/tool-execution-records.md) ## AgentOps Workflow ### Sub-feature Relationship Map AgentOps has five sub-features divided into two tracks: **Pre-launch** uses test datasets and automated evaluation to establish quality benchmarks, while **Post-launch** uses monitoring and tool records to observe real operations. The two tracks form a cycle by "feeding issues discovered in production back into the test datasets."
1. Test Dataset ManagementCreate reusable test case collections — the starting point of the entire testing chain.Test Dataset Management
2. Automated EvaluationBatch test AI Assistants using test datasets, producing success rates and average response times.Automated Evaluation & AI Assistant Monitoring
3. Evaluation Insight ReportAutomatically analyze evaluation results into problem patterns and improvement suggestions, ranked by priority.Evaluation Insights Report
4. AI Assistant MonitoringInspect real conversations one by one after launch — inputs, outputs, duration, tokens, and scores.Automated Evaluation & AI Assistant Monitoring
5. Tool Execution RecordsTrack AI Assistant external tool invocation parameters, results, and failure causes.Tool Execution Records
Sequence and dependencies of each sub-feature: | Sub-feature | Stage | Prerequisites | Output | Next Step | | ----------------------------- | ----------- | ------------------------------------------- | ----------------------------------------------------------------- | -------------------------------------------------------------------------------- | | **Test Dataset Management** | Pre-launch | Common questions and expected responses | Reusable test case collections | Hand off to Automated Evaluation | | **Automated Evaluation** | Pre-launch | Test datasets, AI Assistant to test | Success rate, average response time, per-case pass/fail | Hand off to Evaluation Insight Report for analysis | | **Evaluation Insight Report** | Pre-launch | A completed automated evaluation | Problem patterns, priorities, concrete improvement directions | Adjust Knowledge Base or role instructions based on suggestions, then test again | | **AI Assistant Monitoring** | Post-launch | Real conversations from a live AI Assistant | Input/output, duration, token usage, scores for each conversation | Feed discovered issues back into test datasets | | **Tool Execution Records** | Post-launch | AI Assistant with configured tools | Tool invocation parameters, results, duration, and error messages | Fix tool settings, or add a test case | {% hint style="info" %} **Two commonly confused points:** * "Automated Evaluation" tests **questions from the test datasets**, while "AI Assistant Monitoring" examines **conversations actual users have had** — the former is a pre-launch quality gate, the latter is post-launch observation. * "Evaluation Insight Report" is not a standalone feature — it is an extension of Automated Evaluation: you must first complete a batch test before an insight report is available. {% endhint %} ### Standard Quality Management Flow ``` 1. Create Test Datasets ↓ Collect common questions and expected responses Build structured test cases 2. Run Automated Evaluations ↓ Run tests regularly or after updates Generate success rate and quality reports 3. Analyze Evaluation Results ↓ Identify failed cases Find issues that need improvement 4. Optimize AI Assistant ↓ Update Knowledge Base content Adjust AI settings and prompts 5. Monitor Actual Operations ↓ Use AI Assistant Monitoring to track real conversations Continuously ensure quality stability 6. Continuous Improvement ↓ Discover new issues from monitoring data Update test datasets Repeat the evaluation and optimization cycle ``` ### Quick Start Steps **Week 1: Build Foundational Tests** 1. Collect 20-30 frequently asked questions 2. Create your first test dataset 3. Run the initial automated evaluation 4. Record the current success rate as a baseline **Week 2: Continuous Optimization** 1. Analyze failed test cases 2. Update the Knowledge Base or adjust AI settings 3. Re-run evaluations to verify improvements 4. Start using AI Assistant Monitoring to observe real conversations **Week 3: Build Habits** 1. Run automated evaluations once a week 2. Check monitoring data daily to identify anomalies 3. Continuously add test cases to cover more scenarios 4. Create quality reports to track long-term trends ## AgentOps Integration with Other Features ### AgentOps + Knowledge Base Management **Integration Use Cases:** * Run automated evaluations immediately after Knowledge Base updates * Supplement Knowledge Base content when monitoring data reveals knowledge gaps * Create test cases based on real conversations to optimize the Knowledge Base > Related documentation: [Build an Enterprise Knowledge Base](/maiagent-user-guide/en/km/km.md) ### AgentOps + Usage Analytics **Complementary Analysis:** * **Usage Analytics**: Provides overall trends and statistics (conversation volume, satisfaction, etc.) * **AgentOps**: Provides in-depth quality analysis and individual conversation details Combined use enables you to: * Detect anomalies from overall trends * Identify root causes from monitoring details * Comprehensively evaluate AI Assistant performance > Related documentation: [Usage Analytics](/maiagent-user-guide/en/org/usage.md) ### AgentOps + Response Quality Control **Integration Use Cases:** * AgentOps provides technical quality monitoring * Response Quality Control provides manual review and annotation * Together they form a complete quality assurance system > Related documentation: [Response Quality Control](https://github.com/Playma-Co-Ltd/maiagent-user-guide-gitbook/tree/main/en/channels/quality.md) ## Best Practices ### 1. Establish a Regular Evaluation Schedule **Daily Check (5 minutes):** * Review AI Assistant Monitoring to identify anomalies **Weekly Evaluation (30 minutes):** * Run core test datasets * Check whether success rates remain stable * Update or add test cases **Monthly Analysis (2 hours):** * Run a full evaluation * Export monitoring data for in-depth analysis * Produce quality reports * Plan optimization goals for the next month ### 2. Set Quality Baselines Set reasonable quality targets for your AI Assistant: | Metric | Baseline | Target | Description | | --------------------------------- | ----------- | ----------- | ----------------------------- | | Automated Evaluation Success Rate | ≥ 85% | ≥ 95% | Standard test case pass rate | | Average Response Time | < 5 seconds | < 3 seconds | User-perceived response speed | | User Satisfaction | ≥ 80% | ≥ 90% | Like / total feedback ratio | | Token Usage | - | - | Set limits based on budget | ### 3. Establish an Issue Tracking Workflow When quality issues are discovered: ``` Issue Discovered ↓ Record in test dataset (prevent recurrence) ↓ Analyze root cause (Knowledge Base / AI settings / model capability) ↓ Implement improvements ↓ Run evaluations to verify effectiveness ↓ Continue monitoring to confirm stability ``` ### 4. Team Collaboration Establish AgentOps responsibility assignments: **AI Manager:** * Overall quality monitoring * Develop optimization strategies * Regular evaluations and reporting **Knowledge Base Administrator:** * Update the Knowledge Base based on evaluation results * Supplement missing knowledge content **Technical Staff:** * Analyze performance issues * Optimize system settings * Handle technical anomalies ### 5. Continuous Improvement Cycle The core of AgentOps is continuous improvement: 1. **Measure**: Obtain data through evaluations and monitoring 2. **Analyze**: Identify issues and improvement opportunities 3. **Act**: Implement optimization measures 4. **Verify**: Confirm improvement effectiveness 5. **Standardize**: Incorporate successful practices into workflows ## FAQ ### Q1: What is the difference between AgentOps and "Usage Analytics"? **Usage Analytics:** * Overall trends and statistics (conversation volume, word count, satisfaction) * Suitable for understanding AI Assistant usage overview * Focuses on business metrics **AgentOps:** * In-depth quality analysis and individual conversation details * Suitable for technical quality management * Focuses on technical metrics (success rate, response time, cost) The two are complementary — we recommend using them together. ### Q2: How often should evaluations be run? Recommended frequency: * **Core features**: Once a week * **Full tests**: Once a month * **After major updates**: Run immediately * **When issues are found**: Test and verify at any time Adjust based on your AI Assistant's importance and update frequency. ### Q3: How many test cases should a test dataset contain? Recommended amounts: * **Minimum**: 20 test cases (covering core features) * **Recommended**: 50-100 test cases (balancing coverage and efficiency) * **Comprehensive**: 200+ test cases (for large or critical systems) Start with core features and expand gradually. ### Q4: What success rate is considered acceptable? It depends on the AI Assistant's purpose: * **High-risk applications** (finance, healthcare): ≥ 95% * **Customer support**: ≥ 90% * **General conversation**: ≥ 85% * **Experimental features**: ≥ 80% The key is to establish a baseline and continuously improve. ### Q5: Does AgentOps affect real users? No. AgentOps testing and monitoring run in isolated environments or in the background, without interfering with the actual user experience. ### Q6: Can I create multiple test sets for different business areas? Yes. We recommend creating multiple test datasets for different business areas or features: * Product inquiry test set * Order processing test set * Technical support test set * ...and more This allows you to track quality performance separately for each area. *** ## Get Started with AgentOps Ready to start using AgentOps? We recommend following this order: 1. [**Create Test Datasets**](/maiagent-user-guide/en/agent-ops/test-datasets.md): Collect common questions and create your first test set 2. [**Run Automated Evaluations**](/maiagent-user-guide/en/agent-ops/evaluations.md#auto-evaluation): Test your AI Assistant and review success rates and response times 3. [**Read the Evaluation Insight Report**](/maiagent-user-guide/en/agent-ops/evaluation-insights.md): Understand where the problems are and what to prioritize 4. [**Use AI Assistant Monitoring**](/maiagent-user-guide/en/agent-ops/evaluations.md#ai-agent-monitoring): Gain in-depth understanding of actual operations after launch 5. [**View Tool Execution Records**](/maiagent-user-guide/en/agent-ops/tool-execution-records.md): If your AI Assistant has configured tools, track whether tool invocations are working properly With AgentOps, you can establish a systematic quality management workflow to ensure your AI Assistant consistently delivers excellent service. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.maiagent.ai/maiagent-user-guide/en/agent-ops/agent-ops.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.