Conversation Platform Analytics
Use AI to automatically analyze topic clusters, resolution rates, and improvement suggestions for conversation data on a specific platform
Inbox Analysis can analyze real conversations within a specific platform and date range, using AI to automatically categorize topics, determine resolution status, and compile actionable improvement suggestions to help you continuously optimize your AI assistant's conversation quality.
What is Inbox Analysis?
Once customer service conversations accumulate to a certain volume, looking at totals or satisfaction scores alone isn't enough — you want to know:
What are users actually asking? What are the most frequently asked topics?
How many conversations were truly resolved? Which types of issues have low resolution rates?
Why weren't they resolved? Is it missing knowledge base content, overly restrictive instructions, or tool failures?
How are response times and costs distributed? Which topics take longer and consume more tokens?
Inbox Analysis is designed to answer these questions. The system will:
Retrieve conversations within the specified time range
Use embedding models for topic clustering
Use LLM to analyze each conversation's intent, resolution status, and failure reasons
Generate a comprehensive analysis summary (including improvement suggestions)
Differences from Similar Features
Real-time statistics
AI assistant-level quantitative KPIs (conversation word count, frequency, satisfaction)
Inbox "Conversation Analysis" settings (tab within each inbox's settings)
Real-time, on each new incoming message
Use LLM to automatically tag conversations for routing and management
Inbox Analysis (this page)
Manual trigger, batch review
Topic clustering, resolution rate analysis, and improvement suggestions for a specified date range
How to Use
Navigate to the Inbox Analysis Page
Click "Customer Service" → "Inboxes" in the left menu
Find the target inbox in the inbox list
Click the 📊 icon on the right side of that row (hover shows "Inbox Analysis") to enter the analysis page
The page displays all analysis reports previously created for this inbox, including date range, status, conversation count, processing time, and other information.
Create a New Analysis
Click the "New Analysis" button in the upper right corner and fill in the following required fields:
Date Range: Select the conversation period to analyze (future dates cannot be selected)
Analysis LLM: The large language model used to analyze each conversation
Embedding Model: The embedding model used for topic clustering
Report Language: The output language for the analysis summary (Traditional Chinese, Simplified Chinese, English, Japanese, Korean)
Expand "Advanced Settings" to adjust the sampling strategy (see "Advanced Settings" section below).
After clicking "Confirm", the system will run the analysis asynchronously in the background. The status will progress from "Pending" → "Processing" → "Completed".
View the Report
Once the status changes to "Completed", click the "View Report" icon (👁) on that report to enter the report detail page.
The report content varies depending on conversation data richness, and includes data overview, resolution status, topic clusters, top questions, performance metrics, analysis summary, and per-conversation analysis.
Report Content
1. Data Overview
Total Conversations
Total number of conversations on the inbox within the date range
Analyzed
Number of conversations actually analyzed by LLM (limited by "Max Sample Size")
Sample Rate
Analyzed / Total Conversations (percentage)
Processing Time
Time in seconds from start to completion of this analysis task
When the total number of conversations exceeds the "Max Sample Size", the system uses a post-clustering sampling strategy to select representative conversations for analysis, controlling cost and time.
2. Resolution Status
The core metric "Resolution Rate" formula:
Resolved (green): AI assistant fully answered the user's question
Partially Resolved (orange): Answer was partially correct or in the right direction but incomplete
Unresolved (red): Unable to answer or answered incorrectly
Resolution rates above 50% are displayed in green, below 50% in red, serving as an instant health indicator.
3. Performance Metrics
Shows the AI assistant's computational performance during this period:
Total Token Usage: Input + Output tokens combined
Average Tokens per Turn: Average token consumption per conversation turn
Average Response Time: Total time from receiving a message to completing the response
Average Time to First Token (TTFT): Time from receiving a message to outputting the first token
4. Topic Cluster Distribution
All conversations are clustered by "semantic similarity" and displayed as horizontal bar charts showing the frequency and proportion of each topic. Outlier conversations are categorized as "Noise / Outliers".
This view helps quickly identify the most frequently asked topics.
5. Resolution / Failure Reason Distribution
The system labels each analyzed conversation with a status or failure reason, summarized as follows:
Resolved
AI assistant successfully answered
Partially Resolved
Answer was in the right direction but incomplete
Knowledge Gap
Knowledge base lacks relevant content
Overly Restrictive Instructions
Role instructions blocked a reasonable response
Tool Failure
Tool execution error or not triggered
Unclear User Input
User message was vague or intent could not be determined
Off-Topic
User's question was outside the assistant's scope
Hallucination
AI generated non-existent or incorrect information
Unknown
Cannot be categorized into the above
This distribution directly guides subsequent improvement actions: many knowledge gaps → add content; many overly restrictive instructions → adjust role instructions; many hallucinations → strengthen knowledge base and instruction constraints.
6. Top Questions Ranking
A topic list sorted by "frequency", showing each topic's occurrence count and resolution rate. Expanding any row reveals example intents for that topic, helping you quickly verify whether the clustering matches expectations.
7. Performance Distribution by Topic
Breaks down tokens and response times by topic to identify "expensive" or "slow" topics:
Average Input / Output / Total Tokens
Average Response Time
Average Time to First Token
If a topic has abnormally high token usage, consider optimizing role instructions or knowledge base retrieval; if a topic takes long, it may be due to multiple tool calls or lengthy content.
8. Analysis Summary
A Markdown report generated by LLM synthesizing all analysis results, typically including:
Overall performance summary
Major issue patterns
Priority improvement suggestions
Specific adjustment directions
Displayed as a collapsed section by default; click the title to expand and read.
9. Per-Conversation Analysis
When expanded, you can review each analyzed conversation individually, with each entry containing:
User Intent: What the user actually wanted to ask
Topic: The assigned topic cluster
Resolution Detail: Detailed description of the resolution status
Suggestions: Specific optimization directions for that conversation
Advanced Settings
Advanced settings control sampling and execution strategies. It's recommended to start with default values and adjust as needed:
Samples per Cluster
10
Maximum conversations per topic cluster sent to LLM for analysis
Max Sample Size
150
Maximum conversations analyzed in the entire report (cost ceiling)
Concurrency
8
Number of parallel LLM analysis processes; higher = faster but more resource-intensive
Max Retrieved Conversations
—
Upper limit of conversations fetched from the database
Min Conversation Messages
—
Conversations with fewer messages than this value are filtered out (avoids analyzing meaningless short conversations)
Cost Reminder: Analysis consumes LLM token quota (Input + Output + Embedding). The system checks organization quota before analysis; tasks cannot be created if quota is insufficient.
Use Cases
Scenario 1: Quarterly Quality Review
Need: A manager wants to understand last quarter's conversation quality and main issues on the Web Chat inbox.
Steps:
Set the date range to the previous quarter
Set Max Sample Size to 300 (larger sample)
Run the analysis, focusing on "Resolution / Failure Reason Distribution" and "Top Questions Ranking"
Schedule improvement tasks based on high-frequency reasons such as knowledge gaps and overly restrictive instructions
Scenario 2: Impact Assessment After Version Update
Need: Role instructions were adjusted last week; want to know if conversation quality improved.
Steps:
Create two analysis reports for "one week before the adjustment" and "one week after the adjustment"
Compare resolution rates and failure reason distributions between the two reports
Confirm whether specific failure types have decreased
Scenario 3: Topic Clustering for Knowledge Base Completion
Need: Discovered frequently asked questions and want to systematically supplement the knowledge base.
Steps:
Analyze conversations from the past month
From "Top Questions Ranking", pick high-frequency, low-resolution-rate topics
Expand "Example Intents" to confirm actual phrasings
Go back to the knowledge base and add corresponding FAQ or documents
FAQ
Q: Why weren't some conversations analyzed?
The system filters based on "Max Sample Size" and "Min Conversation Messages", and performs representative sampling after clustering. If the total number of conversations is large, only representative samples are selected to control costs.
Q: How is the resolution rate determined?
The analysis LLM directly classifies each analyzed conversation into one of 9 statuses: Resolved, Partially Resolved, Knowledge Gap, Overly Restrictive Instructions, Tool Failure, Unclear User Input, Off-Topic, Hallucination, Unknown.
Resolution Rate formula:
Both "Resolved" and "Partially Resolved" count toward the resolution rate numerator; the remaining 7 statuses are all considered unresolved.
Q: Can multiple analyses be created for the same inbox simultaneously?
Yes, but date ranges cannot overlap with an in-progress analysis. If you see the error "This inbox already has an in-progress analysis with overlapping date range", wait for that analysis to complete or select a different date range.
Q: How long does the analysis take?
It depends on conversation volume, max sample size, concurrency, and the selected model. Generally, 150 samples take approximately a few minutes to around ten minutes. The system's default processing time limit is 110 minutes; analyses exceeding this will be automatically marked as failed.
Q: What if the analysis fails?
If the status shows "Failed", check the "Error Message" column in the report list. Common messages and how to handle them:
"Analysis timed out — the worker may have crashed during processing." Processing exceeded the time limit: reduce "Max Sample Size" to lower the analysis volume and retry
"Analysis task was never picked up by a worker." Task was not dispatched: try again later
For other messages: take a screenshot and report to MaiAgent support for investigation
If there is insufficient quota, the analysis will be blocked at creation time with a prompt, and will not appear as a "Failed" report.
Q: Why can't I see "Inbox Analysis" on my account?
"Inbox Analysis" is a sub-permission under the "Customer Service" access permission. By default, any MaiAgent role with "Customer Service" access permission will include this feature.
If you cannot see it, it may be because:
Your MaiAgent role does not have "Customer Service" access permission
The organization administrator has disabled the "Inbox Analysis" sub-permission for your role
Contact your organization administrator to check and adjust settings on the Role Permission Management page.
Notes
Permission Notes
"Inbox Analysis" is a sub-permission under the "Customer Service" access permission, included by default in all MaiAgent roles with "Customer Service" permission
Organization administrators can adjust this feature's availability for different roles on the Role Permission Management page
Members with this permission can create and view all analysis reports within their organization
Cost Reminder
Each analysis consumes LLM Input / Output Tokens and Embedding Tokens
Before creating an analysis, confirm the date range and max sample size in the modal
Adjust the max sample size in advanced settings to control cost ceiling
Data Protection
The analysis process sends conversation content to the selected LLM for processing
Choose a compliant model according to your organization's data processing policies
Related Features
Further Reading
Usage Analytics: Quantitative KPI dashboard at the AI assistant level
Inbox Search: Quickly search for specific conversations
Conversation Viewer: View the full context of a single conversation
Evaluation Insights Report: Automated insights after AgentOps batch testing
Last updated
Was this helpful?