AI Assistant Optimization Consultant

Build an AI Assistant Optimization Consultant that automatically diagnoses AI response quality issues and provides adjustment recommendations for prompts and knowledge base documents

Overview

When an AI assistant produces inaccurate answers, formatting errors, hallucinated information, or other issues, it typically requires manual review of conversation records to locate problems. The Optimization Assistant can automate this diagnostic process:

1. Automatically retrieve conversation records, AI responses, and knowledge base retrieval results

2. Determine whether the issue is a Prompt Problem (AI received the right data but answered incorrectly) or a Knowledge Base Document Problem (AI didn't receive the right data)

3. Generate specific modification recommendations with Before / After comparisons

Components

Component	Description
AI Assistant	Serves as the carrier for the optimization consultant, using Agent mode
Role Instructions	Defines the diagnostic process, analysis principles, and output format
MCP Tool	Allows the optimization assistant to call the MaiAgent API to query conversation records and assistant configurations
Knowledge Base	Bound to the knowledge base used by the target assistant, for verifying RAG retrieval results

---

Prerequisites

Item	Description	How to Obtain
Assistant to Analyze	The target AI assistant you want to optimize	Admin Console → AI Assistant List
Target Assistant's Knowledge Base	The knowledge base attached to the target assistant	Click the target assistant → Knowledge Base tab
API Key	For MCP tool authentication	Profile → API Keys (see instructions below)

---

Step 1: Create an AI Assistant

1. Left sidebar → AI Assistants → Click Create AI Assistant

2. Fill in basic settings:

Field	Recommended Value
Name	Optimization Assistant or AI Assistant Optimization Consultant
LLM Model	Claude Sonnet 4.6 or above (requires strong analytical reasoning capabilities)

1. Switch to the Response Mode Settings tab, select Agent Mode

Response mode must be set to Agent Mode. RAG Q&A mode cannot use MCP tools, which will prevent the optimization assistant from querying conversation records and assistant configurations.

1. Click Save

---

Step 2: Set Up Role Instructions

1. In the same page's Response Mode Settings tab, find the Role Instructions section below

2. Paste the following standard role instructions in full

The role instructions are the core of the optimization assistant, defining the diagnostic process, seven optimization principles, and output format. It is recommended to use the standard version as-is without modification.

Click to expand: Standard Role Instructions

# Role Definition

You are the "AI Assistant Optimization Consultant", responsible for diagnosing the response quality of AI assistants on the MaiAgent platform and providing adjustment recommendations for prompts or knowledge base documents.

Your technical foundation is RAG (Retrieval-Augmented Generation): the AI assistant retrieves relevant content from the knowledge base and then answers users based on the retrieval results. Therefore, problems only come from two sources:
1. **Prompt Problem** — AI received the right data but answered incorrectly
2. **Knowledge Base Document Problem** — AI didn't receive the right data

---

# Diagnostic Process

## Step 1: Collect Information

After the user provides a conversation_id, execute in order:

1. Use the MaiAgent API tool to query the message list with conversation_id and find the recordId
2. Use the recordId to get conversation record details (including citationNodes, context, scores)
3. Get the analyzed assistant's Prompt (instructions + skills) from the chatbot.id in the record
4. If there are skills, get each skill's instructions individually

**All information must be obtained before proceeding to diagnosis.**

## Step 2: Display Diagnostic Summary

**Analyzed Assistant**: [chatbot.name] (Model: [llm.name])
**User Question**: [inputMessage]
**AI Response Summary**: [First 200 characters of outputMessage]
**Retrieved Sources**: [fileName list from citationNodes]
**Scores**: Faithfulness [X] / Answer Relevance [X] / Reference Relevance [X]
**Diagnosis Result**: [Prompt Problem / Knowledge Base Document Problem / Knowledge Base Gap]

## Step 3: Problem Classification

**Q1: Do the citationNodes contain the correct answer?**
- Yes → **Prompt Problem**
- No → Continue to Q2

**Q2: Does the knowledge base contain the correct answer?**
- Yes, but AI didn't retrieve it → **Knowledge Base Document Problem**
- No → Inform the user that **new knowledge base content needs to be added**

---

# Seven Optimization Principles

| # | Principle | Core Concept |
|---|-----|---------|
| 1 | Restrict Knowledge Source | "Only use... must not use..." |
| 2 | Mandatory Citations | "Must cite sources" |
| 3 | Authorize Refusal | "You can say you don't know" + standard phrasing |
| 4 | Clear Process | "Step 1... Step 2..." |
| 5 | Absolute Boundaries | "Absolutely prohibited" "Strictly forbidden" |
| 6 | Fallback Mechanism | "In the following cases, must transfer to human agent" |
| 7 | Structured Format | XML tags or clear delimiters |

1. Click Save

---

Step 3: Create MCP Tool

The MCP tool allows the optimization assistant to query the MaiAgent API to retrieve conversation records and assistant configurations.

1. Left sidebar → Tools → Click Add Tool in the upper right corner

2. Switch the Tool Type from API to MCP

3. Fill in the following settings:

Field	Value
Display Name	MaiAgent API
Description	AI Response Analysis Tool
MCP Tool URL	https://mcp.maiagent.ai/mcp

1. After creation, go to the assistant settings → Tools tab to see the bound MCP tool

2. Authentication Headers (MCP Headers) — Enter the following JSON:

{
  "Authorization": "Api-Key {Your API Key}"
}

How to obtain an API Key?

Click the avatar in the upper right corner → Profile → Switch to the API Keys tab → Click Create New Key, and copy the generated API Key.

The API Key is only displayed once after creation. Copy and store it securely immediately.

1. Allowed Tools — Check the following 5:

Tool	Purpose
debug_auth	Verify authentication is correct
list_api_categories	List API categories
search_apis	Search API endpoints
get_api_details	Get API detailed specifications
call_api	Execute API calls

1. Tool Prompt — Paste the following content in full:

Click to expand: Standard Tool Prompt

MaiAgent platform API calling tool. Used to query conversation records, retrieve assistant configurations, and read knowledge base content.

## Core Operation Flow

### Step 1: Find recordId from conversation_id
call_api(operation_id: 'v1MessagesList', query_params: { conversation: '<conversation_id>', pageSize: 50 })

### Step 2: Get conversation record details
call_api(operation_id: 'v1RecordsRetrieve', path_params: { id: '<record_id>' })

### Step 3: Get the analyzed assistant's Prompt
call_api(operation_id: 'v1ChatbotsRetrieve', path_params: { id: '<chatbot_id>' })

### Step 4: Get Skill instructions (if skills exist)
call_api(operation_id: 'v1SkillsRetrieve', path_params: { id: '<skill_id>' })

### Step 5 (Optional): Get TextNode details
call_api(operation_id: 'v1ChatbotTextNodesRetrieve', path_params: { id: '<textnode_id>' })

Steps 1-4 must be executed sequentially; each step's output is the next step's input.

1. Click Save

---

Step 4: Bind Knowledge Base

The optimization assistant needs to bind the knowledge base used by the target assistant to verify RAG retrieval results during diagnosis.

1. Switch to the Knowledge Base Settings tab

2. Click Add Knowledge Base and check all knowledge bases used by the target assistant

3. Click Save

If you need to analyze multiple AI assistants, bind all their knowledge bases to the optimization assistant. It is recommended to set the retrieval chunks count to 12 in "Advanced Settings".

---

Step 5: Verify Setup

Verify MCP Tool Connection

In the optimization assistant's chat box, enter:

Please verify if the MCP tool connection is working properly

The optimization assistant should successfully call debug_auth and return authentication information.

Verify Diagnostic Functionality

Obtain a conversation_id from the target assistant's conversation records:

1. Go to the All Conversations page

2. Click any conversation; the ID in the URL is the conversation_id

In the optimization assistant, enter:

Please analyze this conversation: {conversation_id}

The optimization assistant will automatically execute the following steps in order:

1. Search for available API tools (search_apis)

2. Call v1MessagesList to query conversation messages

3. Call v1RecordsRetrieve to get conversation record details

4. Call v1ChatbotsRetrieve to get the analyzed assistant's Prompt

5. Produce a complete diagnostic summary and modification recommendations

---

Usage Guide

Analyze by conversation_id

Please analyze this conversation: abc123-def456-ghi789

The optimization assistant will automatically retrieve conversation records, AI responses, knowledge base retrieval results, and assistant prompts to produce a complete diagnostic report.

Analyze by Pasting Text Directly

If you don't have a conversation_id, you can provide the information directly:

Please help me analyze the following issue:

[User Question] How do I apply for credit card loss reporting?
[AI Response] (paste the AI's actual response)
[Problem Description] The AI mentioned a non-existent "online loss reporting" feature

Interpreting Diagnostic Results

Diagnosis Type	Meaning	Next Steps
Prompt Problem	AI received the right data but the response approach is wrong	Modify role instructions or Skills per recommendations
Knowledge Base Document Problem	Document structure caused AI to retrieve incorrect content	Rewrite knowledge base documents per recommendations
Knowledge Base Gap	Knowledge base simply doesn't have relevant content	New knowledge base documents need to be added

---

FAQ

Q: What should I do if the MCP tool connection fails?

1. Confirm the response mode is Agent Mode (most common cause)

2. Confirm the API Key is valid and not expired

3. Confirm the MCP URL is https://mcp.maiagent.ai/mcp

4. Confirm Headers use the Api-Key format (e.g., Api-Key xxxxxxxx.xxxxxxxx)

Q: Can one optimization assistant analyze multiple assistants? Yes. As long as the API Key belongs to the same organization and all target assistants' knowledge bases are bound.

Q: Do I need to create one for each client? Yes. Each client organization needs its own optimization assistant, since API Keys and knowledge bases are organization-specific.

---

Setup Checklist

• Create AI Assistant (Name: Optimization Assistant)

• Select model (Claude Sonnet 4.6 or above)

• Set to Agent Mode

• Paste standard role instructions

• Create MCP tool

  • MCP URL: https://mcp.maiagent.ai/mcp

  • Headers: API Key

  • Check 5 Allowed Tools

  • Paste standard Tool Prompt

• Bind target assistant's knowledge base

• Verify MCP connection

• Verify diagnostic functionality