Tool Execution Records

Overview

Tool execution records let you track the complete history of an AI assistant's external tool invocations, including API calls, database queries, web scraping operations, and more. When an AI assistant needs to perform tasks beyond the scope of conversation, it uses tools to do so. Tool execution records help you understand tool usage, identify causes of execution failures, and optimize tool configurations.

With comprehensive execution records, you can identify the root cause of tool failures within 5 minutes, reducing troubleshooting time by 80% and ensuring the AI assistant can reliably deliver advanced functionality.

Features

With the tool execution records feature, you can:

Complete Execution History

The system automatically records every tool invocation, including input parameters, execution results, duration, and success or failure status.

• Use case: An e-commerce platform's order query tool is invoked 500 times daily and needs success rate and response time tracking

• Actual benefit: Discovered 10% of queries failed due to timeouts; after adjusting timeout settings, the success rate improved to 98%

Quickly Identify Failure Causes

When a tool execution fails, the system records detailed error messages to help you quickly understand what went wrong.

• Use case: A weather query API suddenly starts failing frequently and the cause needs to be identified quickly

• Actual benefit: Error records revealed an expired API key; service was restored within 5 minutes of updating it

Tool Usage Statistics

View invocation counts, success rates, average execution times, and other statistics for each tool.

• Use case: An enterprise has configured 10 tools and wants to know which are used most frequently and which have the most issues

• Actual benefit: Discovered 3 tools were never used; removing them reduced system complexity, and other tools saw a 15% improvement in response speed

Optimize Tool Configurations

Based on execution record analysis, adjust tool parameters, optimize invocation logic, and improve error handling.

• Use case: A database query tool frequently times out because the query scope is too large

• Actual benefit: After adjusting query limits based on record analysis, average execution time dropped from 8 seconds to 3 seconds

Tool Types

MaiAgent supports multiple tool types, and tool execution records track the execution status of all types:

API Tools

Call external APIs to retrieve real-time data, such as weather queries, exchange rate queries, inventory queries, etc.

Common execution issues:

• Expired or invalid API keys

• Incorrect request parameter formats

• API service timeout or no response

• Exceeded API call limits

Text-to-SQL Tools

Convert natural language questions into SQL queries to retrieve data from databases.

Common execution issues:

• SQL syntax errors

• Query scope too large causing timeouts

• Database connection failures

• Insufficient permissions to access specific tables

Web Scraping Tools

Scrape real-time information from specified websites, such as latest news, product prices, etc.

Common execution issues:

• Target website unreachable

• Web page structure changes causing parsing failures

• Blocked or rate-limited by the target website

• Scraping timeout

Custom Tools

Enterprise-developed tools for specific functions, such as order processing, member verification, etc.

Common execution issues:

• Internal service errors

• Parameter validation failures

• Business logic errors

• System integration issues

Usage Steps

Step 1: Access Tool Execution Records

1. Enter the admin dashboard and click AgentOps → Tool Execution Records in the left sidebar

2. View the list of recent tool execution records

Step 2: Filter Records

Use filter criteria to find execution records that need attention:

1. Time range:

   • Today, Last 7 days, Last 30 days

   • Or custom start and end dates

2. Tool type:

   • All tools

   • API tools

   • Text-to-SQL

   • Web scraping tools

   • Custom tools

3. Execution status:

   • All

   • Success

   • Failure

   • In progress

4. Specific tool:

   • Select a specific tool name from the dropdown

5. Click "Apply Filters" to view matching records

Priority Items to Check

Recommended priorities:

1. Failed execution records

2. Records with execution time exceeding 10 seconds

3. Execution status of frequently used tools

Step 3: View Execution Details

1. Click any record in the list

2. Expand the detailed information, including:

Basic Information:

• Tool name

• Execution time (when it was invoked)

• Execution duration (how many seconds it took)

• Execution status (Success/Failure)

Input Parameters:

• Parameter content passed by the AI to the tool

• Check whether parameters are correct

Execution Result:

• Complete result returned by the tool

• Shows retrieved data on success

• Shows error message on failure

Associated Conversation:

• The conversation record that triggered this tool invocation

• Trace back to the user's original question

Error Message (if any):

• Detailed error description

• Error code or exception type

• Stack trace information (if applicable)

Step 4: Analyze Failure Causes

When you find failed execution records, follow these steps to analyze:

1. Check the error message:

   • Read the detailed error description

   • Identify whether it's a configuration issue, permission issue, or service error

2. Check input parameters:

   • Confirm whether the parameters passed by the AI are correct

   • Verify the parameter format meets tool requirements

3. Review the associated conversation:

   • Understand what question the user asked

   • Determine whether a specific question type caused the failure

4. Compare with successful cases:

   • Find successful execution records for the same tool

   • Compare the differences between successful and failed cases

Step 5: Optimize Tool Configuration

Based on execution record analysis, take improvement actions:

1. Update tool settings:

   • Correct incorrect API keys or connection information

   • Adjust timeout settings

   • Optimize query parameters or constraints

2. Adjust AI prompts:

   • If parameters are frequently incorrect, add clear guidance in the prompt

   • Explain the correct usage and parameter format of the tool

3. Improve error handling:

   • Add user-friendly error messages for common error scenarios

   • Set up automatic retry mechanisms

4. Remove unused tools:

   • If a tool has not been used for a long time, consider removing it to simplify the configuration

Step 6: Export Execution Records

1. Click the "Export" button on the tool execution records page

2. Select the export format (CSV or Excel)

3. Select the time range and filter criteria

4. Download the report for further analysis or archiving

Exported records can be used for:

• In-depth data analysis and trend tracking

• Reporting issues to the technical team

• Regular reviews of tool usage efficiency

• Compliance auditing and record keeping

Use Cases

Case 1: Diagnosing Tool Failures

Scenario: An e-commerce platform's inventory query tool suddenly starts failing in large numbers, and customer service cannot check product inventory for customers.

Steps:

1. Go to tool execution records and filter by "Inventory Query Tool" + "Failure"

2. Discover 50 failed records in the past 2 hours

3. View the error message in the failed records: "Database connection timeout"

4. Check the database service status and discover the database is under maintenance

5. Temporarily disable the tool and notify customer service to use manual queries

6. Re-enable after database recovery and monitor the success rate returning to normal

Quantified Benefits:

• Identified root cause within 5 minutes

• Avoided hours of troubleshooting by the technical team

• Promptly notified customer service to use backup methods, reducing customer wait times

Case 2: Optimizing Tool Performance

Scenario: The order query tool frequently runs slowly, affecting the conversation experience.

Steps:

1. Export the order query tool's execution records from the past 30 days

2. Analyze the execution time distribution and find:

   • Average execution time: 7.5 seconds

   • 25% of queries exceed 10 seconds

   • Maximum: 18 seconds

3. Review the input parameters of slow queries and find most are querying "all orders from the past year"

4. Optimize the tool configuration:

   • Limit query scope to the past 3 months

   • Add pagination with a maximum of 20 records per request

5. After optimization, average execution time drops to 3.2 seconds

Quantified Benefits:

• Execution time reduced by 57% (7.5 seconds → 3.2 seconds)

• Queries exceeding 10 seconds dropped from 25% to 5%

• Significantly improved conversation experience and customer satisfaction

Case 3: Discovering Unused Tools

Scenario: An enterprise configured 12 tools for the AI assistant and wants to understand actual usage to simplify the configuration.

Steps:

1. Export all tool execution records from the past 90 days

2. Count each tool's invocation count:

   • Order query: 1,500 times

   • Member query: 800 times

   • Promotion query: 300 times

   • Product specification query: 200 times

   • Other 8 tools: 0 times

3. Review the configuration of unused tools and find:

   • Some tools have unclear definitions, so the AI doesn't know when to use them

   • Some tools have overlapping functionality

   • Some tools are outdated

4. Remove 6 unused tools and optimize the descriptions of 2 low-usage tools

Quantified Benefits:

• Simplified configuration, improving AI tool selection accuracy by 20%

• System response speed improved by 10% (reduced tool selection judgment time)

• Lower maintenance costs

Case 4: Monitoring API Usage

Scenario: An enterprise uses a third-party weather API with a monthly invocation limit and needs to monitor usage to avoid exceeding the quota.

Steps:

1. Export the weather query tool's execution records every Monday

2. Track the invocation count trend:

   • Week 1: 320 times

   • Week 2: 450 times

   • Week 3: 680 times

   • Week 4: Estimated 900 times (monthly limit: 2,000 times)

3. Discover rapidly growing invocation volume and analyze the cause:

   • Increased frequency of customer weather inquiries

   • Some unnecessary duplicate queries

4. Take action:

   • Add a caching mechanism to avoid duplicate queries for the same city within 1 hour

   • Optimize the prompt to prevent the AI from over-relying on the weather tool

5. After optimization, weekly invocations drop to 400, ensuring the quota is not exceeded

Quantified Benefits:

• Avoid exceeding the API quota, which would cause service disruption

• 40% reduction in API invocation costs

• Maintain service quality while controlling costs

Frequently Asked Questions

Q: How long are tool execution records retained?

A: The system retains tool execution records for 3 months by default. It is recommended to regularly export important data for long-term storage and analysis. For longer retention periods, contact customer service to upgrade your plan.

Q: Why did a tool fail but the conversation seems normal?

A: When a tool execution fails, the AI assistant may:

1. Tell the user "Unable to query at this time, please try again later"

2. Use knowledge base information to answer instead (if related content exists)

3. Ask the user for more information and retry

Although the conversation continues, functionality is limited. It is recommended to track failure causes through execution records and fix them to ensure stable tool operation.

Q: How can I reduce the tool execution failure rate?

Recommended optimization approaches:

Configuration level:

• Ensure API keys, database connections, and other configurations are correct

• Set reasonable timeout periods (recommended 10-15 seconds)

• Add error retry mechanisms

Prompt level:

• Clearly describe when to use tools and the parameter format

• Add examples to help the AI invoke tools correctly

Monitoring level:

• Regularly check execution records

• Promptly identify and fix issues

• Track success rate trends

Q: What if input parameters in execution records contain sensitive data?

A: The system automatically masks common sensitive data (such as passwords and keys). If your tools handle other sensitive data (such as customer personal information), it is recommended to:

1. Enable the "Parameter Masking" feature in tool configuration

2. Control who can view execution records, granting access only to authorized personnel

3. Regularly clean up expired execution records

Q: Can I re-execute a failed tool invocation?

A: The system does not currently support directly re-executing historical records. To test a tool, it is recommended to:

1. Ask a question in the chat platform to trigger a tool invocation

2. Use the tool testing feature (if available)

3. Fix the issue and wait for actual users to trigger it

Q: How can I see which tools were invoked in a specific conversation?

A: There are two ways:

1. From conversation records: The conversation details show all tools invoked during that conversation

2. From tool records: Click the "Associated Conversation" link in the tool execution records

Best Practices

Regular Check Recommendations

• Daily check: Review failed records from the past 24 hours to ensure critical tools are operating normally

• Weekly review: Count each tool's invocation count and success rate to identify items needing optimization

• Monthly analysis: Export a complete report to analyze long-term trends and usage patterns

Alert Configuration Recommendations

Recommended alert settings (if supported by the system):

• A tool fails more than 10 times within 1 hour

• A tool's success rate drops below 80%

• A tool's average execution time exceeds 10 seconds

• A tool has not been used for more than 7 days (check if it's still needed)

Documentation Recommendations

Create usage documentation for each tool, recording:

• Tool purpose and applicable scenarios

• Input parameter descriptions and examples

• Expected output format

• Common errors and solutions

• Optimization history