> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/maiagent-user-guide/en/build/agent-ops/test-datasets.md).

# Test Dataset Management

Create test datasets, systematically manage test cases, and verify Agent response quality

## What Is a Test Dataset? <a href="#what-is-test-dataset" id="what-is-test-dataset"></a>

A test dataset is a collection of prepared test cases used to verify the quality and accuracy of Agent responses. By creating standardized test datasets, you can:

**Ensure Response Quality**

* Verify that Agent responses to common questions meet expectations
* Ensure that responses do not degrade after knowledge base updates
* Continuously track trends in response quality

**Improve Efficiency**

* Test multiple questions in batches without manually checking each one
* Create reusable test cases to save testing time
* Quickly identify issues and make timely adjustments

**Team Collaboration**

* Manage all test cases in one place
* Let team members maintain test data together
* Track who created each test case and when it was updated

## How to Access Test Dataset Management <a href="#how-to-access-test-dataset-management" id="how-to-access-test-dataset-management"></a>

Open "<mark style="color:blue;">**AgentOps**</mark>" in the left sidebar and click "<mark style="color:blue;">**Test Datasets**</mark>" to view the test dataset management page.

The page displays the following information:

* **Test Dataset ID**: A unique identifier automatically generated by the system
* **Test Dataset Name**: The name you assign to the dataset
* <mark style="color:blue;">Single-Turn Test Case Count</mark>: The number of enabled single-turn cases in this dataset
* <mark style="color:blue;">Multi-Turn Test Case Count</mark>: The number of enabled multi-turn cases in this dataset, rather than the number of conversation turns
* **Created By**: The user who created the dataset
* **Created At**: The date and time the dataset was created
* **Last Updated**: The time the dataset was last modified
* **Action Buttons**: Functions such as editing and deleting

## Use Case: Checking Case Counts <a href="#check-case-counts" id="check-case-counts"></a>

A customer service supervisor wants to confirm that the question bank covers both single question-and-answer interactions and service scenarios that require back-and-forth confirmation. She opens <mark style="color:blue;">AgentOps</mark> → <mark style="color:blue;">Test Datasets</mark>, finds the dataset to check, and looks at <mark style="color:blue;">Single-Turn Test Case Count</mark> and <mark style="color:blue;">Multi-Turn Test Case Count</mark> separately. This shows how many enabled cases each type contains, helping her decide whether to add more questions.

<figure><img src="https://1360999650-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F6v6TNkkOQVfRYfcNirHL%2Fuploads%2Fgit-blob-e7bc63ad313c5f8fdc217c59f8a2ae9016a7279d%2Ftest-datasets-case-counts.png?alt=media" alt="The test dataset list shows a single-turn test case count of 3 and a multi-turn test case count of 2 in separate columns"><figcaption><p>The two case types are counted in separate columns; the multi-turn count of 2 in the image means two cases, rather than two conversation turns.</p></figcaption></figure>

{% hint style="info" %}
Both columns count only enabled cases. Disabled cases are excluded; when a type has no enabled cases, its column displays 0.
{% endhint %}

## Create a Test Dataset <a href="#create-test-dataset" id="create-test-dataset"></a>

### 1. Start Creating <a href="#start-creating" id="start-creating"></a>

Click the "<mark style="color:blue;">**Create Test Dataset**</mark>" button in the upper-right corner of the page to start creating a new test dataset.

### 2. Set Basic Information <a href="#set-basic-info" id="set-basic-info"></a>

In the dialog that appears, enter the following information:

**Test Dataset Name** (Required)

* Use a meaningful name, such as "Product Inquiry Test Dataset" or "Customer Service FAQ Tests"
* Make it easy to manage and identify the dataset's purpose later

**Test Dataset Description** (Optional)

* Describe the purpose and scope of this test dataset
* For example: "Tests the product specification inquiry feature, covering 50 common product questions"

<mark style="color:blue;">Select AI Assistant</mark> (Required)

* Select the Agent to test
* A test dataset can be associated with one or more Agents

### 3. Add Test Cases <a href="#add-test-cases" id="add-test-cases"></a>

After creating the dataset, open it, click "<mark style="color:blue;">Test Cases</mark>" on the left, and then click "<mark style="color:blue;">Add Test Case</mark>". The first field in the dialog, "<mark style="color:blue;">Select Creation Method</mark>", offers three options:

| Creation Method                                                     | Suitable Scenario                                            | Description                                                              |
| ------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------ |
| <mark style="color:blue;">Manual Entry</mark>                       | Adding one or two questions at a time                        | Enter the question and expected answer yourself                          |
| <mark style="color:blue;">Import</mark>                             | You already have an organized question bank                  | Upload CSV／XLS／XLSX files to import multiple questions at once           |
| <mark style="color:blue;">Generate from Knowledge Base Files</mark> | You just uploaded a knowledge base and have no questions yet | AI automatically creates questions based on knowledge base file contents |

Each test case includes:

**Question**

* A question users might ask (up to 1000 characters)
* Include a variety of phrasings and scenarios

**Expected Answer**

* How the Agent should respond (up to 2000 characters)
* Can be keywords, sentences, or complete paragraphs
* During evaluation, the evaluation model compares the actual response with the expected answer

#### Manual Entry <a href="#add-manually" id="add-manually"></a>

Select "<mark style="color:blue;">Manual Entry</mark>", fill in "<mark style="color:blue;">Question</mark>" and "<mark style="color:blue;">Expected Answer</mark>", and click "<mark style="color:blue;">Confirm</mark>". For the "<mark style="color:blue;">Flaky</mark>" toggle below, see [Mark Flaky Test Cases](#flaky-test-cases).

#### Import from Files <a href="#import-from-file" id="import-from-file"></a>

Select "<mark style="color:blue;">Import</mark>" and drag files into the "<mark style="color:blue;">Upload Files</mark>" area or click to select them. You can upload multiple files at once:

* Supported formats: CSV, XLS, XLSX; each file must not exceed 10MB
* The first row must contain column headers, including `question` (question) and `ground_truth` (expected answer); other columns are ignored
* After a successful import, "Test cases imported successfully" appears, and the corresponding number of cases is added to the list; imported cases show Import under "<mark style="color:blue;">Creation Method</mark>"

```
question,ground_truth
What is your return policy?,You can return items unconditionally within 7 days of purchase,items must remain new and unused
How do I request a refund?,Please request a refund through the order management page in the member center
```

#### Generate from Knowledge Base Files <a href="#generate-from-knowledge" id="generate-from-knowledge"></a>

Select "<mark style="color:blue;">Generate from Knowledge Base Files</mark>" to have AI read the knowledge base files you specify and automatically generate "question + expected answer" pairs:

1. Enter the number of questions to generate (1–50) in "<mark style="color:blue;">Number of Questions to Add</mark>".
2. Click "<mark style="color:blue;">Select Knowledge Base Files</mark>". In the dialog, search for and select a knowledge base on the left, check the files to use on the right (up to 50 files; the progress bar above shows how many are selected), and click "<mark style="color:blue;">Confirm Selection</mark>". Only fully processed documents can be selected; unsupported file types are automatically filtered out.
3. Click "<mark style="color:blue;">Confirm</mark>". The system displays "Test case generation task submitted and processing in the background", and the progress "Generating test cases (N/M questions generated)" appears above the test case list. Progress also appears in the "Single-Turn Test Case Count" column of the dataset list. Refresh after completion to see the new questions, whose "Creation Method" is Generate from Knowledge Base Files.

{% hint style="info" %}
The requested number of questions is the upper limit; fewer questions may be generated if the file contents are insufficient. After generation completes, review each question and edit expected answers or remove unsuitable questions as needed—automatic generation is a starting point, not the final answer.
{% endhint %}

**Generation History and Generation Details**

"<mark style="color:blue;">Generation History</mark>" in the upper-right corner of the test case list shows each automatic generation task for this dataset:

| Field                                               | Description                                                 |
| --------------------------------------------------- | ----------------------------------------------------------- |
| <mark style="color:blue;">Status</mark>             | Pending／Generating／Completed／Partially Completed／Failed     |
| <mark style="color:blue;">Actual / Requested</mark> | Number of questions generated／Number of questions requested |
| <mark style="color:blue;">Error Summary</mark>      | A summary of the cause of failure or partial completion     |
| <mark style="color:blue;">Created At</mark>         | The time the task was submitted                             |

Click "<mark style="color:blue;">View</mark>" in the action column to open "<mark style="color:blue;">Generation Details</mark>". You can see the number of "<mark style="color:blue;">Processed Files</mark>" and "<mark style="color:blue;">Results by File</mark>": each file's status, "<mark style="color:blue;">Generated Count</mark>", and "<mark style="color:blue;">Error Reason</mark>". A "Partially Completed" status usually means some files could not be parsed; the results by file identify which ones.

#### Mark Flaky Test Cases <a href="#flaky-test-cases" id="flaky-test-cases"></a>

When adding or editing an individual test case, you can enable the "<mark style="color:blue;">Flaky</mark>" toggle. During automatic evaluation, cases marked as flaky **still run and retain their results, but are excluded from the pass rate** (both numerator and denominator). This is useful for questions with multiple reasonable answers or unstable results due to dependencies on external live data: keep them to observe trends without lowering the overall pass rate. Metric statistics on the evaluation results page cover all completed cases, including flaky ones.

#### Batch Delete <a href="#batch-delete-test-cases" id="batch-delete-test-cases"></a>

After selecting multiple entries in the test case list, "<mark style="color:blue;">Batch Delete (N)</mark>" appears above the list. Confirm to delete all selected cases at once.

**Test Case Examples:**

```
Question: What is your return policy?
Expected response: You can return items unconditionally within 7 days of purchase,items must remain new and unused

Question: How do I request a refund?
Expected response: Please request a refund through the order management page in the member center,refunds are issued within 3-5 business days after approval

Question: Who pays for return shipping?
Expected response: We cover shipping for defective products,the buyer covers shipping for returns for personal reasons
```

### 4. Multi-Turn Conversation Test Cases <a href="#multi-turn-test-cases" id="multi-turn-test-cases"></a>

Single-turn test cases check a "single question and answer". To test tasks that require back-and-forth confirmation (such as asking for an order number before confirming the reason for a return), click "<mark style="color:blue;">Multi-Turn Conversation Test Cases</mark>" on the left side of the dataset. These cases are only used by evaluations of the "Multi-Turn Conversation" type; see [Evaluate Multi-Turn Conversations](/maiagent-user-guide/en/build/agent-ops/evaluations.md#multi-turn-evaluation).

Click "<mark style="color:blue;">Add Multi-Turn Test Case</mark>" and fill in:

| Field                                                              | Description                                                                                                                                                                                                        |
| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| <mark style="color:blue;">Scenario</mark> (Required)               | The task scenario the simulated user wants to accomplish, such as "The user wants to check last month's orders and request a return"                                                                               |
| <mark style="color:blue;">Expected Outcome</mark> (Required)       | The outcome that should be achieved in a successful conversation, which also serves as the condition for stopping the simulation, such as "Successfully provide the return procedure and confirm the order number" |
| <mark style="color:blue;">Simulated User Persona</mark> (Optional) | The simulated user's tone, background, and expertise, such as "An older user unfamiliar with online shopping, polite but easily confused"                                                                          |
| <mark style="color:blue;">Status</mark>                            | <mark style="color:blue;">Enabled</mark>／<mark style="color:blue;">Disabled</mark>; disabled cases are not run during evaluation                                                                                   |
| <mark style="color:blue;">Flaky</mark>                             | As with single-turn cases, marked cases are excluded from the pass rate                                                                                                                                            |

You can also click "<mark style="color:blue;">Generate from Knowledge Base</mark>", enter the number of questions, and select knowledge base files. AI synthesizes multi-turn cases with "scenario + expected outcome" based on file contents. The process is the same as [Generate from Knowledge Base Files](#generate-from-knowledge) for single-turn cases; refresh the list after completion.

## Manage Test Datasets <a href="#manage-test-datasets" id="manage-test-datasets"></a>

### Search for Test Datasets <a href="#search-test-dataset" id="search-test-dataset"></a>

Use the search bar at the top of the page to quickly find a dataset by:

* Test dataset ID
* Test dataset name
* Keywords in the description

After you enter keywords, the system immediately filters and displays matching datasets.

### Edit Test Datasets <a href="#edit-test-dataset" id="edit-test-dataset"></a>

Click the "<mark style="color:blue;">**Edit**</mark>" button in the action column to:

* Modify the test dataset name and description
* Add, edit, or delete test cases
* Adjust the order of test cases

{% hint style="info" %}
Review and update test cases regularly to ensure they stay aligned with actual business needs
{% endhint %}

### Delete Test Datasets <a href="#delete-test-dataset" id="delete-test-dataset"></a>

Click the "<mark style="color:blue;">**Delete**</mark>" button in the action column to delete test datasets you no longer need.

{% hint style="warning" %}
Deleting a test dataset is irreversible, so proceed carefully. If the dataset has been used for automatic evaluation, the associated evaluation records are retained, but the same test cannot be run again.
{% endhint %}

## Test Dataset Best Practices <a href="#test-dataset-best-practices" id="test-dataset-best-practices"></a>

### 1. Create Structured Test Categories <a href="#structured-test-categories" id="structured-test-categories"></a>

Group test cases according to business needs:

```
Product knowledge base tests:
├── Product Inquiry Test Dataset (50 cases)
├── Pricing Inquiry Test Dataset (30 cases)
└── Inventory Check Test Dataset (20 cases)

Customer service workflow tests:
├── Order Processing Test Dataset (40 cases)
├── Returns and Exchanges Test Dataset (35 cases)
└── Member Services Test Dataset (25 cases)
```

### 2. Cover Multiple Phrasings <a href="#cover-multiple-phrasings" id="cover-multiple-phrasings"></a>

The same concept can be phrased in different ways. Include the following in your test cases:

```
Formal: Could you please tell me your business hours?
Conversational: What time do you open?
Brief: Business hours
Full sentence: I'd like to know what time your business hours start and end
```

### 3. Include Edge Cases <a href="#include-edge-cases" id="include-edge-cases"></a>

Test extreme situations and ambiguous questions:

* Incomplete questions: "Price", "What should I do"
* Questions with typos: "Product specifcations"
* Compound questions: "How much does it cost? Can I return it? What colors are available?"
* Questions outside the knowledge scope: Test how the Agent responds

### 4. Update Test Data Regularly <a href="#regular-data-updates" id="regular-data-updates"></a>

* **When new features launch**: Add corresponding test cases
* **After knowledge base updates**: Check whether related test cases need adjustments
* **When an issue is discovered**: Immediately add a test case for that scenario to prevent recurrence
* **Regular reviews**: Review test cases monthly or quarterly to ensure they still reflect current conditions

### 5. Set an Appropriate Number of Tests <a href="#appropriate-test-count" id="appropriate-test-count"></a>

Each test dataset should contain:

* **At least 20 test cases**: Cover basic scenarios
* **Recommended 30-50 test cases**: Balance test coverage and execution time
* **100+ cases for special situations**: Suitable for complex business logic

## Frequently Asked Questions <a href="#faq" id="faq"></a>

### Q1: How Are Test Datasets Related to Automatic Evaluation? <a href="#faq-test-dataset-vs-auto-evaluation" id="faq-test-dataset-vs-auto-evaluation"></a>

Test datasets are the foundation of automatic evaluation. Create a test dataset first, then select the dataset and Agent in the "Automatic Evaluation" feature. The system automatically runs the tests and generates an evaluation report.

> For more information, see: [Automatic Evaluation](/maiagent-user-guide/en/build/agent-ops/evaluations.md)

### Q2: Can One Test Dataset Test Multiple Agents? <a href="#faq-one-dataset-multiple-agents" id="faq-one-dataset-multiple-agents"></a>

Yes. During automatic evaluation, you can use the same test dataset with different Agents to compare their performance on identical test cases.

### Q3: Do Expected Responses Need to Match Exactly? <a href="#faq-expected-response-exact-match" id="faq-expected-response-exact-match"></a>

No. The system checks whether the Agent's actual response "contains" the key content of the expected response. You can specify keywords or important sentences; the response passes as long as it contains that content.

### Q4: How Do I Import a Large Number of Test Cases? <a href="#faq-bulk-import-test-cases" id="faq-bulk-import-test-cases"></a>

Select "<mark style="color:blue;">Import</mark>" in "Add Test Case" and upload a CSV／XLS／XLSX file containing the `question` and `ground_truth` columns; see [Import from Files](#import-from-file). If you do not have a question bank yet, use [Generate from Knowledge Base Files](#generate-from-knowledge) to let AI create initial questions.

### Q5: Can Test Datasets Be Shared with Other Organization Members? <a href="#faq-share-dataset-with-members" id="faq-share-dataset-with-members"></a>

Yes. All members of the same organization can view and use test datasets. The "Created By" column identifies who created each dataset, making team collaboration easier.

### Q6: Do Related Test Datasets Disappear When an Agent Is Deleted? <a href="#faq-dataset-after-agent-deleted" id="faq-dataset-after-agent-deleted"></a>

No. Test datasets are managed independently and remain even if an Agent is deleted. You can use the same dataset to test other Agents.

***

## Next Steps <a href="#next-steps" id="next-steps"></a>

After creating a test dataset, you can:

* Go to [Automatic Evaluation](/maiagent-user-guide/en/build/agent-ops/evaluations.md) to run tests and view results
* Track actual Agent performance through [Agent Monitoring](/maiagent-user-guide/en/build/agent-ops/evaluations.md#ai-agent-monitoring)
* Adjust knowledge base content or Agent settings based on evaluation results


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.maiagent.ai/maiagent-user-guide/en/build/agent-ops/test-datasets.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.