Test Dataset Management

Create test datasets, systematically manage test cases, and validate AI assistant response quality

What Is a Test Dataset?

A test dataset is a collection of pre-prepared test cases used to validate the response quality and accuracy of AI assistants. By creating standardized test datasets, you can:

Ensure Response Quality

• Verify that the AI assistant's responses to common questions meet expectations

• Ensure responses don't degrade after knowledge base updates

• Continuously track response quality trends

Improve Efficiency

• Batch test multiple questions without manual verification one by one

• Create reusable test cases to save testing time

• Quickly identify issues and make timely adjustments

Team Collaboration

• Centrally manage all test cases

• Different team members can jointly maintain test data

• Track who created and last updated test cases

How to Access Test Dataset Management

Go to "AgentOps" in the left sidebar, then click "Test Datasets" to see the test dataset management page.

The page displays the following information:

• Test Dataset ID: A unique identifier automatically generated by the system

• Test Dataset Name: The name you assigned to the dataset

• Test Case Count: The total number of test cases in the dataset

• Created By: The user who created this dataset

• Created At: The date and time the dataset was created

• Last Updated: The time the dataset was last modified

• Action Buttons: Edit, delete, and other functions

Creating a Test Dataset

1. Start Creating

Click the "Create Dataset" button in the upper right corner to start creating a new test dataset.

2. Set Basic Information

In the dialog that appears, enter the following information:

Test Dataset Name (required)

• Use a meaningful name, such as: "Product Query Test Set" or "Customer Service FAQ Test"

• This makes it easier to manage and identify purposes later

Test Dataset Description (optional)

• Describe the purpose and coverage of this test set

• Example: "For testing product specification query functionality, containing 50 common product questions"

Select AI Assistant (required)

• Choose the AI assistant to test

• A test dataset can correspond to one or more AI assistants

3. Add Test Cases

After creating the test dataset, you need to add test cases. Each test case contains:

Test Question

• Questions that users might ask

• Recommended to cover various phrasings and scenarios

Expected Response

• How the AI assistant should answer

• Can be keywords, sentences, or complete paragraphs

• The system compares whether the actual response contains the expected content

Test Case Examples:

Question: What is your return policy?
Expected Response: Unconditional returns within 7 days of purchase, items must remain new and unused

Question: How do I apply for a refund?
Expected Response: Apply for a refund through the order management page in the member center, refund will be processed within 3-5 business days after approval

Question: Who pays for return shipping?
Expected Response: We cover shipping for defective products, buyers cover shipping for personal preference returns

Managing Test Datasets

Search Test Datasets

Use the search bar at the top of the page to quickly find target datasets by:

• Test dataset ID

• Test dataset name

• Description keywords

After entering keywords, the system filters and displays matching datasets in real time.

Edit Test Datasets

Click the "Edit" button in the action column to:

• Modify the test dataset name and description

• Add, edit, or delete test cases

• Adjust the order of test cases

Regularly review and update test cases to ensure test content stays aligned with actual business needs

Delete Test Datasets

Click the "Delete" button in the action column to delete test datasets that are no longer needed.

Deleted test datasets cannot be recovered. Proceed with caution. If the dataset has been used in automated evaluations, the related evaluation records will be retained, but the same test cannot be run again.

Test Dataset Best Practices

1. Create Structured Test Categories

Group test cases by business needs:

Product-related knowledge base tests:
├── Product Query Test Set (50 cases)
├── Price Inquiry Test Set (30 cases)
└── Inventory Verification Test Set (20 cases)

Customer service workflow tests:
├── Order Processing Test Set (40 cases)
├── Return & Exchange Workflow Test Set (35 cases)
└── Membership Service Test Set (25 cases)

2. Cover Multiple Phrasings

The same concept can be phrased in different ways. Include various phrasings in your test cases:

Formal phrasing: Could you please tell me your business hours?
Colloquial phrasing: What time do you open?
Short phrasing: Business hours
Full sentence: I'd like to know what your business hours are, from when to when

3. Include Edge Cases

Test extreme situations and ambiguous questions:

• Incomplete questions: "price," "what do I do"

• Questions with typos: "prodcut specs"

• Compound questions: "How much does it cost? Can I return it? What colors are available?"

• Out-of-scope questions: Test how the AI assistant responds

4. Regularly Update Test Data

• When new features launch: Add corresponding test cases

• After knowledge base updates: Check whether related test cases need adjustment

• When issues are discovered: Immediately add test cases for that scenario to prevent recurrence

• Regular reviews: Review test cases monthly or quarterly to ensure they still reflect current conditions

5. Set Appropriate Test Quantities

Recommended test cases per dataset:

• Minimum 20 test cases: Ensures coverage of basic scenarios

• Recommended 30-50 test cases: Balances test coverage and execution time

• 100+ cases for special situations: Suitable for complex business logic

Frequently Asked Questions

Q1: What is the relationship between test datasets and automated evaluations?

Test datasets are the foundation of automated evaluations. You need to create a test dataset first, then select the test dataset and AI assistant in the "Automated Evaluation" feature. The system automatically runs the tests and generates an evaluation report.

For more information, see: Automated Evaluations

Q2: Can one test dataset be used to test multiple AI assistants?

Yes. You can select the same test dataset with different AI assistants during automated evaluation to compare how different AI assistants perform on the same test cases.

Q3: Does the expected response need to match exactly?

No. The system checks whether the AI assistant's actual response "contains" the key content of the expected response. You can set keywords or important sentences, and the test passes as long as the actual response includes that content.

Q4: How do I import a large number of test cases?

Currently, you can batch-add test cases by copying and pasting. Future versions will support Excel or CSV file import — stay tuned.

Q5: Can test datasets be shared with other organization members?

Yes. All members within the same organization can view and use test datasets. You can identify who created a dataset through the "Created By" column to facilitate team collaboration.

Q6: If I delete an AI assistant, will the related test datasets disappear?

No. Test datasets are managed independently. Even if you delete an AI assistant, the test datasets are retained. You can use the same test dataset to test other AI assistants.

---

Next Steps

After creating test datasets, you can:

• Go to Automated Evaluations to run tests and view results

• Use AI Assistant Monitoring to track the AI assistant's actual operational performance

• Adjust knowledge base content or AI assistant settings based on evaluation results