Evaluations and Test Sets
The evaluation and test set endpoints correspond to "Test Sets" and "Automated Testing" in the AgentOps back office, letting you batch-test Agents from your own systems. The full flow is:
Create a test set: Use
POST /api/v1/dataset/to create an empty test set, or usePOST /api/v1/dataset/upload/to upload a CSV/Excel file and create one in a single step.Add test cases: For single-turn Q&A, use
/api/v1/test-case/(create one by one, upload a file, or generate from a knowledge base file); for multi-turn conversation scenarios, use/api/v1/multi-turn-test-case/. Generating from a knowledge base file runs in the background and returns202with ajobId; track it with Retrieve test case generation job progress.Run an evaluation: Use
POST /api/v1/evaluations/evaluate/(single-turn) orPOST /api/v1/evaluations/multi-turn-evaluate/(multi-turn) to specify the test set and Agent. The response is202with anevaluationId.Read results and export: Poll
GET /api/v1/evaluations/{id}/untilstatusbecomescompleted,partial_completed, orfailed. Per-question results for single-turn evaluations are indetails, and per-conversation results for multi-turn evaluations are inmulti-turn-results. A completed single-turn evaluation can be exported as an Excel report withexport, and the download URL is available fromexport-status.
Required permissions: The API Key calls the API as the member who created it. Listing and all write operations on /dataset/, /test-case/, and /multi-turn-test-case/ require a role with the "Test Set permission"; listing and all write operations on /evaluations/ require the "Automated Testing permission" (the AgentOps permissions in Role Permissions; owners always have them). Insufficient permissions return 403. Members other than the owner can only see and evaluate Agents that their role is allowed to use.
Credit: Generating test cases from knowledge base files deducts the pricing item agentops_test_gen based on the number of questions actually generated. Running an evaluation deducts agentops_test_run and agentops_deepeval when it finishes, based on the number of completed test cases, and the evaluation model calls used for scoring are billed separately according to that model's usage. Before a generation or evaluation request is submitted, the system checks the organization's Credit and quota, and returns 402 if they are insufficient. Manual creation, file upload, and reading results do not deduct Credit; for unit prices, see List pricing items.
For features and back-office operations, see the user manual pages Test Set Management and Automated Evaluation and Agent Monitoring.
List Test Sets
GET /api/v1/dataset/
Parameters
isActive
❌
boolean
List only active (true) or inactive (false) datasets
ordering
❌
string
Sort field: created_at or updated_at. Prefix with - for newest first (e.g. -updated_at). If not specified, sorted by creation time, newest first
page
❌
integer
Page number, starting from 1. Returns 404 if beyond the last page
pageSize
❌
integer
Number of items per page. Default 20, maximum 100 (values above 100 are treated as 100)
query
❌
string
Keyword; matches the dataset name, description, or ID (partial match)
search
❌
string
Keyword; matches the dataset name or description (partial, case-insensitive match; when multiple keywords are separated by spaces, all must match)
Code Examples
Response Body
Status Code: 200
Response Schema Example
Example Response Values
Create an empty test set
POST /api/v1/dataset/
Request Content
Request Parameters
name
string
Yes
description
string
No
Example Request Structure
Example Request Values
Code Examples
Response Content
Status code: 201
Example Response Structure
Example Response Values
Upload CSV or Excel to create a test set
POST /api/v1/dataset/upload/
Code Examples
Response Content
Status code: 201
Example Response Structure
Example Response Values
Generate a test set from a knowledge base file
POST /api/v1/dataset/generate-from-file/
Request Content
Request Parameters
name
string
Yes
description
string
No
chatbotFileId
string (uuid)
No
knowledgeBaseFileId
string (uuid)
No
numTestCases
integer
No
Example Request Structure
Example Request Values
Code Examples
Response Content
Status code: 202
Example Response Structure
Example Response Values
List knowledge base files available for test case generation
GET /api/v1/dataset/knowledge-base-files/
Parameters
pageSize
❌
integer
Maximum number of files returned per knowledge base (default 5, maximum 100)
search
❌
string
Filter by filename (case-insensitive partial match)
Code examples
Response body
Status code: 200
Example response schema
Example response values
Retrieve a specific test set
GET /api/v1/dataset/{id}/
Parameters
id
✅
string
Dataset ID
Code Examples
Response Content
Status code: 200
Example Response Structure
Example Response Values
Retrieve a test set with its single-turn test cases
GET /api/v1/dataset/{id}/details/
Parameters
id
✅
string
Dataset ID
Code Examples
Response Content
Status code: 200
Example Response Structure
Example Response Values
Update a test set
PUT /api/v1/dataset/{id}/
Parameters
id
✅
string
Dataset ID
Request Content
Request Parameters
name
string
Yes
description
string
No
isActive
boolean
No
Example Request Structure
Example Request Values
Code Examples
Response Content
Status Code: 200
Example Response Structure
Example Response Values
Partially update a test set
PATCH /api/v1/dataset/{id}/
Parameters
id
✅
string
Dataset ID
Request Content
Request Parameters
name
string
No
description
string
No
isActive
boolean
No
Example Request Structure
Example Request Values
Code Examples
Response Content
Status Code: 200
Example Response Structure
Example Response Values
Delete a test set
DELETE /api/v1/dataset/{id}/
Parameters
id
✅
string
Dataset ID
Code Examples
Response Content
204
No response body
List Single-Turn Test Cases
GET /api/v1/test-case/
Parameters
creationMethod
❌
string
List only test cases created by this method. Allowed values: `manual`: Manual ; `file`: File import ; `generate`: Generated;
datasetId
❌
string
List only test cases in this dataset
ordering
❌
string
Sort field: created_at or updated_at. Prefix with - for newest first (e.g. -updated_at). If not specified, sorted by creation time, newest first
page
❌
integer
Page number, starting from 1. Returns 404 if beyond the last page
pageSize
❌
integer
Number of items per page. Default 20, maximum 100 (values above 100 are treated as 100)
query
❌
string
Keyword; matches the question, ground truth, or ID (partial match)
search
❌
string
Keyword; matches the question or ground truth (partial, case-insensitive match; when multiple keywords are separated by spaces, all must match)
status
❌
string
List only test cases with this status. Allowed values: `active`: Active ; `inactive`: Inactive ; `pending`: Pending Review;
Code Examples
Response Content
Status Code: 200
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Create a Single-Turn Test Case
POST /api/v1/test-case/
Request Content
Request Parameters
datasetId
string (uuid)
Yes
question
string
Yes
groundTruth
string
Yes
status
string
No
flaky
boolean
No
Example Request Structure
Example Request Values
Code Examples
Response Content
Status Code: 201
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Upload a CSV or Excel File to Add Test Cases
POST /api/v1/test-case/upload/
Code Examples
Response Content
Status Code: 201
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Generate Single-Turn Test Cases from Knowledge Base Files
POST /api/v1/test-case/generate-from-file/
Request Content
Request Parameters
chatbotFileIds
array[string]
No
List of KnowledgeBaseFile IDs to generate test cases from (deprecated)
knowledgeBaseFileIds
array[string]
No
List of KnowledgeBaseFile IDs to generate test cases from
knowledgeBaseIds
array[string]
No
List of KnowledgeBase IDs to generate test cases from all their files
datasetId
string (uuid)
Yes
numTestCases
integer
No
Example Request Structure
Example Request Values
Code Examples
Response Content
Status Code: 202
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
List Test Case Generation Jobs
GET /api/v1/test-case/generation-jobs/
Parameters
datasetId
❌
string
Only list generation jobs of this dataset
page
❌
integer
Page number, starting from 1; returns 404 if it exceeds the last page
pageSize
❌
integer
Number of items per page. Default 20, maximum 100 (values above 100 are treated as 100)
status
❌
string
Filter by status: pending / running / completed / partial_completed / failed
Code Examples
Response Body
Status Code: 200
Response Schema Example
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Retrieve Test Case Generation Job Progress
GET /api/v1/test-case/generation-jobs/{jobId}/
Parameters
jobId
✅
string
Generation job ID, i.e. the jobId returned by "Generate Single-Turn Test Cases from Knowledge Base Files"
Code Examples
Response Body
Status Code: 200
Response Schema Example
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Update a Single-Turn Test Case
PUT /api/v1/test-case/{id}/
Parameters
id
✅
string
Test case ID
Request Content
Request Parameters
question
string
Yes
groundTruth
string
No
status
string (enum: active, inactive, pending)
No
active: Active ; inactive: Inactive ; pending: Pending Review;
flaky
boolean
No
Flaky test cases still run and keep their results, but are excluded from the pass rate.
Example Request Structure
Example Request Values
Code Examples
Response Body
Status Code: 200
Response Schema Example
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Partially Update a Single-Turn Test Case
PATCH /api/v1/test-case/{id}/
Parameters
id
✅
string
Test case ID
Request Content
Request Parameters
question
string
No
groundTruth
string
No
status
string (enum: active, inactive, pending)
No
active: Active ; inactive: Inactive ; pending: Pending Review;
flaky
boolean
No
Flaky test cases still run and keep their results, but are excluded from the pass rate.
Example Request Structure
Example Request Values
Code Examples
Response Body
Status Code: 200
Response Schema Example
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Delete a Single-Turn Test Case
DELETE /api/v1/test-case/{id}/
Parameters
id
✅
string
Test case ID
Code Examples
Response Body
204
No response body
4XX
Error response; X-Error-Code carries its code (see "Error codes").
Bulk delete single-turn test cases
POST /api/v1/test-case/bulk-delete/
Request Content
Request Parameters
test_case_ids
array[string]
Yes
List of test case IDs to delete
Example Request Structure
Request Example Values
Code Examples
Response Body
204
No response body
4XX
Error response; X-Error-Code carries its code (see "Error codes").
List Multi-Turn Test Cases
GET /api/v1/multi-turn-test-case/
Parameters
dataset
❌
string
Same as datasetId. Lists only the multi-turn conversation test cases of this dataset
datasetId
❌
string
Lists only the multi-turn conversation test cases of this dataset
ordering
❌
string
Sort field: created_at or updated_at. Prefix with - for newest first (e.g. -updated_at). If not specified, results are sorted by creation time, newest first
page
❌
integer
Page number, starting from 1. Returns 404 if beyond the last page
pageSize
❌
integer
Number of items per page. Default 20, maximum 100 (values above 100 are treated as 100)
search
❌
string
Keyword. Matches the scenario or the expected outcome (expected_outcome) (partial match, case-insensitive; when multiple keywords are separated by spaces, all of them must match)
status
❌
string
Lists only multi-turn conversation test cases with this status. Allowed values: `active`: Active ; `inactive`: Inactive;
Code Examples
Response Body
Status Code: 200
Response Schema Example
Example Response Values
Create a multi-turn test case
POST /api/v1/multi-turn-test-case/
Request
Request Parameters
datasetId
string (uuid)
Yes
scenario
string
Yes
expectedOutcome
string
Yes
userDescription
string
No
status
object
No
flaky
boolean
No
Request Schema Example
Request Example Values
Code Examples
Response
Status code: 201
Response Schema Example
Response Example Values
Generate Multi-Turn Test Cases from Knowledge Base Files
POST /api/v1/multi-turn-test-case/generate-from-file/
Request Content
Request Parameters
datasetId
string (uuid)
Yes
knowledgeBaseFileIds
array[string]
Yes
List of KnowledgeBaseFile IDs to generate multi-turn test cases from
numTestCases
integer
No
Example Request Structure
Example Request Values
Code Examples
Response Content
Status Code: 202
Example Response Structure
Example Response Values
Retrieve a Specific Multi-Turn Test Case
GET /api/v1/multi-turn-test-case/{id}/
Parameters
id
✅
string
Multi-turn test case ID
Code Examples
Response Content
Status Code: 200
Example Response Structure
Example Response Values
Update a Multi-Turn Test Case
PUT /api/v1/multi-turn-test-case/{id}/
Parameters
id
✅
string
Multi-turn test case ID
Request Content
Request Parameters
scenario
string
Yes
Task scenario the simulated user should accomplish
expectedOutcome
string
Yes
Outcome a successful conversation should reach, also used as the simulation stop condition
userDescription
string
No
Persona of the simulated user, such as tone, background and level of expertise
status
string (enum: active, inactive)
No
active: Active ; inactive: Inactive;
flaky
boolean
No
Flaky test cases still run and keep their results, but are excluded from the pass rate.
Example Request Structure
Example Request Values
Code Examples
Response Content
Status Code: 200
Example Response Structure
Example Response Values
Partially Update a Multi-Turn Test Case
PATCH /api/v1/multi-turn-test-case/{id}/
Parameters
id
✅
string
Multi-turn test case ID
Request Content
Request Parameters
scenario
string
No
Task scenario the simulated user should accomplish
expectedOutcome
string
No
Outcome a successful conversation should reach, also used as the simulation stop condition
userDescription
string
No
Persona of the simulated user, such as tone, background and level of expertise
status
string (enum: active, inactive)
No
active: Active ; inactive: Inactive;
flaky
boolean
No
Flaky test cases still run and keep their results, but are excluded from the pass rate.
Example Request Structure
Example Request Values
Code Examples
Response Content
Status Code: 200
Example Response Structure
Example Response Values
Delete a Multi-Turn Test Case
DELETE /api/v1/multi-turn-test-case/{id}/
Parameters
id
✅
string
Multi-turn test case ID
Code Examples
Response Body
204
No response body
List Available Evaluation Models
GET /api/v1/evaluations/evaluation-llms/
Code Examples
Response
Status code: 200
Response Schema
Response Example
Status code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Run a Single-Turn Evaluation
POST /api/v1/evaluations/evaluate/
Request
Request Parameters
datasetId
string (uuid)
Yes
chatbotId
string (uuid)
Yes
evaluationLlmId
string (uuid)
No
name
string
No
description
string
No
metricConfig
object
No
customMetrics
array[DatasetEvaluationCustomMetric]
No
Request Schema
Request Example
Code Examples
Response
Status code: 202
Response Schema
Response Example
Status code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Run a Multi-Turn Evaluation
POST /api/v1/evaluations/multi-turn-evaluate/
Request
Request Parameters
datasetId
string (uuid)
Yes
chatbotId
string (uuid)
Yes
evaluationLlmId
string (uuid)
No
name
string
No
description
string
No
maxTurns
integer
No
minTurns
integer
No
turnFaithfulness
boolean
No
turnRelevancy
boolean
No
conversationCompleteness
boolean
No
language
object
No
customMetrics
array[MultiTurnCustomMetric]
No
Request Schema
Request Example
Code Examples
Response
Status code: 202
Response Schema
Response Example
Status code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
List Evaluations
GET /api/v1/evaluations/
Parameters
chatbotId
❌
string
Only list evaluations of this AI assistant
datasetId
❌
string
Only list evaluations that use this dataset
excludeBuiltIn
❌
string
When true, excludes data of official built-in Agents (locked instances created by renting from the Agent marketplace) and keeps only assistants built by the organization; copies created with "Copy as self-built" count as self-built. Omitting it or passing false is the same as not applying this filter.
ordering
❌
string
Sort field: created_at or updated_at; prefix with - for newest first (e.g., -updated_at). If not specified, results are sorted by creation time, newest first
page
❌
integer
Page number, starting from 1; returns 404 if beyond the last page
pageSize
❌
integer
Number of items per page; default 20, maximum 100 (values above 100 are treated as 100)
query
❌
string
Keyword, matched against the evaluation name, description, or ID (partial match)
search
❌
string
Keyword, matched against the evaluation name, dataset name, or AI assistant name (partial match, case-insensitive; when multiple keywords are separated by spaces, all must match)
status
❌
string
Only list evaluations with this status. Available values: `pending`: Pending evaluation ; `running`: Evaluating ; `stopping`: Stopping ; `completed`: Completed ; `partial_completed`: Partially completed ; `failed`: Evaluation failed;
Code Examples
Response
Status code: 200
Response Schema
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Retrieve Evaluation Result
GET /api/v1/evaluations/{id}/
Parameters
id
✅
string
Evaluation ID
Code Examples
Response Content
Status Code: 200
Response Schema Example
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
List Per-Question Results of a Single-Turn Evaluation
GET /api/v1/evaluations/{id}/details/
Parameters
id
✅
string
Evaluation ID
Code Examples
Response Content
Status code: 200
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
List Per-Conversation Results of a Multi-Turn Evaluation
GET /api/v1/evaluations/{id}/multi-turn-results/
Parameters
id
✅
string
Evaluation ID
page
❌
integer
Page number, starting from 1; a page beyond the last page returns 404
pageSize
❌
integer
Items per page. Default 20, maximum 100 (values above 100 are treated as 100)
Code Examples
Response Content
Status code: 200
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Cancel a Running Evaluation
POST /api/v1/evaluations/{id}/cancel/
Parameters
id
✅
string
Evaluation ID
Code Examples
Response Content
Status code: 200
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Resume an incomplete evaluation
POST /api/v1/evaluations/{id}/resume/
Parameters
id
✅
string
Evaluation ID
Code Examples
Response
Status Code: 202
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Export evaluation report
POST /api/v1/evaluations/{id}/export/
Parameters
id
✅
string
Evaluation ID
Code Examples
Response
Status Code: 201
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Get latest export status
GET /api/v1/evaluations/{id}/export-status/
Parameters
id
✅
string
Evaluation ID
Code Examples
Response
Status Code: 200
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
List export records
GET /api/v1/evaluations/{id}/exports/
Parameters
id
✅
string
Evaluation ID
Code Examples
Response
Status Code: 200
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Edit evaluation name and description
PATCH /api/v1/evaluations/{id}/
Parameters
id
✅
string
Evaluation ID
Request Body
Request Parameters
name
string
No
description
string
No
Example Request Structure
Example Request Values
Code Examples
Response
Status Code: 200
Example Response Structure
Example Response Values
Status Code: 4XX - Error response; X-Error-Code carries its code (see "Error codes").
Delete evaluation
DELETE /api/v1/evaluations/{id}/
Parameters
id
✅
string
Evaluation ID
Code Examples
Response Content
204
No response body
4XX
Error response; X-Error-Code carries its code (see "Error codes").
Last updated
Was this helpful?