> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/tech/en/advanced-genai-tech/evaluation.md).
# Response Evaluation and Monitoring Results
Technical details of MaiAgent response evaluation：metrics used by Agent monitoring and automated evaluation, required data, score direction, and test case pass criteria.
MaiAgent evaluates Agent response quality in two ways: **Agent monitoring**, which scores each response after deployment, and **automated evaluation**, which runs batches of tests from a test set. This page explains the metrics and calculations used by each. For instructions on creating evaluations and viewing results in the admin console, see [Automated Evaluation and Agent Monitoring](https://docs.maiagent.ai/build/agent-ops/evaluations) in the user manual. To create test sets, run evaluations, and retrieve results from your own system, see [Evaluations and Test Sets](https://docs.maiagent.ai/api/api-reference/ping-gu-he-ce-shi-ji) in the API documentation.
## Differences Between the Two Evaluation Modes
| Item | Agent Monitoring | Automated Evaluation |
| ---------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Evaluation target | Each response in an actual conversation | Questions in a test set, scored after the Agent answers them |
| How to enable | The Agent's Enable Automated Evaluation setting | Create an evaluation in AgentOps |
| Scoring framework | Ragas | DeepEval（4.1.7） |
| Model used for scoring | The evaluation model configured for the platform | The Evaluation Model selected when creating the evaluation |
| Metrics | 3 fixed metrics (see below) | Combine built-in metrics and add custom metrics |
| Cost | Scoring consumes the organization's usage quota | Both answering and scoring consume the organization's usage quota |
{% hint style="info" %}
For on-premises deployments, the evaluation model and Embedding model used by Agent monitoring are specified in the platform's system settings. They must point to models reachable from the on-premises environment; otherwise, scoring will fail.
{% endhint %}
## Agent Monitoring Metrics
When Enable Automated Evaluation is enabled for an Agent, the platform calculates the following scores in the background after each response is completed. Results appear in Agent monitoring:
| Metric | What It Measures |
| ----------------- | --------------------------------------------------------------------------------------- |
| Faithfulness | Whether every part of the response is supported by the retrieved passages |
| Answer Relevancy | Whether the response addresses the question |
| Context Precision | How relevant the retrieved passages are to the question; no expected answer is required |
Actual conversations have no expected answers, so monitoring does not calculate metrics that require an expected answer.
## Automated Evaluation Metrics
### Single-Turn Evaluation
After the Agent answers each question, the response is scored using the following metrics (the names shown in the admin console are listed in the "Admin Console Name" column):
| Metric | Admin Console Name | What It Measures | When Creating an Evaluation in the Admin Console |
| -------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| Faithfulness | Quality Score | Whether the response is faithful to the retrieved passages | Always calculated |
| Answer Relevancy | Answer Relevancy | Whether the response addresses the question | Always calculated |
| Contextual Precision | Contextual Precision | Whether relevant passages are ranked first | Always calculated (requires an expected answer) |
| Contextual Relevancy | Contextual Relevancy | How relevant the retrieved passages as a whole are to the question | Always calculated (requires an expected answer) |
| Contextual Recall | Contextual Recall | Whether all information needed for the expected answer was retrieved | Always calculated (requires an expected answer) |
| Bias | Bias | Whether the response contains bias | Always calculated |
| Toxicity | Toxicity | Whether the response contains offensive or harmful content | Always calculated |
| Hallucination | Hallucination | Whether the response contradicts the retrieved content | Always calculated |
| PII Leakage | PII Leakage | Whether the response discloses personally identifiable information | Optional (Safety Metrics) |
| Role Violation | Role Violation | Whether the response deviates from the Agent's intended role | Optional (Safety Metrics) |
| Misuse | Misuse | Whether the Agent is used for unintended purposes | Optional (Safety Metrics) |
| Non-Advice | Avoid Professional Advice | Whether the response avoids providing professional advice in areas such as medicine, law, or finance | Optional (Safety Metrics) |
* Single-turn evaluations created in the admin console calculate all three contextual metrics, so **every question requires an ****Expected Answer**. Questions without an expected answer are marked Skipped; the Agent does not answer them, and they are not scored.
* The platform's evaluation service also supports Task Completion, Tool Correctness, and Summarization. These three metrics are not enabled when creating evaluations in the admin console.
### Multi-Turn Evaluation
In multi-turn evaluations, a simulated user converses with the Agent based on the test case scenario (maximum turns: 2～10, default: 6). The conversation is then scored using the following metrics. At least one built-in metric must be enabled, or a custom metric must be configured:
| Metric | What It Measures |
| ---------------------------------------------------------- | ----------------------------------------------------------------- |
| Turn Faithfulness | Whether each turn's response is faithful to the retrieved content |
| Turn Relevancy | Whether each turn's response addresses the question |
| Conversation Completeness | Whether the conversation as a whole achieves the user's goal |
### Custom Metrics (GEval)
Both single-turn and multi-turn evaluations support Custom Metrics. Write Evaluation Criteria in natural language or list Evaluation Steps, and the evaluation model assigns a score from 0～1 based on those criteria.
| Limit | Rule |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Number | Up to 10 per evaluation |
| Name | Up to 100 characters; names must be unique within an evaluation (case-insensitive) and must not match a built-in metric in that mode |
| Evaluation criteria | Up to 2,000 characters |
| Evaluation steps | Up to 10 steps, each up to 500 characters |
| Pass Threshold | 0～1, default: 0.5 |
Custom metrics in single-turn evaluations score the question, the Agent's response, and the retrieved passages. Custom metrics in multi-turn evaluations score the entire conversation transcript without referring to retrieved passages.
## Score Direction and Pass Criteria
* Scores range from 0～1. Built-in metrics use DeepEval's default pass threshold of 0.5.
* Individual test case scores retain DeepEval's original direction: **lower scores are better for Bias, Toxicity, Hallucination, and Misuse**; higher scores are better for all other metrics. Evaluation statistics convert these four metrics to "1 − score" so that higher values are always better in the statistics.
* To determine whether a single-turn test case passes, three groups of key metrics are checked in order. If any metric in any group fails to meet its threshold, the test case fails:
1. Safety: Bias, Toxicity, Hallucination, PII Leakage, Role Violation, Misuse, and Non-Advice
2. Outcome: Answer Relevancy (and Task Completion)
3. Grounding: Contextual Recall
* Other metrics (Quality Score, Contextual Precision, Contextual Relevancy, custom metrics, etc.) must also meet their thresholds. Their failure to meet a threshold does not affect the pass result only when all three groups above have been calculated and have passed in full.
* Test cases marked as flaky still run and retain their results, but are excluded from the pass rate.
## Differences Between Faithfulness and Hallucination
Both examine the relationship between a response and retrieved content, but from different perspectives:
| Aspect | Faithfulness (Quality Score) | Hallucination |
| ------------------- | --------------------------------------------------------------------------- | --------------------------------------------------- |
| Evaluation question | How many statements in the response are supported by the retrieved content? | Does the response contradict the retrieved content? |
| Score direction | Higher is better | Lower is better for individual test case scores |
For example, suppose the retrieved content says "Taipei 101 is 508 meters tall and was completed in 2004," and the response adds "It was once the world's tallest building." This additional statement is not supported by the retrieved content, so Faithfulness decreases. Hallucination checks whether the response contradicts the retrieved content.
## Troubleshooting Low Scores
* **Low Contextual Recall or Contextual Precision**: Retrieval did not find the correct passages. First verify with [Search Testing](https://docs.maiagent.ai/build/knowledge-base/test-search-result), then adjust the knowledge base content, parser, or retrieval parameters. See [RAG Knowledge Base Retrieval System](/tech/en/quickstart/rag.md#parameters).
* **Low Quality Score with normal contextual scores**: The passages were found, but the response was not based on them. Check the system prompt or use a more capable model.
* **Low Answer Relevancy**: The response did not address the question. This commonly occurs when the system prompt is too restrictive or the question falls outside the knowledge base's scope.
* **Generally unstable scores**: Scoring itself depends on the evaluation model's judgment. Choose a more capable evaluation model.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.maiagent.ai/tech/en/advanced-genai-tech/evaluation.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.