Reply evaluation and monitoring results

View reply evaluation results

AI → Configuration → Reply Evaluation

MaiAgent reply evaluation page

Scoring metrics

The MaiAgent platform provides reply evaluation functionality, recording and automatically scoring each Q&A; scores include

Metric	Description	Impact causes	Question	Reply	Retrieved context	Correct answer
Faithfulness score	Whether the LLM answered truthfully rather than fabricating an answer	LLM, RAG, Knowledge		✅	✅
Answer Relevance score	Whether the LLM addressed the key points, and whether the answer is incomplete or contains redundant text	LLM, RAG, Knowledge	✅	✅
Context Precision score	Whether the content retrieved by RAG is relevant to the question	RAG, Knowledge	✅		✅
Answer Correctness score	The correctness of the reply compared to the correct answer	LLM, RAG, Knowledge		✅		✅
Answer Similarity score	The semantic similarity between the reply and the correct answer	LLM, RAG, Knowledge		✅		✅
Context Recall score	Whether the content retrieved by RAG included the information compared to the correct answer	RAG, Knowledge			✅	✅

Relationship diagram of reply evaluation metrics

Clarifying causes of low scores and ways to resolve them

• LLM capability issues: unable to answer based on reference materials

  • Solution: switch to a more capable LLM

• RAG retrieval capability: whether it found data related to the question

  • Solution: contact MaiAgent support

• Whether the knowledge base provides sufficient data

  • Solution: supplement the knowledge base with correct data and FAQ common questions