1. Introduction: Why RAG Evaluation Can’t Rely on “Feelings” Alone?

Imagine this scenario: You painstakingly build a RAG system. The knowledge base contains tens of thousands of professional documents, and the large language model is an industry-recognized top-tier closed-source model. A user asks, “How will blood potassium levels change in hypertensive patients taking diuretics?” The system provides an answer that cites authoritative sources, is logically coherent, and even includes references. You feel quite proud, thinking the system is amazing. However, suddenly a new user asks, “Will blood potassium decrease in hypertensive patients taking thiazide diuretics?”, and the system replies, “No need to worry about blood potassium issues, because the side effects of modern diuretics are controllable.” This is clearly wrong: the most common side effect of thiazide diuretics is precisely hypokalemia. What’s your first reaction? Don’t you feel the large model is “hallucinating”?

This dilemma of “good metrics but poor experience” is the nightmare of all RAG practitioners. Traditionally, we might use a few simple metrics like BLEU and ROUGE to evaluate the similarity of generated text, or Recall@K and Precision@K to evaluate the quality of retrieved results. But the problem is that these metrics are often isolated and one-sided. Even if the retrieval relevance Recall@K reaches 0.9 and the BLEU score of the generated text is high, the final answer the user gets may still be incorrect, incomplete, or even hallucinated.

Why? Because a RAG system is a complex end-to-end pipeline. From the encoding of the Embedding model, to the retrieval of the vector database, to the fusion generation of the large model, noise is introduced at every step. If you only focus on a single component—retrieval or generation—it’s like the blind men and the elephant. More critically, many teams fall into the “metric optimization trap”: to improve Recall@K, they retrieve a large number of irrelevant documents; to improve BLEU scores, they make the model output formulaic, safe responses, completely sacrificing relevance and practicality.

Therefore, we need a systematic, end-to-end evaluation framework. This is why we introduce new-generation evaluation frameworks represented by the Ragas metric system. Ragas no longer only focuses on the surface similarity of text but delves into the three core capabilities of a RAG system: Context Relevancy (whether retrieved documents are relevant to the question), Faithfulness (whether the generated answer is based on the retrieved context), and Answer Relevancy (whether the final answer directly addresses the user’s question).

This article will break down a practical RAG evaluation metric system, from top-level design to underlying implementation. You will learn:

  • Core Principles of Evaluation: How to avoid “self-congratulatory evaluation” and truly align with business objectives.
  • Overview of the Ragas Metric System: From discrete to continuous, from text to multimodal—which core metrics deserve attention.
  • Detailed Explanation of End-to-End Evaluation Metrics: What each metric is, why it’s important, how it’s calculated, with code examples.
  • Hands-On: Build Your Evaluation Pipeline: Directly run the Ragas evaluation process with Python code and learn to visualize results with radar charts.
  • Advanced Tips and Common Pitfalls: How to choose metrics, handle Chinese scenarios, deal with multimodal extensions, and solutions to frequent traps.

After reading this article, you will have the ability to build a RAG evaluation system that is “visible, explainable, and improvable,” saying goodbye to “mystical tuning” and “metric hallucinations.” Let’s begin.

2. Core Principles of RAG Evaluation: White-Box + Black-Box, Aligned with Business Goals

Before diving into specific metrics, we must clarify the top-level design principles of evaluation. A common mistake is that teams directly copy evaluation methods from academic papers without considering their own business scenarios. This often leads to a huge gap between evaluation results and real user experience.

A healthy RAG evaluation system must follow the principle of combining white-box and black-box approaches.

  • White-Box Evaluation: Focus on the Process, Pursue Observability. White-box evaluation concerns the performance of each internal component of the RAG system. This is like inspecting each workstation on an assembly line: Is the raw material (query) correctly encoded? Did the retrieval module find the most relevant parts (context)? Did the large model strictly follow the part specifications (context) during generation?

White-box metrics include Context Precision, Context Entities Recall in the retrieval stage, and Faithfulness and Groundedness in the generation stage. These metrics help us pinpoint which component is causing the problem—whether the retrieval is inaccurate or the model is making things up.

  • Black-Box Evaluation: Focus on the Outcome, Pursue User Experience. Black-box evaluation focuses on the final outcome that the user perceives. It doesn’t care about internal workings; only whether the final answer is accurate, complete, and convincing, and whether the user is satisfied. Black-box metrics include Response Relevancy, Tool Call Accuracy (for Agent scenarios), and more business-oriented metrics such as resolution rate for customer service or compliance pass rate for financial scenarios.

Aligning with business objectives is the soul of evaluation system design. Evaluation not aligned with business is like a ship without a direction. For example, for an analyst in financial research, the most important concerns are factual accuracy and traceability of cited sources. In this case, Faithfulness and Context Precision should have extremely high weight. If it’s a customer service chatbot for general knowledge Q&A, users may care more about the comprehensiveness and ease of use of the answer, so Response Relevancy and Context Entities Recall would be more important.

Best Practice: At the project initiation stage, define the core “North Star Metric” together with the business stakeholders. For example, for a RAG system in finance, you can set “factual accuracy (annotated by domain experts)” as the ultimate evaluation criterion. Then, work backward to derive the automated evaluation metrics (like Faithfulness) that have the highest correlation with this North Star metric, and repeatedly verify the strong correlation between them.

3. Overview of the Ragas Metric System: From Discrete to Continuous, from Text to Multimodal

Now that we understand the evaluation principles, let’s get a panoramic view of the rich metric types in the Ragas ecosystem. Ragas’ design philosophy is to provide a modular, extensible framework that allows developers to freely combine metrics according to their needs. The Ragas metric system can be roughly divided into three categories: Discrete Metrics, Continuous Metrics, and Extended Metrics.

3.1 Discrete Metrics: Qualitative Classification, Quick Decisions

Discrete metrics return predefined category results, not specific numerical values. This type of metric is very suitable for coarse-grained quick screening and manual review. For example, a simple discrete_metric can return "pass" or "fail". You can use it to mark binary issues like “whether there is hallucination” or “whether it contains offensive language.” In Ragas, you can define your own discrete metric using the @discrete_metric decorator:

1
2
3
4
5
6
7
8
9
# Define a discrete metric: response quality
@discrete_metric(name="response_quality", allowed_values=["high", "medium", "low"])
def assess_quality(response: str) -> str:
    if len(response) > 200:
        return "high"
    elif len(response) > 50:
        return "medium"
    else:
        return "low"

The advantage of discrete metrics is their strong interpretability—team members can understand them at a glance. During rapid iterative development, you can first use discrete metrics for “pass/fail” smoke testing to ensure basic functionality works.

3.2 Continuous Metrics: Quantitative Scoring, Fine-Grained Measurement

Continuous metrics are the core of the Ragas system; they return a continuous numerical value (usually between 0 and 1) for fine-grained quantification of system performance. These metrics form the basis of automated benchmarks. The most commonly used continuous metrics in Ragas include:

  • Context Precision: Measures the degree to which the retrieved contexts are relevant to the question. A score of 0.8 means 80% of the retrieved results are directly related to the question.
  • Context Entities Recall: Measures whether the retrieved contexts cover the key entities mentioned in the question (such as names of people, places, technical terms).
  • Faithfulness: Ensures that every claim in the generated answer can be supported by evidence in the retrieved contexts. This is one of the most important metrics for a RAG system.
  • Response Relevancy: Evaluates whether the generated answer directly and completely addresses the user’s question.

The value of these continuous metrics is that you can use thresholds to drive automated workflows. For example, you can set a rule: if the Faithfulness score is below 0.85, label the answer as “requires manual review.”

3.3 Extended Metrics: Embracing Multimodal and Complex Scenarios

As RAG scenarios become more complex, Ragas is continuously expanding its metric family to address emerging needs like multimodality and tool calling.

  • Multimodal Faithfulness: When the RAG system processes content containing images, audio, or video, traditional text faithfulness metrics become ineffective. Multimodal faithfulness uses cross-modal alignment techniques to evaluate whether the generated answer is consistent with the visual or auditory information. For example, in medical imaging reports, it needs to determine whether the AI-generated conclusion matches the lesion area in the CT scan.
  • Tool Call Accuracy: In Agent scenarios, the RAG system calls external APIs or tools (e.g., database queries, calculators, calendars). Tool Call Accuracy evaluates whether the agent correctly selected the tool, and whether the parameter passing and calling timing are accurate. This is crucial in code generation (text-to-SQL) or multi-step task planning.

Ragas’ metric ecosystem has become quite mature, evolving from basic text evaluation to covering multimodality and intelligent agents, providing developers with a powerful arsenal for building comprehensive evaluation systems.

4. Detailed Explanation of End-to-End Evaluation Metrics: Retrieval, Generation, and Overall Quality

Now that we understand the metric categories, let’s break down the core metrics of the RAG pipeline one by one, seeing how they work and why they are important.

4.1 Retrieval Stage: Precision is King

Context Precision: This metric measures, within the set of retrieved results, how many are truly relevant to the query. Imagine a user asks “What is today’s weather in Shanghai?” The retrieval system returns three documents: “Beijing Weather,” “Shanghai Historical Weather,” and “Shanghai Weather Forecast.” If the first two are noise, then Context Precision is 1/3 = 0.33.

High Context Precision means the retrieval results are very focused, reducing noise interference for the large model and effectively lowering the risk of hallucinations during generation. In Ragas, the calculation logic of this metric can be simplified as: for each query, sort the retrieved documents by relevance to the question, then calculate the cumulative ratio of relevant documents in the entire sequence.

Context Entities Recall: This is a key metric for measuring retrieval coverage, especially suitable for knowledge-intensive queries, such as legal provisions or disease diagnoses. For example, a user asks, “What are the main metabolites of methamphetamine?” The question contains two key entities: “methamphetamine” and “metabolites.” If your context only mentions “methamphetamine” but misses “metabolites,” the entity recall will be low.

This metric works by identifying named entities in the question (NER) and then checking whether they are covered in the contexts. A high Context Entities Recall ensures that downstream models can produce complete, well-supported answers.

Tip: Entity recall is often harder to optimize than precision because it depends on the embedding model’s ability to understand domain entities. If your system often misses key information, consider using hybrid retrieval (dense + sparse) or a reranker model to improve it.

4.2 Generation Stage: Facts and Relevance Equally Important

Faithfulness: This is one of the most critical metrics in the RAG evaluation system, directly measuring whether the large model is “reading aloud” or “making things up.” Its workflow usually involves: first, breaking down the generated answer into several independent claims; then checking each claim to see if there is direct evidence in the provided context. If a claim says, “According to the latest medical research, this drug can cure cancer,” but the context never mentions “latest medical research,” that claim is marked as “unfaithful.”

In a highly faithful system, every factual point in the answer should be traceable back to the knowledge base. Ragas implements this metric using a multi-step prompt design (see faithfulness.py); essentially, it uses an LLM (or a specialized evaluation model) to perform the decomposition and verification process.

Response Relevancy: This metric evaluates whether the final answer is on-topic. An interesting counterexample: when a user asks “Why is the sky blue?” and the system answers “Because of Rayleigh scattering, and also the air is dry, and the ground temperature is high.” Although the answer includes the correct concept of “Rayleigh scattering,” the latter part about “dry air” and “ground temperature” is completely off track. Response Relevancy measures whether the answer fully and directly addresses the user’s question without introducing irrelevant information.

It is typically quantified by calculating the semantic similarity between the question and the generated answer.

Response Groundedness: This metric is similar to Faithfulness but has a different perspective. Faithfulness checks, from the perspective of the generated content, whether it is faithful to the context. Groundedness, on the other hand, looks from the perspective of the context, measuring whether the generated answer is “grounded in” (i.e., fully utilizes) the provided context. Imagine a system that produces a correct but vague, generic answer like “This depends on the specific situation.” Even if the answer has no factual errors, it does not leverage the specific context information at all; such an answer lacks groundedness. High groundedness means the answer is based on the retrieved specific information rather than the model’s prior knowledge.

4.3 End-to-End Capability: Special Tests in Agent Scenarios

Tool Call Accuracy: In more advanced Agent+RAG scenarios, the system may not only retrieve text but also call external APIs. For example, a financial analysis Agent is asked, “Query Apple Inc.’s revenue for the fourth quarter of 2023.” It will call a tool like get_company_financial_data(company="AAPL", quarter="Q4", year=2023). Tool Call Accuracy comprehensively evaluates: 1) Whether the correct tool was selected (e.g., did it mistakenly call get_stock_price?); 2) Whether the parameters were passed correctly (e.g., was the year 2024 or 2023?); 3) Whether the calling timing was appropriate (e.g., calling blindly before obtaining knowledge base background?). Ragas takes the question, tool_calls, and execution_results as input to evaluate the correctness of this series of operations.

This is especially important when building autonomous AI Agents.

5. Hands-On: Build Your RAG Evaluation Pipeline with Ragas (with Code)

Enough theory; a working piece of code is worth a thousand words. Below we use a complete Python hands-on example to demonstrate how to use the Ragas library to evaluate a simple RAG system. We will use three core metrics: Context Precision, Faithfulness, and Response Relevancy, and finally visualize the results with a radar chart.

5.1 Installation and Preparation

First, ensure you have installed Ragas and related dependencies.

1
2
# Install ragas and core dependencies
pip install ragas datasets matplotlib

5.2 Construct the Evaluation Dataset

Ragas evaluation is based on datasets. The dataset needs three core fields: question (the user question), answer (the model-generated answer), and contexts (the list of retrieved contexts). Let’s create a simplified example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from datasets import Dataset
import pandas as pd

# Simulate an evaluation dataset
# Note: In a real project, you can batch-generate this data from your RAG pipeline logs
data_samples = {
    'question': [
        'What is the capital of France?',
        'What is the capital of Germany?',
        'What is the chemical formula for water?'
    ],
    'answer': [
        'Paris is the capital of France.', # Correct, context-based answer
        'Berlin is the capital of Germany.', # Correct answer
        'H2O is the chemical formula for water.' # Correct answer
    ],
    'contexts': [
        # Contexts for the first question (contains noise)
        ['France is a country in Europe. Its capital is Paris. The Eiffel Tower is in Paris.'],
        # Contexts for the second question (perfect match)
        ['Germany, officially the Federal Republic of Germany, is a country in Central Europe. Its capital is Berlin.'],
        # Contexts for the third question (incomplete, only contains water)
        ['Water is a liquid. It can be solid, liquid, or gas.']
        # Note: The third context does not mention H2O, which will result in low Faithfulness and Groundedness scores.
    ]
}

# Build a Hugging Face Dataset
dataset = Dataset.from_pandas(pd.DataFrame(data_samples))
print(dataset)

Best Practice: Data preparation is the most critical step in evaluation. In production, it is recommended to persist the question, answer, and retrieved contexts from every request of your RAG pipeline into a database or log, then periodically use scripts to batch-construct Datasets for evaluation.

5.3 Define and Run the Evaluation

Now we define the metrics to evaluate and run the evaluation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
from ragas.metrics import (
    context_precision,
    faithfulness,
    response_relevancy
)
from ragas import evaluate

# Define the list of metrics to evaluate
metrics = [
    faithfulness,      # Faithfulness: whether the answer is faithful to the context
    context_precision, # Context Precision: whether the context is relevant to the question
    response_relevancy # Response Relevancy: whether the answer addresses the question
]

# Run evaluation
result = evaluate(
    dataset=dataset,
    metrics=metrics,
    # Optional: if you have a dedicated LLM for evaluation, specify it here
    # llm=your_llm_object
)

# Convert result to a DataFrame for easy viewing
df = result.to_pandas()
print(df)

Running this code will give you a DataFrame where each row corresponds to a test sample and each column corresponds to the score of a specific metric. For example, you will see that the third sample (water) likely has a low faithfulness score because it fills in an answer, but the context does not provide the chemical formula.

5.4 Result Visualization: Using a Radar Chart

A table alone is not intuitive; let’s use a radar chart to quickly compare the performance of different samples across various dimensions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import numpy as np
import matplotlib.pyplot as plt

# Prepare data: calculate the average score for each metric
# You can also plot per sample, but for demonstration we show the average
metrics_names = [m.name for m in metrics]
scores = [df[m.name].mean() for m in metrics]

# Draw radar chart
angles = np.linspace(0, 2 * np.pi, len(metrics_names), endpoint=False).tolist()
# Close the shape
scores.append(scores[0])
angles.append(angles[0])

fig, ax = plt.subplots(figsize=(6, 6), subplot_kw=dict(polar=True))
ax.fill(angles, scores, color='blue', alpha=0.25)
ax.plot(angles, scores, color='blue', linewidth=2)

# Set labels
ax.set_thetagrids(np.degrees(angles[:-1]), metrics_names)
ax.set_title("RAG System Performance Metrics (Average)")
plt.show()

This radar chart can very intuitively show which aspects of your RAG system are doing well (e.g., response relevancy) and which need improvement (e.g., context precision).

5.5 CI/CD Integration

To make evaluation continuous, integrate it into the CI/CD pipeline. For example, in GitHub Actions, automatically run an evaluation on every merge and set a quality gate.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# .github/workflows/ragas-eval.yml
name: RAG Evaluation
on:
  push:
    branches: [ main ]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
      - run: pip install ragas datasets
      - run: python your_evaluation_script.py > evaluation_report.json
      - uses: actions/upload-artifact@v4
        with:
          name: evaluation-report
          path: evaluation_report.json
      # You can add a step to check if thresholds are met (e.g., faithfulness > 0.8)

6. Advanced Tips: Metric Selection Strategy, Interpretability, and Multimodal Extension

Now that you have the basics, let’s talk about how to make your evaluation system “smarter.”

6.1 Metric Selection Strategy: “Tailor” According to Business Scenarios

Not all metrics apply to every business. In a mature evaluation system, you should have a “metric matrix,” choosing different combinations for different scenarios.

  • Financial Research/Legal Compliance: The highest priority is Faithfulness and Context Precision. A single wrong word could lead to huge risks. You can give up excessive pursuit of comprehensiveness (i.e., not overemphasizing context recall).
  • Customer Service Chatbot: High priority is Response Relevancy and Context Entities Recall. Customer service wants to solve problems quickly; answers must be on point and cannot miss key entities (e.g., order numbers, usernames).
  • Text-to-SQL System: This is a typical tool-calling scenario. Prefer using “execution accuracy” instead of semantic similarity-based metrics.

For example, two SQL statements may be written differently but produce the same execution result; both should be considered correct. The Tool Call Accuracy metric, specifically the Execution based Datacompy Score, is designed for this.

6.2 Interpretability: Make It Understandable for Non-Technical Teams

The value of an evaluation system is to guide decision-making. Therefore, metrics must be interpretable. Choose metrics that the entire team can understand. In Text-to-SQL systems, “execution accuracy” (whether the result returned by executing the SQL matches the correct answer) is easier to explain to business stakeholders than abstract “semantic similarity.” Similarly, in RAG systems, “whether the answer contains factual errors” (a simple explanation of Faithfulness) is more straightforward than “cosine similarity score.”

You can try mapping continuous metrics to interpretable categories. For example, map a faithfulness score >= 0.9 to “Excellent,” 0.7-0.9 to “Needs Attention,” and <0.7 to “Unqualified.”

6.3 Multimodal Extension and Customized Evaluation

  • Multimodal Faithfulness: When the RAG system starts handling documents containing images, tables, or audio/video, evaluation becomes more complex. Ragas’ Multimodal Faithfulness metric is an extension direction. For example, a medical report contains a CT image and a text description. The AI-generated conclusion says, “There is a ground-glass nodule in the lung.” The evaluation process requires cross-validation: Does the text description support the concept of a “nodule”? Does the relevant region in the CT image also show ground-glass features? Multimodal Faithfulness is achieved through cross-modal matching of images and text.
  • Instance Specific Rubrics Scoring: This is key for fine-grained customized evaluation. Sometimes, general metrics cannot capture the subtle requirements of a specific domain. For example, when evaluating a creative writing RAG system, you might want its answers to be “more literary” and “cite fewer references.” You can create custom scoring criteria for a specific test case (e.g., “Write a poem about autumn”): 1) Include personal emotions not present in the original text (+0.5); 2) Overuse of references (-0.3). By writing rubrics, you can obtain a highly customized quality score.

7. Pitfalls and Solutions: Common Traps

Over long-term use of Ragas for RAG evaluation, you will inevitably encounter various problems. Here are several common “pitfalls” and my suggested solutions.

7.1 Annotation Data Bias Leading to Faithfulness Misjudgment

The faithfulness metric relies on an “evaluator LLM” to decide if a claim is faithful to the context. However, this evaluator LLM itself may be biased. For example, if your knowledge base contains many vague sentences like “according to research,” and the generated answer becomes “researchers unanimously agree,” although a human might consider these equivalent, the evaluation model might judge it as “unfaithful.” Solution: Regularly sample and manually review evaluation results, especially low-scoring samples, to analyze whether they are true hallucinations or evaluation model misjudgments. You can try using multiple different evaluation models (e.g., GPT-4, Llama3-70B) and take a majority vote.

7.2 Metric Conflict: High Precision but Low Recall

You may find that a task has a high Context Precision score (all retrieved documents are relevant) but a low Context Entities Recall score (missing key entities). This happens because the system is too conservative, retrieving only the most relevant small set of documents at the cost of completeness. This is a classic trade-off between “precision” and “recall” in RAG systems. Solution: Make a clear trade-off based on the business scenario. If the business scenario requires finding precise legal provisions (high precision), low recall is acceptable; if the scenario involves answering comprehensive questions, you need to optimize the retrieval strategy to improve recall. Consider adaptive retrieval or multi-turn retrieval.

7.3 Computational Cost of Multimodal Evaluation

Starting multimodal evaluation (e.g., using CLIP models for image-text comparison) is computationally expensive. If you evaluate all data daily, the GPU bill will be staggering. Solution: Adopt a staged evaluation strategy. First, use lightweight text metrics (such as Faithfulness) for preliminary screening. Only when text metrics pass, then perform multimodal evaluation on necessary samples. Alternatively, use more efficient distilled models for multimodal evaluation.

7.4 Adaptability for Chinese Scenarios

Ragas’ default models (especially the LLMs used for evaluation) may not understand Chinese as well as English. For example, coreference resolution and polysemy in Chinese may cause the evaluation model to make mistakes. Solution:

  • Replace the evaluation LLM: Use models that support Chinese better, such as Qwen, Yi, etc., as the evaluation backend for Ragas.
  • Adjust Prompts: Modify the underlying evaluation prompt templates in Ragas to better suit the Chinese context. For example, in the Faithfulness prompt, add “Based on the provided Chinese paragraph, check sentence by sentence whether the Chinese statements in the answer can find supporting evidence.”
  • More Manual Annotation: In Chinese projects, it is recommended to first have domain experts manually annotate a certain number of samples as a gold standard, then compare with Ragas’ automated evaluation results to calibrate thresholds.

8. Summary and Outlook: The Continuous Iteration Evaluation Loop

In summary, RAG system evaluation is not a one-time task, but a closed-loop process that requires continuous iteration and optimization.

Review of Core Knowledge:

  • The evaluation system must be end-to-end, covering both retrieval (Context Precision, Context Entities Recall) and generation (Faithfulness, Response Relevancy, Groundedness).
  • The Ragas metric system provides a powerful set of evaluation tools, from basic to multimodal, helping you shift from subjective judgment to objective quantification.
  • Evaluation is not the end goal; optimization is. By analyzing evaluation results (e.g., identifying which dimension has low scores), you can pinpoint problems (whether it’s the embedding model, retrieval strategy, or large model prompt), then make targeted optimizations, redeploy, and re-evaluate, creating a flywheel effect.

Future Outlook:

  1. Automated Evaluation Driven by Large Models: Using large models themselves (e.g., GPT-4 as a judge) to perform more complex, human-like automated evaluation. Ragas already supports this pattern, and it will become more mature.
  2. Introducing Adversarial Testing: Instead of only using standard test sets, actively generate “adversarial samples”—deliberately constructed confusing or tricky questions to test the system’s robustness. This can uncover many edge cases that regular testing misses.
  3. Unified Cross-Modal Evaluation Framework: As RAG systems become ubiquitous (from text to video), a unified framework is needed to evaluate faithfulness, accuracy, and quality across different modalities. Ragas’ multimodal faithfulness metric is just the beginning.

Finally, and most importantly: Always pursue your own evaluation system. Don’t blindly trust any off-the-shelf tool, including Ragas. Treat it as a powerful starting point. Based on your business data and user experience, continuously adjust metric weights and introduce new evaluation dimensions. Only an evaluation system that is locally adapted and continuously iterated can truly become the navigator for building reliable, trustworthy, high-intelligence RAG applications and even excellent AI Agents. Now, start by building your first evaluation pipeline.

Summary

Through this article, I believe you have gained a deeper understanding of the “RAG Evaluation Metric System.” It is recommended to practice more with actual projects. If you have any questions, feel free to discuss!