1. Introduction: Why Is the Embedding Model the “Invisible Bottleneck” of RAG?

Have you ever encountered this scenario? Your RAG (Retrieval-Augmented Generation) system is fully set up, with tens of thousands of meticulously organized documents stored in the knowledge base. Yet, when a user asks a question, the retrieval results are completely off the mark. For example, a user asks, “How should hypertension patients adjust their diet?”, but the system returns a document about “Nutritional Guidelines for Marathon Runners.” Worse still, different phrasings of the same question yield almost identical documents, failing to provide the downstream large model with accurate answers to generate reliable responses.

The culprit is often not that the large model is unintelligent, but that the Embedding model has “lost its hearing.” The Embedding model is the “translator” of the RAG system, responsible for converting human language into vectors that machines can understand, and then using vector similarity to find the most relevant knowledge snippets. If this translator doesn’t understand your domain’s language, or if the “hypertension” you ask about isn’t the same “hypertension” it understands, the entire system’s accuracy suffers dramatically.

General-purpose Embedding models (e.g., OpenAI’s text-embedding-ada-002, BAAI’s bge-base-en) perform well in general scenarios and are more than capable of handling news, encyclopedias, and common Q&A. However, in vertical domains (medicine, law, finance, industry, etc.), they frequently “crash and burn” for three reasons:

  1. Vocabulary Distribution Shift: The training corpus for general models consists mainly of popular text, while domain-specific texts are filled with sparse vocabulary like “hypertrophic cardiomyopathy,” “false securities statements,” or “cascade hydropower station scheduling.” The model’s vector representations for these words are inaccurate, often confusing them with unrelated general terms.

  2. Missing Semantic Associations: Relationships between synonyms and near-synonyms within a domain are complex. For instance, in the financial domain, “limit-down” and “circuit breaker” are highly correlated, but in a general model, they might be encoded as two stylistically different vectors, preventing effective association during retrieval.

  3. Length Adaptation Issues: Many general Embedding models only support inputs of up to 512 tokens (e.g., BERT variants). When faced with medical guidelines or legal statutes that are thousands of words long, you must truncate them forcibly, losing substantial context and leading to semantic fragmentation.

Therefore, to make your RAG system truly “hear and answer correctly” in a vertical domain, you must take two routes: scientifically selecting the base model and domain-adaptive fine-tuning. This article will break down the principles, methods, and hands-on code for each step, helping you build a RAG system from scratch that can accurately understand business language.

What you will gain from reading this article:

  • A clear understanding of the core role of Embedding models in RAG and why they fail.
  • How to choose the best open-source/commercial Embedding model based on your business scenario, forming your own RAG Embedding Model Selection Guide.
  • A reproducible method for domain-adaptive fine-tuning of an Embedding model, including data construction, training, and evaluation.
  • Insights into techniques like hybrid retrieval and hard negative mining to further improve retrieval accuracy.

Tip: This article is targeted at developers with some Python experience who are building RAG systems. If you are new to these concepts, it’s advisable to first read some introductory material on vector search basics.

2. Core Principles of Embedding Models: From Semantic Vectors to Domain Adaptation

Understanding the essence of Embedding models is the first step to proper selection and fine-tuning. Simply put, Embedding maps human language (words, sentences, paragraphs) into a low-dimensional, dense vector space. In this space, semantically similar texts should have vectors that are closer together.

Static Embedding vs. Dynamic Contextual Embedding

Early embedding techniques, such as Word2Vec or GloVe, were “static.” Each word had only one fixed vector representation, regardless of its context. For example, the word “apple” in “I love eating apples” and “Apple released a new phone” would have the same vector. This obviously cannot handle polysemy.

Modern Embedding models, such as BERT and its variants, are “dynamic contextual.” They dynamically adjust the vector representation of each word based on the surrounding words in the sentence. For instance, in the sentence “I love eating apples,” the vector for “apples” would lean towards the semantic meaning of “fruit”; in “Apple released a new phone,” it would lean towards the meaning of “company.” This contextual awareness greatly improves the accuracy of semantic understanding and is the reason why current RAG systems choose BERT variants as the primary Embedding model.

The Essence of Domain-Adaptive Fine-Tuning

A BERT model pre-trained on large-scale general corpora (e.g., Wikipedia, books, web pages) already possesses strong language understanding abilities. However, it cannot deeply understand the specialized terminology, jargon, and unique expressions of your specific domain. Domain-adaptive fine-tuning is essentially a small-scale, supervised “realignment.”

The goal of this process is to help the model learn more precise semantic similarities on the data distribution of your specific domain. Specifically, you train the model using a large number of domain-specific text pairs (e.g., a sentence from a financial research report and its correct summary) to guide the model to update its parameters. After fine-tuning, the model can more accurately pull closely related texts like “loose monetary policy” and “lowering the reserve requirement ratio” closer in the vector space, while pushing unrelated texts like “loose monetary policy” and “loose pants” further apart.

Core Formula: Fine-tuning effectiveness ∝ (Domain Data Quantity × Data Quality × Negative Sample Difficulty)

In short, domain data determines the direction of fine-tuning, data quality sets the upper bound, and the technique for constructing negative samples defines the model’s ability to distinguish subtle differences.

3. Mainstream Embedding Model Selection Guide: BERT Variants vs. bge-m3 vs. Other Notable Models

In the current open-source ecosystem, there is a dazzling array of Embedding models to choose from. Blindly following trends is unwise; you need a clear selection guide. We can evaluate models from four dimensions: language support, maximum input length, cross-domain capability, and performance metrics.

Model Landscape

  1. BERT Variant Series:

    • Representative Model: BAAI/bge-large-zh-v1.5
    • Features: This is a classic choice for Chinese scenarios. It performs excellently on retrieval tasks, optimized through contrastive learning. However, its input length is limited to 512 tokens.
    • Applicable Scenarios: QA scenarios with short, clear knowledge points, such as customer service Q&A libraries.

  2. Multilingual All-Rounder:

    • Representative Model: BAAI/bge-m3
    • Features: A multilingual universal Embedding model supporting over 100 languages. Its maximum input length is 8192 tokens (8K), allowing it to handle longer texts. Its performance ranks high on several MTEB tasks.
    • Applicable Scenarios: Multilingual mixed knowledge bases, scenarios requiring handling of long documents (e.g., contracts, reports).

Using bge-m3 for multilingual Embedding is the recommended solution for handling such business needs.

  1. Long Text Expert:

    • Representative Model: jinaai/jina-embeddings-v2-base-zh
    • Features: Optimized for long text scenarios, supporting 8192 token inputs.
    • Applicable Scenarios: Retrieval of extremely long documents like legal statutes or scientific papers.

  2. Commercial Benchmark:

    • Representative Model: text-embedding-3-small (OpenAI)
    • Features: Powerful performance, no local deployment needed, but charges per token, leading to higher long-term costs.
    • Applicable Scenarios: Scenarios where cost sensitivity is low and rapid deployment is desired.

Three-Dimensional Decision Table

Dimension Options Recommendation
Language Pure Chinese BAAI/bge-large-zh-v1.5
Multilingual (Chinese & English mix) BAAI/bge-m3 or OpenAI
Long text (>512 tokens) BAAI/bge-m3 or jina-embeddings-v2
Budget Free BAAI/bge-large-zh-v1.5, bge-m3
Paid OpenAI
Domain General Any of the above models
Vertical Specialized (Finance, Law, Medical) Strongly recommend fine-tuning the above open-source models for your domain

Note: The MTEB Leaderboard (https://huggingface.co/spaces/mteb/leaderboard) is an authoritative reference for objectively evaluating model performance. When selecting, focus on metrics related to your domain (e.g., “Retrieval” tasks).

4. Hands-on 1: Model Selection Based on MTEB Leaderboard and Chunking Strategy (with Code)

Selection shouldn’t stay on paper; we need hands-on practice. This hands-on section demonstrates how to evaluate model performance using HuggingFace’s mteb library and discusses chunking strategies for long text scenarios.

4.1 Evaluating Models

First, we evaluate models using our own test data. Assume we have a QA dataset eval_data.jsonl.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# 1. Install required libraries
# pip install sentence-transformers mteb datasets

from sentence_transformers import SentenceTransformer
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

# Load models
# Model 1: BGE Chinese version
model_bge = SentenceTransformer('BAAI/bge-large-zh-v1.5')
# Model 2: bge-m3 multilingual version
model_m3 = SentenceTransformer('BAAI/bge-m3')

# Simulate evaluation data: one query and two candidate documents
query = "What fruits can hypertension patients eat?"
doc_1 = "Apples are rich in dietary fiber, which helps lower cholesterol." # Relevant document
doc_2 = "Marathon runners need to consume a lot of carbohydrates before the race." # Irrelevant document

# Encode
query_emb_bge = model_bge.encode(query)
doc1_emb_bge = model_bge.encode(doc_1)
doc2_emb_bge = model_bge.encode(doc_2)

query_emb_m3 = model_m3.encode(query)
doc1_emb_m3 = model_m3.encode(doc_1)
doc2_emb_m3 = model_m3.encode(doc_2)

# Compute cosine similarity
sim_bge_1 = cosine_similarity([query_emb_bge], [doc1_emb_bge])[0][0]
sim_bge_2 = cosine_similarity([query_emb_bge], [doc2_emb_bge])[0][0]

sim_m3_1 = cosine_similarity([query_emb_m3], [doc1_emb_m3])[0][0]
sim_m3_2 = cosine_similarity([query_emb_m3], [doc2_emb_m3])[0][0]

print(f"BGE model - Similarity with relevant doc: {sim_bge_1:.4f}, with irrelevant doc: {sim_bge_2:.4f}")
print(f"bge-m3 model - Similarity with relevant doc: {sim_m3_1:.4f}, with irrelevant doc: {sim_m3_2:.4f}")

# Ideally, the similarity with the relevant document should be significantly greater than with the irrelevant one.
# You can extend this logic to compute metrics like Recall@k on a full dataset.

Best Practice: Using the official mteb library for evaluation is more systematic and comprehensive. You can use mteb.run(...) to easily get scores on your chosen subtasks, providing quantitative evidence.

4.2 Chunking Strategy: Handling Long Text Limitations

Even if you choose a model that supports 8K tokens, chunking strategies are still crucial when dealing with even longer texts. Proper chunking allows for finer retrieval granularity.

  • Fixed-length chunking: The simplest, but may truncate sentences, breaking semantics.
  • Recursive chunking: Use LangChain or LlamaIndex‘s RecursiveCharacterTextSplitter, trying to use sentences or paragraphs as separators to ensure semantic integrity.
  • Semantic chunking: Chunks by detecting topic changes (e.g., using embeddings to calculate points of similarity mutation between sentences). This yields the best results but has high computational cost.

Hands-on Advice: Prioritize recursive chunking with appropriate chunk size and overlap. For example, for a 512 token model, set chunk_size=450, chunk_overlap=50 to ensure continuity between text segments.

5. Hands-on 2: Financial Domain Embedding Fine-tuning – Data Preparation and Training

This is the most valuable step in the entire process. We’ll use the financial domain as an example to demonstrate supervised fine-tuning using the FlagEmbedding library.

5.1 Data Preparation: Constructing a High-Quality Fine-Tuning Dataset

Fine-tuning requires triplets of (query, positive_passage, negative_passage).

  • Positive Example Construction:

    • Direct Method: Use existing QA pairs: the question as the query, the corresponding answer as the positive passage.
    • GPT Generation Method: Extract paragraphs from your financial documents (e.g., research reports, annual reports), then use a GPT model (e.g., gpt-3.5-turbo) to generate questions based on that paragraph. This is one of the most effective methods.

    Assuming you have a financial document snippet doc_segment = "The current P/E ratio is at a historical low, suggesting an increase in holdings...", you could construct a prompt like: “Generate a question and its answer based on the following text: {doc_segment}” and send it to GPT.

  • Negative Example Construction:

    • Random Negative Sampling: Randomly pick paragraphs from the knowledge base that are unrelated to the query. Simple but effective.
    • In-Batch Negative Sampling: During training, samples within a batch automatically serve as negative examples for each other. This is the most efficient method.
    • Hard Negative Mining: Use the original model to retrieve paragraphs that have high similarity to the query but are actually irrelevant. This forces the model to learn to distinguish very subtle differences and is key to improving model robustness.

Data Format: Your training data should be saved as a JSONL file, where each line is a dictionary containing query, pos, and neg fields.

1
{"query": "How to determine if a stock is undervalued?", "pos": "You can compare indicators such as P/E ratio and P/B ratio with peer companies.", "neg": "The stock market closed up 1.5% yesterday."}

5.2 Supervised Fine-tuning: Using FlagEmbedding

FlagEmbedding is the official fine-tuning tool from BAAI, fully compatible with the bge model series.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Training script train.py
# 1. Install dependencies
# pip install torch transformers datasets accelerate sentence-transformers FlagEmbedding

from FlagEmbedding import FlagReranker
from FlagEmbedding import FlagModel
import torch

# It's recommended to execute training from the command line, but we can write the core parameters in a script
# Below is an equivalent Python code snippet (illustrative) for command-line execution

# Set training parameters
model_name = "BAAI/bge-large-zh-v1.5"
train_data_path = "./financial_train_data.jsonl"
output_dir = "./fine-tuned-bge-financial"

# Construct training command (better to use subprocess or shell to execute)
# python -m FlagEmbedding.baai_general_embedding.finetune \
#     --model_name_or_path model_name \
#     --train_data train_data_path \
#     --output_dir output_dir \
#     --num_train_epochs 5 \
#     --per_device_train_batch_size 16 \
#     --learning_rate 5e-6 \
#     --save_steps 500 \
#     --warmup_ratio 0.1 \
#     --overwrite_output_dir \
#     --q_max_len 64 \
#     --p_max_len 128 \
#     --negative_cross_device \
#     --fp16 \

print("Training started! Monitor the logs and watch training loss changes.")

Code Explanation:

  • --model_name_or_path: Specify your chosen base model, e.g., BAAI/bge-large-zh-v1.5.
  • --train_data: Point to the jsonl file you just prepared.
  • --num_train_epochs: Number of training epochs. Usually 3-5 is sufficient; too many can cause overfitting.
  • --per_device_train_batch_size: Training batch size per GPU. Adjust based on your GPU memory.
  • --learning_rate: Learning rate. For fine-tuning, small learning rates like 2e-5 to 5e-6 are typical.
  • --q_max_len and --p_max_len: Maximum token length for Query and passage, set based on your text length.
  • --negative_cross_device: During multi-GPU training, share negative samples across devices to effectively increase difficulty.
  • --fp16: Use half-precision training to speed up and save memory.

6. Hands-on 3: Optimizing Medical Text Embedding Models – Effect Evaluation and Iteration

Fine-tuning is not a one-time job; we need rigorous evaluation to verify effectiveness and iterate based on results.

6.1 Evaluation Metrics

On the test set, we can quantify results using the following metrics:

  • Recall@k: How many correct answers are found among the top k returned documents.
  • NDCG@k (Normalized Discounted Cumulative Gain): Considers the position of results. Earlier correct answers are given higher weight.
  • MRR (Mean Reciprocal Rank): The average of the reciprocal ranks of the first correct answer in the result list.

6.2 Effect Comparison Case Study

Assume we evaluate on a medical QA dataset (e.g., a subset of MedQA).

Query Before Fine-tuning (bge-large-zh-v1.5) After Fine-tuning (fine-tuned-bge-medical)
“Side effects of aspirin enteric-coated tablets” [“When taking aspirin, avoid concurrent alcohol consumption…”] (General answer) [“Common side effects of aspirin enteric-coated tablets include gastrointestinal bleeding, allergic reactions…”] (More precise specialized answer)
“Indications for beta-blockers” [“Beta-blockers can slow heart rate…”] (Partially relevant) [“Beta-blockers are mainly used to treat hypertension, coronary heart disease, heart failure…”] (Complete and accurate list of indications)

From this comparison, the fine-tuned model can more accurately understand the specific formulation “aspirin enteric-coated tablets” and the exact medical indications of the drug class “beta-blockers.”

6.3 Ablation Studies: Guiding Iteration Direction

To understand the effect of different factors, we can perform ablation experiments:

Experiment Group Setting Recall@10 Change Conclusion
Data Quantity 1000 vs 5000 samples +15% Data quantity is foundational, but data quality is even more critical.
Negative Sampling Strategy Random vs Hard Negative Mining +8% Introducing high-quality hard negatives significantly improves the model’s discriminative power.
Learning Rate 1e-5 vs 5e-6 +2% Smaller learning rates are generally more stable and yield slightly better results.

Key Insight: If training loss keeps decreasing but metrics on the validation set stagnate or decline, it indicates overfitting. In that case, you should increase training data, use more diverse negative samples, or reduce model size and training epochs.

7. Advanced Techniques: Long Text Input Limits and Chunking Strategies in Depth

Even after fine-tuning, facing the 512-token limit of BERT variants, you still need elegant chunking strategies. The quality of chunking directly affects retrieval “granularity” and “recall.”

Comparison of Three Chunking Strategies

Strategy Advantages Disadvantages Applicable Scenarios
Fixed Length Simple implementation, fast May truncate mid-sentence, breaking semantics, poor recall Scenarios with low semantic integrity requirements
Recursive Ensures semantic integrity, relatively simple implementation Uncontrollable chunk size Most general scenarios, preferred choice
Paragraph-based Best semantic integrity, aligns with document’s natural structure Highly variable chunk sizes, short paragraphs may lack information Scenarios with clear document structure (e.g., papers, reports)

Hybrid Chunking Strategy: Short Queries + Long Documents

This is a very practical technique. For short queries (user questions), we don’t want them to match a fragmented, uninformative text snippet. Therefore, we can adopt two sets of chunking schemes:

  1. Create two types of chunks for knowledge base documents:
    • Coarse-grained chunks: For example, entire paragraphs or content under subheadings, used for initial recall. Encode using bge-m3 (8K tokens).
    • Fine-grained chunks: Smaller segments further split from coarse-grained chunks, used for precise understanding.
  2. Two-Stage Retrieval:
    • First round: Match the user query vector against coarse-grained chunk vectors, recall top-N candidates.
  • Second round: Re-rank the query against fine-grained chunks within the top-N candidates to select the most precise top-K results.

This method balances recall and precision and is standard in industrial-grade RAG systems.

8. Pitfall Record: Common Traps and Countermeasures in Fine-Tuning Embedding Models

Using my hard-learned lessons, let me help you avoid common pitfalls.

Pitfall 1: Overfitting (Training loss decreases, but retrieval performance declines)

  • Symptom: Model performs perfectly on the training set but performs worse on unseen test data.

  • Cause: Model memorizes the training data by rote, without learning true semantic generalization.

  • Countermeasures:

    • Increase data volume: The most fundamental solution.
    • Stronger regularization: Use higher warmup_ratio (e.g., 0.2), lower learning rate (below 5e-6).

  • Increase negative sample difficulty: Introduce more hard negatives, forcing the model to learn finer distinguishing features instead of memorizing.

    • Early stopping: Monitor Recall@k on the validation set; stop training if it doesn’t improve for several consecutive epochs.

Pitfall 2: Imbalanced Positive and Negative Sample Construction

  • Symptom: Model tends to judge all inputs as either all relevant or all irrelevant to the query.
  • Cause: Severe imbalance between positive and negative samples in training data.
  • Countermeasure: Maintain a positive-to-negative sample ratio between 1:3 and 1:5. Using in-batch negative sampling effectively solves this because it naturally ensures enough negative samples in each batch.

Pitfall 3: Insufficient Domain Data (<1000 samples)

  • Symptom: Fine-tuning effect is insignificant.

  • Cause: Too little data to drive effective model parameter updates.

  • Countermeasures:

    • Transfer learning: Instead of fine-tuning from scratch, first pre-train for several epochs on a related but larger dataset (e.g., general QA), then transfer to your small dataset.

  • Data augmentation: Use LLMs (e.g., GPT-3.5) to generate more diverse queries and passages based on your limited existing data, expanding the dataset.

    • Prefer simpler models: Consider using smaller models (e.g., bge-small-zh) with fewer parameters, which converge more easily on small data.

Pitfall 4: Cross-lingual Capability Degradation After Multilingual Model Fine-tuning

  • Symptom: Your business requires multilingual retrieval, but fine-tuning bge-m3 only with Chinese data degrades its ability to retrieve English documents.
  • Cause: Model parameters shift excessively towards the Chinese semantic space.
  • Countermeasure: Mix a proportion of positive and negative samples from other languages into the fine-tuning data, or use an even smaller learning rate for all layers of the model during fine-tuning to limit the magnitude of changes.

9. Summary and Extension: From Embedding to a Complete RAG Offline Architecture

At this point, we have fully walked through the entire process from Embedding model selection to domain-adaptive fine-tuning. You have gained a practical methodology that can be put into action.

Summary of Core Points:

  1. Selection is the foundation: Based on your language, text length, and budget, choose the best base model using the three-dimensional decision table or MTEB leaderboard.
  2. Fine-tuning is the key: High-quality domain data is the “gold mine” for fine-tuning. Make good use of LLMs to generate data and carefully construct positive and negative samples, especially hard negatives, which can exponentially improve retrieval accuracy.
  3. Evaluation is the standard: Quantitatively evaluate using metrics like Recall@k, NDCG@k, and guide direction with ablation studies to avoid blind parameter tuning.

Future Expansion Directions: A Single Embedding Can’t Solve Everything

A fine-tuned Embedding model is just one piece of the puzzle in a RAG offline architecture. Combining it with other technologies can build a more powerful and robust system.

  • Integration with Metadata Enhancement: You can also convert metadata like document creation time, author, and source into vectors or use them as filtering conditions, combining with semantic retrieval from embeddings. For example, first filter by metadata to get financial reports from 2023, then perform semantic retrieval.
  • Integration with Knowledge Graph Embeddings (Graph Embeddings): As mentioned in our previous article, “RAG Offline Part: Metadata Enhancement and Knowledge Graph Fusion Preprocessing,” you can build a domain knowledge graph and then use Graph Embedding techniques to map entities and relationships from the knowledge graph into the same vector space.

This way, your RAG system can not only retrieve documents but also directly retrieve knowledge about relationships between entities, understanding complex logical connections.

  • Integration with Vector Databases and RAG Pipelines: The fine-tuned Embedding model can be seamlessly integrated with vector databases like Milvus, Weaviate, Qdrant, etc. Simply encode your knowledge documents using the fine-tuned model and store them in the database. Then, during online querying, encode the user question with the same model and perform ANN (Approximate Nearest Neighbor) search in the database to complete the end-to-end RAG pipeline.

Preview of Next Article:
This series on the RAG offline part will continue. The next article will focus on RAG Offline Part: Multi-Source Heterogeneous Data Cleaning and Deduplication Strategies, discussing how to handle messy data from different sources and formats to build a solid data foundation for your RAG system. Stay tuned!

Conclusion

Through this article, I believe you have gained a deeper understanding of RAG. I suggest you practice more with real projects. If you have questions, feel free to discuss!