Choosing Embedding Models for RAG: Answers to the Questions Engineers Actually Ask

Choosing an embedding model for your RAG system is one of those decisions that feels simple until you are in production and retrieval quality is not where it needs to be. The answers to the most common questions are not difficult — but they are not obvious either, and the wrong defaults compound into real retrieval quality problems.

Here are the questions engineers actually ask when building RAG systems, answered directly.

What is the most important factor when choosing an embedding model?

Fit to your data distribution. Benchmark leaderboard rankings (MTEB scores, for example) reflect performance across a broad set of tasks and datasets, not performance on your specific corpus and query types.

The most important evaluation you can do is run candidate embedding models against your own labeled dataset — 50 to 200 query-answer pairs that represent real user behavior. Measure Recall@5 for each model on your data. The model that wins on your benchmark is the right model, regardless of where it ranks on general leaderboards.

Does embedding dimension matter for retrieval quality?

Larger dimensions generally capture more semantic nuance, which can improve retrieval quality on complex corpora. But the relationship is not linear, and the cost scales with dimension:

• Storage: a 3072-dimension embedding takes 4x the storage of a 768-dimension embedding
• Query latency: distance computation scales with dimension, though HNSW indexes compress this somewhat
• Cost: if using an API-based embedding service, you pay per token regardless of output dimension

Start with a mid-range dimension model (768 or 1024). Move to larger dimensions only if your evaluation shows a retrieval quality gap that cannot be closed by other means.

Some models (including OpenAI's text-embedding-3 series) support Matryoshka Representation Learning, which lets you truncate embedding dimensions at inference time without significant quality loss. This gives you flexibility to tune the dimension-quality trade-off after indexing.

How much does the embedding model context window matter?

It matters a great deal if your chunks are longer than 512 tokens. Most popular embedding models have a 512-token context window. Text beyond that limit is typically truncated. If your chunking strategy produces 800-token chunks and your embedding model has a 512-token context window, roughly a third of every chunk is invisible to the embedding.

Match your chunk size to your model's context window. If you need longer chunks — for technical documentation or long-form content — use a model with a longer context window (JinaAI, Voyage AI, and others offer 8k+ context window embedding models).

Should I use an API-based model or self-host?

This is primarily a cost and latency trade-off:

API-based models (OpenAI text-embedding-3, Voyage AI, Cohere Embed) are simple to integrate, require no infrastructure, and are updated by the provider. They are appropriate when embedding volume is moderate and operational simplicity is a priority. The risk: if you re-embed a large corpus, per-token API costs can be significant. And if the provider deprecates the model, you will need to re-embed.

Self-hosted models (Sentence-Transformers, E5, BGE) run on your own infrastructure. Embedding a large corpus costs compute, not per-token API fees. Latency is lower and predictable. The trade-off is infrastructure — you are responsible for model serving, GPU provisioning, and updates.

For large corpora or high-throughput applications, self-hosted models are usually more economical. For smaller corpora or teams prioritizing simplicity, API-based models are the sensible starting point.

How do I handle multilingual corpora?

Use a multilingual embedding model from the start if your corpus or user queries are in multiple languages. Retrofitting multilingual support requires full re-indexing.

Models like multilingual-e5-large, BGE-M3, and some Cohere and Voyage models support a wide range of languages in a single embedding space. Multilingual models tend to underperform language-specific models on any single language, but they enable cross-lingual retrieval (a French query matching an English document) that single-language models cannot do.

If your queries and documents are always in the same language, use a language-specific model optimized for that language rather than a multilingual model.

What is fine-tuning an embedding model, and when does it help?

Fine-tuning adapts a pre-trained embedding model to your specific domain by training it on domain-specific (query, relevant document) pairs. The result is a model whose embedding space is calibrated to your vocabulary, query patterns, and document structure.

Fine-tuning helps when:

• Your domain has specialized vocabulary that general-purpose models do not represent well (medical, legal, scientific, financial)
• Your query patterns are systematically different from the model's training distribution
• You have a labeled dataset of (query, relevant document) pairs to fine-tune on

Fine-tuning requires labeled data (at minimum several hundred pairs, ideally thousands) and compute resources. It is not the right first step — exhaust off-the-shelf model selection and retrieval architecture improvements first. Fine-tuning is appropriate when those measures have been tried and retrieval quality is still insufficient.

Can I mix embeddings from different models in the same index?

No. Embeddings from different models live in different vector spaces and are not comparable. A similarity score between a query embedded with model A and a document embedded with model B is meaningless. Every document in your index must be embedded with the same model version, and queries must be embedded with the same model at retrieval time.

This has a practical implication: when you change embedding models, you must re-index the entire corpus. There is no partial migration. Plan for this by storing your raw document and chunk text in a separate store so re-indexing is a batch job, not an emergency.

What happens when an embedding model is deprecated?

If you are using an API-based embedding model and the provider deprecates it, you have a migration window to re-embed with the new model. Missing that window means you cannot add new documents to your index without inconsistency — new documents would use the new model, old documents would use the old model, and similarity scores would be unreliable across the corpus.

Mitigate this by: tracking which model version was used to embed each chunk (store it as chunk metadata), subscribing to provider deprecation notices, and having a tested re-indexing pipeline ready to run.

What is the practical recommendation for most RAG projects?

Start with a well-supported, mid-range model (768-1024 dimensions, 512-token context window) from a reputable provider. Run your labeled evaluation before committing to it. Enable hybrid search — the embedding model is one component of retrieval quality, not the whole answer. Add a reranker. Revisit the model choice when you have production data showing specific retrieval failures.

If you are building a RAG system and want experienced help making these infrastructure decisions, talk to the Clixo team. We design and ship production retrieval systems for AI-powered products.