RAG on embeddings: how to connect a model to your knowledge base

You need RAG when a model has to answer from your own data: documentation, tickets, contracts, things that are not in its weights. Fine-tuning for that is expensive and slow. It is cheaper to find the relevant pieces of text and put them into the prompt.

The four-step scheme

1. Split documents into chunks.
2. Compute embeddings via POST /v1/embeddings.
3. Store the vectors (pgvector, Qdrant, or even numpy for a prototype).
4. On each query, embed the question, find the nearest chunks, and hand them to the model.

Embeddings with bge-m3

bge-m3 is an everyday model: multilingual (it handles Russian confidently), 1024 dimensions, up to 8K input tokens. For most knowledge bases that is enough.

from openai import OpenAI
client = OpenAI(api_key=KEY, base_url="https://api.yenisei.ru/v1")
def embed(texts: list[str]) -> list[list[float]]:
    resp = client.embeddings.create(model="bge-m3", input=texts)
    return [d.embedding for d in resp.data]

Send texts in batches of 32 to 64: fewer round trips. Embeddings are cheap (around 1 to 2 ₽ per 1M tokens), but indexing a large base still adds up, so estimate it in advance.

Chunking matters more than the model

The classic beginner mistake is cutting every 512 characters, blindly. A sentence gets torn in the middle and the vector becomes meaningless. What works:

• Split along structure: headings, paragraphs, list items.
• Keep chunks at 200 to 400 tokens with 15 to 20% overlap.
• Store metadata alongside: source, section heading, date. You will later show it to the user as a link.

Search and context assembly

Cosine similarity in pgvector:

SELECT chunk, source, 1 - (embedding <=> %(q)s) AS score
FROM kb
ORDER BY embedding <=> %(q)s
LIMIT 8;

Eight chunks is a reasonable start. Then cut off by a score threshold (for bge-m3 usually 0.5 to 0.6) and put the rest into the prompt:

context = "\n\n".join(f"[{c['source']}] {c['chunk']}" for c in hits)
messages = [
    {"role": "system", "content":
     "Answer ONLY from the context below. If the answer is not there, say you don't know."},
    {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
]
resp = client.chat.completions.create(model="qwen-flash", messages=messages)

Where RAG breaks

• The model ignores the context and answers from memory. Fix it with a strict system prompt and an explicit requirement to cite the source.
• Similar chunks, but not the right ones. Pure vector search confuses sections with similar wording. Add hybrid search: BM25 on keywords plus the vector, and merge the rankings (reciprocal rank fusion).
• Context that is too long. Do not stuff in 50 chunks "just in case": the model drowns and the bill grows. Six to eight precise ones are better.
• A stale base. Embeddings do not update themselves. Recompute changed documents on a save hook.

Which model to use for generation

Answering from ready context is an easy task; heavy reasoning is not needed here. Start with a fast, cheap model (qwen-flash, deepseek-flash). If the answers come out shallow on hard questions, move up to deepseek-pro. Embeddings stay on bge-m3 either way: search quality does not depend on the generating model.

Build a minimal version first, on numpy and ten documents, make sure the search finds what it should, and only then bring in Qdrant and hybrid search. Most RAG problems come from chunking, not from infrastructure.