Support bots: cheap and fast on Flash models

A support bot is no place for a flagship model. 90% of requests are "how do I reset my password", "where is my order", "what payment methods do you accept". On questions like these the user will not notice the difference between a heavy and a light model, while the difference in the bill is an order of magnitude.

Bet on Flash/Mini

deepseek-flash, qwen-flash, gemini-flash are models for mass, cheap inference. They answer in a fraction of a second and cost pennies per 1M tokens. Do not expect deep reasoning from them, but on routine requests they are fast and stable, and that is exactly what the first line needs.

Streaming is mandatory

A user in a chat should not stare at "the bot is typing..." for three seconds. Turn on stream=True: the answer flows as it is generated, and the person starts reading right away:

stream = client.chat.completions.create(
    model="deepseek-flash",
    messages=messages,
    stream=True,
    temperature=0.3,
)
async for chunk in stream:
    token = chunk.choices[0].delta.content
    if token:
        await ws.send(token)

A low temperature (0.2 to 0.4) keeps the bot in line: fewer flights of fancy, more predictability. For support that is what you want.

RAG instead of memory

Do not make the model know your knowledge base by heart; it cannot, and it will start making things up. Feed it relevant articles from your documentation through embeddings (bge-m3) and constrain it firmly: "answer only from these materials, otherwise hand over to a human".

Escalation to a human

The bot must be able to say "I don't know" and call an operator. Set that up as a tool:

tools = [{"type": "function", "function": {
    "name": "escalate_to_human",
    "description": "Hand the conversation over to a live operator when the bot cannot help",
    "parameters": {"type": "object", "properties": {
        "reason": {"type": "string"}}, "required": ["reason"]},
}}]

Escalation triggers: three failed answers in a row, any mention of a refund or a complaint, a direct request like "get me a human". A cheap model handles these signals fine if you describe them in the system prompt.

Pitfalls

• The bot apologizes on every turn. Put it in the system prompt: "do not apologize without reason, answer to the point".
• It forgets the conversation. Flash models with a short context lose the beginning of a long exchange. Keep the history trimmed to the last 8 to 10 turns plus a summary of the older part.
• It quotes prices and terms from memory. Anything that changes (plans, deadlines) comes only through RAG or a tool, never from the model's weights.

What it costs

A rough estimate: an average conversation is 5 turns with about 300 tokens of answer each. On a Flash model that is a fraction of a ruble per conversation. Even 10,000 requests a month cost an amount that a single avoided call to a live operator pays for. 1 credit = 1 ₽, so count straight in rubles from your logs.

Build the first version on one Flash model with RAG and escalation, measure "the share of conversations closed without a human", and tune the prompt against that metric. The model is secondary here; what decides the outcome is the quality of the knowledge base and the escalation rules.