Voice and audio: scenarios with speech models

Speech models cover three common scenarios: transcription (calls and meetings into text), audio understanding (pulling decisions and tasks straight out of a recording), and voice bots (speech in, speech out). Each needs its own approach.

Transcription

The basic scenario is turning a recording into text. The recording goes in the message, and the transcript comes back:

import base64
with open("call.mp3", "rb") as f:
    b64_audio = base64.b64encode(f.read()).decode()
tr = client.chat.completions.create(
    model="voxtral-small",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Transcribe the recording verbatim, in Russian."},
        {"type": "input_audio", "input_audio": {"data": b64_audio, "format": "mp3"}},
    ]}],
)
print(tr.choices[0].message.content)

Voxtral Small handles Russian well, including conversational speech with noise and interruptions. State the language of the recording explicitly in the request: auto-detection sometimes fails on short or multilingual fragments.

Questions about a recording without a transcript

Models for text and sound like MiMo v2.5 take audio directly and answer questions about it right away; no intermediate text is needed. This is handy for "pull the agreements and deadlines out of this call":

resp = client.chat.completions.create(
    model="mimo-v2.5",
    messages=[{"role": "user", "content": [
        {"type": "text", "text":
         "This is a recording of a call. List the decisions made and the tasks with their owners."},
        {"type": "input_audio", "input_audio": {
            "data": b64_audio, "format": "mp3"}},
    ]}],
)

Long recordings

An hour-long meeting will not fit in one request, neither by context nor by limits. The strategy:

1. Transcribe the whole thing with Voxtral (transcription scales with length).
2. Cut the resulting text into pieces and compress each into a short summary.
3. Assemble the final summary from those.

For diarization (who said what), ask the model to label turns by speaker if the recording allows it, but labeling accuracy drops on overlapping speech, so plan for a check.

Voice bots

The full cycle: speech to text (Voxtral), the LLM processes it, text to speech (TTS). The bottleneck is latency: a user on a phone call will not tolerate pauses. So:

• On the LLM step, use a Flash model and streaming.
• Start speech synthesis without waiting for the text generation to finish: sentence by sentence.
• Keep the bot's turns short: a long answer in voice is more irritating than in chat.

Pitfalls

• Terms and names. The model writes "Yenisei" as "Енисей" or "Енисеи". Pass a hint with a list of names and terms in the prompt if the API supports it, or clean it up in post-processing.
• Format and bitrate. Overly compressed audio lowers accuracy. mp3/wav at normal quality is fine; 8kHz telephone audio is recognized worse.
• Silence and noise. Long pauses and background hum sometimes get "filled in" with words. On important transcripts, check the suspiciously smooth passages.
• Cost by duration. Audio is billed by minutes or tokens of the recording, not by words. Hour-long calls add up; estimate on a real sample.

Start with transcription of one type of call, assess the accuracy in Russian by hand on a dozen recordings, and only then build out understanding and the voice loop.