Working with images: vision for documents and screenshots

Vision models cover what classic OCR struggles with: handwritten notes, skewed scans, tables with complex layouts, interface screenshots. The model recognizes both the characters and the structure: where the heading is, where the total is, where the signature is.

How to pass an image

The image goes into content as a separate element, either a URL or base64:

import base64
with open("invoice.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()
resp = client.chat.completions.create(
    model="qwen-flash",
    messages=[{"role": "user", "content": [
        {"type": "text", "text": "Extract the supplier, amount, and date from the invoice. Return JSON."},
        {"type": "image_url", "image_url": {
            "url": f"data:image/png;base64,{b64}"}},
    ]}],
    response_format={"type": "json_object"},
)

Choosing a model

• Qwen3.8 Flash (qwen-flash) is strong on documents and tables, has a large context, and handles Russian text on scans well.
• DeepSeek V4 Flash Vision (deepseek-flash-vision), Gemma 4 31B (gemma-4-31b) are solid options for interface screenshots and diagrams.
• Yenisei Docs (yenisei-doc) is our model for documents: it accepts whole files.
• For clean printed text with no structure, classic OCR is still cheaper; use vision where you need an understanding of the layout, not just the characters.

Extracting structure

Vision is especially good when you need specific fields from a document rather than "all the text". Combine it with JSON mode (as in the example above) and a strict schema. For tables, ask for an array of objects:

Extract the table as a JSON array, each row an object with keys
from the header. Empty cells are null. Do not invent rows that are not there.

Image tokens: where the money leaks

An image costs tokens, and quite a few: a large picture can eat as much as a page of text, or more. Rules for saving:

• Downscale the image before sending. For documents, the model usually does not need detail above about 1500px on the long side. Halving the size cuts the bill without hurting quality.
• Do not send a multi-page PDF as one stack of images "just in case". Process the pages you need.
• The detail setting (where supported): low for "roughly understand what is in the picture", high for small text in a table.

Pitfalls

• The model "reads" what is not there. On a poor scan, vision tends to fill in plausible digits. For critical fields (amounts!) add a check: a checksum, a range check, a "low confidence" flag.
• Rotation and skew. A document photographed at an angle lowers accuracy. Straighten it before sending if you control the source.
• Russian handwriting. Printed text reads very well; handwritten Russian comes with errors. Do not rely on it without a human check.
• Shifted fields in tables. If the columns are narrow, the model shifts values to the neighboring row. Ask it to verify that the number of rows matches the original.

A practical scenario

Processing incoming invoices: scan, then a vision model with a JSON schema, then validation of the amount and date; if confidence is low or the amount is out of range, send to manual review, otherwise post it to accounting. A pipeline like this takes the routine of data entry off people, leaving them only the disputed cases. Estimate the cost by image tokens on a real sample: 1 credit = 1 ₽, and large scans can be a surprise.