InferenceCapabilities

Tool calling

How to give the model access to your program's functions, so that it can ask to call them and use the result in its reply.

With tool calling, the model can use the functions of your program, for example to find out the weather in a city. The model does not run the function itself: it asks you to call it and passes the arguments, while you run the call on your side and return the result. This is needed when the model's reply requires data or actions it does not have.

Model support

The API passes the tools field to the model as is, in the OpenAI format. Whether a model can call tools is shown in the catalog and in the response of the GET /v1/models method: the tools flag in the capabilities field.

Describing a tool in the request

{
  "model": "yenisei-100",
  "messages": [{"role": "user", "content": "What is the weather in Krasnoyarsk?"}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Weather in a city",
      "parameters": {
        "type": "object",
        "properties": { "city": { "type": "string" } },
        "required": ["city"]
      }
    }
  }]
}

Model reply and the call result

Instead of text, the model replies with a request to call the function: it comes in the choices[].message.tool_calls field. You call the function on your side and send the next request, adding to messages a message with the tool role and the result. After that, the model replies in words.