> ## Documentation Index > Fetch the complete documentation index at: https://www.latitude.sh/docs/llms.txt > Use this file to discover all available pages before exploring further. # Inference > Access to AI models through a unified, OpenAI-compatible API with support for text, vision, code, and reasoning models Latitude.sh AI Inference provides access to AI models through a unified, OpenAI-compatible API. Run open-weight text, vision, and reasoning models from DeepSeek, Z.ai, OpenAI, and Qwen. > [!NOTE] > > AI Inference is in preview. [Contact support](https://www.latitude.sh/docs/support/contact-support.md) to request access. Models, prices, and limits may change. See [Availability during preview](#availability-during-preview) for what to expect. ## Features * **API keys**: Generate and manage keys to authenticate your API requests * **Endpoints**: Browse available models with pricing, context length, and capabilities * **Playground**: Test models interactively before integrating them ## Managing API keys **1. Access API keys** [Log in to the dashboard](https://www.latitude.sh/dashboard), select a project, and navigate to **AI Inference > API keys**. **2. Create a key** Click **Create key** and provide a name for your key. **3. Copy your key** Copy the generated API key immediately. For security, the full key is only shown once. API keys are prefixed with `aitok_` and can be deleted at any time from the API keys page. ## Browsing models **1. Access the endpoints page** [Log in to the dashboard](https://www.latitude.sh/dashboard), select a project, and navigate to **AI Inference > Endpoints**. **2. Filter models** Click **Create endpoint** to search models by name and filter by provider or capability (text, vision, code, reasoning). **3. View model details** Select a model, then click **Create endpoint** to see its endpoint URL, model ID, provider, context length, and input/output pricing per million tokens. ## Using the playground **1. Access the playground** [Log in to the dashboard](https://www.latitude.sh/dashboard), select a project, and navigate to **AI Inference > Playground**. **2. Configure your request** Pick a model and an API key with the **Ready** status. If the **Key value** field appears, paste the full key. Click **Try out**, then use the settings button to adjust the system prompt, temperature, and max tokens. **3. Send a message** Type your prompt and send it to see the model's response in real-time. ## Making API requests The AI Inference API is available at `https://api.lsh.ai` and is fully compatible with the OpenAI SDK. You can also make direct HTTP requests. ```python Python theme={null} from openai import OpenAI client = OpenAI( base_url="https://api.lsh.ai/v1", api_key="YOUR_API_KEY", ) response = client.chat.completions.create( model="deepseek-v4.1-flash", messages=[{"role": "user", "content": "Hello!"}], ) print(response.choices[0].message.content) ``` ```bash cURL theme={null} curl https://api.lsh.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "model": "deepseek-v4.1-flash", "messages": [{"role": "user", "content": "Hello!"}] }' ``` ```javascript Node.js theme={null} import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://api.lsh.ai/v1", apiKey: "YOUR_API_KEY", }); const response = await client.chat.completions.create({ model: "deepseek-v4.1-flash", messages: [{ role: "user", content: "Hello!" }], }); console.log(response.choices[0].message.content); ``` Replace `YOUR_API_KEY` with your API key. You can find available model IDs on the [Endpoints page](#browsing-models) or with [`GET /v1/models`](#listing-models). The SDK examples below use the `client` created above. > [!NOTE] > > Send a `User-Agent` header with every request. The OpenAI SDKs and cURL send one. Python's `urllib` sends a default that's blocked with `403` and `error code: 1010`, so set your own when you use it. ## Listing models `GET /v1/models` lists the models currently offered. It requires a valid API key. Use a model's `id` as the `model` value in your requests. ```python Python theme={null} for model in client.models.list(): print(model.id) ``` ```bash cURL theme={null} curl https://api.lsh.ai/v1/models \ -H "Authorization: Bearer YOUR_API_KEY" ``` ```javascript Node.js theme={null} for await (const model of client.models.list()) { console.log(model.id); } ``` ## Models | Model ID | Context length | Input | Cached input | Output | Capabilities | | - | - | - | - | - | - | | `deepseek-v4.1-flash` | 1,048,576 | \$0.18 | \$0.005 | \$0.70 | Text, vision, tools, reasoning | | `glm-5.3` | 131,072 | \$0.60 | \$0.12 | \$2.60 | Text, tools, reasoning | | `glm-5.3-flash` | 131,072 | \$0.12 | \$0.025 | \$0.42 | Text, tools, reasoning | | `gpt-oss-120b` | 32,768 | \$0.08 | \$0.04 | \$0.40 | Text, tools, reasoning | | `qwen3.8-27b` | 131,072 | \$0.15 | — | \$1.90 | Text, vision, tools, reasoning | Prices are in USD per million tokens. The context length counts input and output tokens together. [`GET /v1/models`](#listing-models) returns each model's `context_length` and `capabilities`. Every response reports its token counts in `usage`. Input tokens include the model's chat template, so they're slightly higher than the tokens in your messages alone. When a request starts with the same tokens as a recent one, those input tokens come from the prompt cache. They're reported in `usage.prompt_tokens_details.cached_tokens` and billed at the cached input price. `qwen3.8-27b` doesn't use the prompt cache. ## Reasoning Every model reasons before it answers unless you turn reasoning off. The reasoning comes back apart from the answer, in `message.reasoning`, or in `message.reasoning_content` for `glm-5.3` and `glm-5.3-flash`. When streaming, each chunk's `delta` uses the same field. Reasoning tokens are billed as output tokens. Set `reasoning_effort` to choose how much a model reasons. To turn reasoning off, pass `"chat_template_kwargs": {"enable_thinking": false}`. A `reasoning_effort` value that a model doesn't accept returns a `400` error. | Model | `reasoning_effort` values | Turn reasoning off | | - | - | - | | `deepseek-v4.1-flash` | `none`, `low`, `high`, `xhigh`, `max` | `none` or `enable_thinking: false` | | `glm-5.3` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max` | Not supported | | `glm-5.3-flash` | `none`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max` | Not supported | | `gpt-oss-120b` | `low`, `medium`, `high` | Not supported | | `qwen3.8-27b` | `none`, `low`, `medium`, `xhigh` | `none` or `enable_thinking: false` | * **glm-5.3** always reasons. It groups the values into three levels: `none`, `minimal`, `low`, and `enable_thinking: false` reason briefly, `medium` and `high` reason more, and `xhigh`, `max`, and the default reason the most. * **glm-5.3-flash** always reasons. It groups the values into three levels: `none`, `minimal`, `low`, and `enable_thinking: false` reason briefly, `medium` and `high` reason more, and `xhigh`, `max`, and the default reason the most. * **gpt-oss-120b** always reasons and ignores `enable_thinking`. Its `usage.completion_tokens_details.reasoning_tokens` is always `0`. The reasoning tokens are still counted, and billed, in `completion_tokens`. ```python Python theme={null} response = client.chat.completions.create( model="deepseek-v4.1-flash", messages=[{"role": "user", "content": "What is 17 * 23?"}], extra_body={"chat_template_kwargs": {"enable_thinking": False}}, ) ``` ```bash cURL theme={null} curl https://api.lsh.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "model": "deepseek-v4.1-flash", "messages": [{"role": "user", "content": "What is 17 * 23?"}], "reasoning_effort": "low" }' ``` ## Tool calling Pass your functions in `tools`. When the model decides to call one, the response has `tool_calls` instead of an answer. Run the function, append each result as a `tool` message, and send the conversation again to get the final answer. Every model on this page supports tool calling. ```python Python theme={null} import json tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather in a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, }, }] messages = [{"role": "user", "content": "What's the weather in Paris?"}] response = client.chat.completions.create(model="deepseek-v4.1-flash", messages=messages, tools=tools) message = response.choices[0].message if message.tool_calls: messages.append(message) for call in message.tool_calls: args = json.loads(call.function.arguments) result = {"city": args["city"], "temperature_c": 18} # call your own function here messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}) response = client.chat.completions.create(model="deepseek-v4.1-flash", messages=messages, tools=tools) print(response.choices[0].message.content) ``` ```bash cURL theme={null} curl https://api.lsh.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "model": "deepseek-v4.1-flash", "messages": [{"role": "user", "content": "What'"'"'s the weather in Paris?"}], "tools": [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather in a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"] } } }] }' ``` ## Structured outputs To get JSON that matches a schema, set `response_format` to `json_schema` with your [JSON Schema](https://json-schema.org/). The answer in `message.content` is a JSON string that follows the schema. Every model on this page supports structured outputs. ```python Python theme={null} import json response = client.chat.completions.create( model="deepseek-v4.1-flash", messages=[{"role": "user", "content": "What's the largest city in Japan?"}], response_format={ "type": "json_schema", "json_schema": { "name": "city", "strict": True, "schema": { "type": "object", "properties": { "city": {"type": "string"}, "country": {"type": "string"}, "population": {"type": "integer"}, }, "required": ["city", "country", "population"], "additionalProperties": False, }, }, }, ) print(json.loads(response.choices[0].message.content)) ``` ```bash cURL theme={null} curl https://api.lsh.ai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "model": "deepseek-v4.1-flash", "messages": [{"role": "user", "content": "What'"'"'s the largest city in Japan?"}], "response_format": { "type": "json_schema", "json_schema": { "name": "city", "strict": true, "schema": { "type": "object", "properties": { "city": {"type": "string"}, "country": {"type": "string"}, "population": {"type": "integer"} }, "required": ["city", "country", "population"], "additionalProperties": false } } } }' ``` ## Streaming responses Set `stream: true` to receive the response as server-sent events while the model generates it. Each event is a `data:` line that carries a JSON chunk, and the stream ends with `data: [DONE]`. For chat completions, the last chunk before `[DONE]` has an empty `choices` array and a `usage` object with the token counts. It's always sent, even if you don't set `stream_options.include_usage`. ```python Python theme={null} stream = client.chat.completions.create( model="deepseek-v4.1-flash", messages=[{"role": "user", "content": "Hello!"}], stream=True, ) for chunk in stream: if chunk.choices: print(chunk.choices[0].delta.content or "", end="", flush=True) if chunk.usage: print(f"\nTotal tokens: {chunk.usage.total_tokens}") ``` ```bash cURL theme={null} curl https://api.lsh.ai/v1/chat/completions \ -N \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_API_KEY" \ -d '{ "model": "deepseek-v4.1-flash", "messages": [{"role": "user", "content": "Hello!"}], "stream": true }' ``` ```javascript Node.js theme={null} const stream = await client.chat.completions.create({ model: "deepseek-v4.1-flash", messages: [{ role: "user", content: "Hello!" }], stream: true, }); for await (const chunk of stream) { if (chunk.choices.length) process.stdout.write(chunk.choices[0].delta.content ?? ""); if (chunk.usage) console.log(`\nTotal tokens: ${chunk.usage.total_tokens}`); } ``` ## Long generations Use streaming for any response that can take longer than about 2 minutes to complete. A non-streaming response arrives only once it's complete, so a non-streaming request that takes longer than about 2 minutes fails with a `504` error. A streaming response sends tokens as they're generated, which keeps the connection active. Every request, including streams, is limited to 20 minutes. A stream that sends nothing for about 2 minutes, or reaches that limit, ends without `data: [DONE]`. ## Canceling requests Close the connection to cancel a streaming request. Generation stops when the connection closes. * **Python SDK**: call `stream.close()`, or `break` out of a `with` block that wraps the stream. * **Node.js SDK**: `break` out of the `for await` loop, or call `stream.controller.abort()`. * **cURL**: press Ctrl+C. ## Request IDs Every response, including errors, carries an `x-request-id` header. Include it when you contact support about a request. The OpenAI SDKs expose this header as `response._request_id`. When a request fails, read it from the error: `error.request_id` in Python, or `error.requestID` in Node.js. With cURL, add `-i` to print the response headers. ## Rate limits Each project has its own limits on each model: * **Requests**: 60 per minute. * **Tokens**: 250,000 per minute, input and output combined. A request's tokens count when it finishes, so one long request can use up the budget and hold the next request until the budget refills. * **Concurrent requests**: a share of the model's capacity, and at least 4. The error message states your current limit. A request over a limit gets a `429` with the `rate_limit_exceeded` code and a `Retry-After` header. Wait that many seconds before you retry. ```json theme={null} { "error": { "message": "rate limit exceeded", "type": "inference_error", "code": "rate_limit_exceeded", "detail": "This project is over its limit of 12 concurrent requests on this model." } } ``` Contact support if your workload needs higher limits. ## Errors Errors use the OpenAI error format. Use the HTTP status to decide how to handle an error; `error.code` identifies the cause. ```json theme={null} { "error": { "message": "model not found", "type": "inference_error", "code": "model_not_found" } } ``` The table lists the common codes. Other codes can appear. | Status | Code | Meaning | | - | - | - | | `400` | `invalid_request` | The request is invalid, for example `model` is missing. Errors the model returns, such as a prompt longer than its context length, carry the model's own message and code. | | `401` | `invalid_api_key` | The API key is missing, invalid, or revoked. Check that the key has the **Ready** status. | | `404` | `model_not_found` | The model isn't offered. Check the ID with [`GET /v1/models`](#listing-models). | | `404` | `unsupported_endpoint` | The API doesn't serve this path. | | `405` | `method_not_allowed` | The path doesn't accept this HTTP method. The `Allow` header lists the method it accepts. | | `413` | `request_too_large` | The request body is larger than 15 MiB. | | `429` | `rate_limit_exceeded` | Your project is over one of its [rate limits](#rate-limits). Retry after the `Retry-After` delay. | | `429` | `capacity_exhausted` | The model is at capacity. Retry after the `Retry-After` delay. | | `502` | `upstream_unreachable`, `upstream_error` | The connection to the model failed, and the request may have run. | | `503` | `model_unavailable`, `cluster_unavailable`, `routing_unavailable`, `deployment_not_ready`, `model_not_deployed`, `identity_unavailable`, `admission_unavailable`, `upstream_unavailable`, `edge_overloaded` | The model can't be served right now. Retry after the `Retry-After` delay, when present. | | `504` | `504`, `timeout`, `upstream_timeout` | The model sent nothing for about 2 minutes, or the connection to it timed out. Use [streaming](#streaming-responses) for long generations. | ## Retrying requests A response with `x-should-retry: true` is safe to retry, and `x-should-retry: false` means it isn't. Every `5xx` response carries the header: it's `false` whenever the request may already have run, such as a `502` or a `504` timeout. The OpenAI SDKs follow this header and the `Retry-After` delay automatically. If you use another client, retry only responses marked `x-should-retry: true`, after the `Retry-After` delay. Never retry a stream that has already returned events. A stream that ends without `data: [DONE]` was interrupted, and a retry runs the whole request again. ## Availability during preview AI Inference is in preview, and some models run in a single location. While that location has a hardware failure, or while we update the model, requests to that model fail with a `503` until it's back. Retry `503` responses after the `Retry-After` delay. The list of models can change. Check [`GET /v1/models`](#listing-models) for the models offered now. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.