Skip to main content
Post

AI Gateway now supports Private Inference

Vercel added Private Inference to AI Gateway, so you can connect your own model endpoints and call them like catalog models. That covers fine-tuned, self-hosted, and prerelease models. They use the same authentication, SDKs, APIs, and observability as the catalog models, so they don't need separate keys, client code, or logs.

To set one up, open the Private Inference page and add your endpoint as a private provider. Enter its base URL, pick its API format, and add credentials. Then add the models it serves. The Vercel CLI (version 65.16.0 or later) can do the same with vercel ai-gateway private-inference providers create and models create. For example, a vLLM endpoint's fine-tune can be exposed as private/support-fine-tune.

You call the model by that private/ ID through Chat Completions, Responses, or Messages. AI Gateway translates each request to your endpoint's format, so switching between a private and a catalog model only changes the model ID. Private models also work in coding agents connected to the gateway, including Claude Code, Codex, and Cursor.

  • Only your Vercel team can view or call private models, and they aren't listed in the public catalog.
  • Credentials are stored encrypted, and you can turn on Zero Data Retention per provider.
  • Private model requests show up next to catalog traffic in logs, trace drains, and custom reporting. Observability charts break down requests, tokens, and time to first token by private model.
  • You can optionally set a catalog model as a fallback if your private endpoint fails.
Loading replies