Axiom StudioAXIOMSTUDIO
All docs

Axiom LLM Gateway

Axiom LLM Gateway gives teams one controlled entry point for AI inference across managed and self-hosted model providers. Applications use an OpenAI-compatible API while Axiom handles provider credentials, model routing, fallback behavior, load balancing, usage analytics, audit logs, and Prometheus-compatible metrics.

Use these docs when you are setting up the gateway for the first time, adding providers, connecting an application, or operating the gateway in production.

What the gateway provides

  • One API for multiple providers: Use OpenAI-compatible chat and embeddings endpoints for supported providers such as OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, Google Vertex AI, Mistral, Cohere, Groq, Ollama, OpenRouter, Perplexity, Cerebras, Hugging Face, ElevenLabs, Nebius, xAI, and Parasail.
  • Centralized credentials: Store provider credentials once, scope them to your organization, disable or rotate them without changing application code, and keep provider-specific settings with the credential.
  • Model-aware routing: Route requests by requested model and credential configuration so the right provider key is used for the workload.
  • Weighted load balancing: Split traffic across credentials within the same provider and model group.
  • Fallback routing: Define ordered fallback paths so traffic can continue when a primary provider is disabled, unavailable, rate limited, or returning errors.
  • Streaming support: Use Server-Sent Events for streaming chat completions, and WebSockets for realtime transcription and streaming text-to-speech.
  • Observability: Track request rate, latency, provider duration, token usage, costs, errors, and active requests through the embedded UI and metrics endpoint.
  • Auditability: Review credential changes, fallback events, and provider errors with tenant-scoped audit logs.
  • Kubernetes-native operation: Run as part of Axiom with embedded management UI, REST API, inference gateway, and metrics endpoints.

Documentation map

DocumentUse it for
Getting startedFirst provider, first request, and application connection steps.
Configuration guideCredentials, provider settings, session/API access, and environment planning.
Routing and reliabilityModel routing, load balancing, fallback chains, and operational patterns.
Observability and FinOpsMetrics, dashboards, Prometheus/VictoriaMetrics, token usage, cost tracking, and budgets.
Operations and securityTenant isolation, credential hygiene, production deployment, and runbook practices.

Core endpoints

The LLM Gateway is exposed under your Axiom Cloud base URL:

https://cloud.axiomstudio.ai/rest/v1/llm-gateway

OpenAI-compatible inference endpoints are available below:

/rest/v1/llm-gateway/v1/chat/completions
/rest/v1/llm-gateway/v1/embeddings

Audio endpoints:

POST /rest/v1/llm-gateway/v1/audio/speech                   text to speech (whole text, buffered or streamed audio)
GET  /rest/v1/llm-gateway/v1/audio/speech/realtime          streaming text to speech (WebSocket, ElevenLabs)
POST /rest/v1/llm-gateway/v1/audio/transcriptions           speech to text (file upload)
GET  /rest/v1/llm-gateway/v1/audio/transcriptions/realtime  realtime speech to text (WebSocket, ElevenLabs)
GET  /rest/v1/llm-gateway/v1/audio/voices                   voices a speech model can use

Management APIs, such as credentials, providers, analytics, fallback configuration, metrics queries, overview, and audit logs, are exposed under:

/rest/v1/llm-gateway/*

Metrics for Prometheus-compatible scrapers are served through the shared AI gateway metrics endpoint:

/metrics/ai-gateway

Streaming text-to-speech

GET /rest/v1/llm-gateway/v1/audio/speech/realtime upgrades to a WebSocket. Push text as you have it, for example an LLM reply token by token, and audio comes back while you are still sending. It serves ElevenLabs speech models; the gateway opens ElevenLabs' input-streaming socket with the selected credential, so the provider key never reaches the client. Eleven v3 and v4 models (eleven_v4, eleven_v4_turbo) run on ElevenLabs' text-to-dialogue socket and the others (eleven_flash_v2_5, eleven_multilingual_v2, ...) on its text-to-speech socket; the protocol below is the same for both.

Query parameters:

ParameterRequiredDescription
modelYeselevenlabs/<model>, for example elevenlabs/eleven_flash_v2_5
voiceYesVoice name or ID. Built-in models accept only their premade (or operator-listed) voices.
output_formatNopcm_8000, pcm_16000, pcm_22050, pcm_24000 (default), pcm_44100, pcm_48000, ulaw_8000 or mp3_44100_128
language_codeNoISO 639 language code
apply_text_normalizationNoauto, on or off
seedNo0 to 4294967295
credential_sourceNobuilt_in uses only operator-enabled built-in models, never the org's own credential
refusalNoframe: a refused session (no credits, unknown voice, ...) is upgraded and answered with a {message_type, error, status} frame, since a browser can't read a refused handshake

Any other parameter is refused. A browser must open the socket from an allowed Axiom Cloud origin and authenticates with the session cookie, like the realtime transcription socket. Agent runtime grants can't open it.

Client frames are JSON text frames:

{"text": "Hello there. "}
{"text": "How can I help?", "flush": true}
{"flush": true}
{"end": true}

A text frame carries 1 to 5,000 characters. flush asks for audio for everything sent so far without waiting for more text; end says the text is complete and can be combined with a last text. ElevenLabs buffers text into natural chunks, so end words with a space or punctuation as an LLM stream naturally does.

Server frames:

  • {"message_type": "session_started", "model", "voice_id", "output_format"} once, first.
  • Binary frames: raw audio in output_format. For pcm_* that is signed 16-bit little-endian mono PCM at the format's sample rate.
  • {"message_type": "final", "characters": N} after the last audio, then a normal close. characters is what the session was billed for.
  • {"message_type": "input_error", "error": "..."} for a frame the gateway can't use; the session stays open.
  • {"message_type": <type>, "error": "..."} then a close for anything that ends the session: insufficient_credits, character_limit_exceeded, input_timeout_exceeded, session_time_limit_exceeded, auth_error (organization access ended), and ElevenLabs failures (provider_auth_error, quota_exceeded, rate_limited, invalid_voice, invalid_request, provider_error). Provider messages are never relayed.

Limits: 50,000 characters and 30 minutes per session, 180 seconds without a client frame before end, and 25 concurrent sessions per organization per gateway pod.

Metering: speech is billed per character (Unicode code points) of text sent to ElevenLabs. Text is paid for before it is sent, so text the balance can't cover never reaches the provider and the session closes with insufficient_credits. Built-in models are charged at their speech rate on the organization's credits: the gateway holds the cost of 1,000 characters ahead, charges each completed block, and charges only the characters used of the last one. An organization's own ElevenLabs credential pays one gateway credit per started 5,000 characters. A session that sends no text is not charged. Usage records (request type speech_realtime) carry the characters, so analytics and the ElevenLabs cost reconciliation count them with buffered speech.

const query = new URLSearchParams({ model: 'elevenlabs/eleven_flash_v2_5', voice: 'Rachel', output_format: 'pcm_24000' });
const socket = new WebSocket(`wss://cloud.axiomstudio.ai/rest/v1/llm-gateway/v1/audio/speech/realtime?${query}`);
socket.binaryType = 'arraybuffer';
socket.onmessage = (event) => {
  if (event.data instanceof ArrayBuffer) playPcm16(event.data, 24000); // your audio sink
  else console.log(JSON.parse(event.data));
};
socket.onopen = async () => {
  for await (const token of llmReplyTokens()) socket.send(JSON.stringify({ text: token }));
  socket.send(JSON.stringify({ end: true }));
};

Authentication model

Interactive users authenticate with their Axiom Cloud session. Application examples in this documentation show session-token placeholders because the gateway accepts OpenAI-compatible clients with the session token configured as the API key.

Metrics scraping uses a separate bearer token generated by an operator. Store that token in your monitoring system's secret store and reference it from Prometheus or VictoriaMetrics configuration.