Axiom LLM Gateway
Axiom LLM Gateway gives teams one controlled entry point for AI inference across managed and self-hosted model providers. Applications use an OpenAI-compatible API while Axiom handles provider credentials, model routing, fallback behavior, load balancing, usage analytics, audit logs, and Prometheus-compatible metrics.
Use these docs when you are setting up the gateway for the first time, adding providers, connecting an application, or operating the gateway in production.
What the gateway provides
- One API for multiple providers: Use OpenAI-compatible chat and embeddings endpoints for supported providers such as OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, Google Vertex AI, Mistral, Cohere, Groq, Ollama, OpenRouter, Perplexity, Cerebras, Hugging Face, ElevenLabs, Nebius, xAI, and Parasail.
- Centralized credentials: Store provider credentials once, scope them to your organization, disable or rotate them without changing application code, and keep provider-specific settings with the credential.
- Model-aware routing: Route requests by requested model and credential configuration so the right provider key is used for the workload.
- Weighted load balancing: Split traffic across credentials within the same provider and model group.
- Fallback routing: Define ordered fallback paths so traffic can continue when a primary provider is disabled, unavailable, rate limited, or returning errors.
- Streaming support: Use Server-Sent Events for streaming chat completions, and WebSockets for realtime transcription and streaming text-to-speech.
- Observability: Track request rate, latency, provider duration, token usage, costs, errors, and active requests through the embedded UI and metrics endpoint.
- Auditability: Review credential changes, fallback events, and provider errors with tenant-scoped audit logs.
- Kubernetes-native operation: Run as part of Axiom with embedded management UI, REST API, inference gateway, and metrics endpoints.
Documentation map
| Document | Use it for |
|---|---|
| Getting started | First provider, first request, and application connection steps. |
| Configuration guide | Credentials, provider settings, session/API access, and environment planning. |
| Routing and reliability | Model routing, load balancing, fallback chains, and operational patterns. |
| Observability and FinOps | Metrics, dashboards, Prometheus/VictoriaMetrics, token usage, cost tracking, and budgets. |
| Operations and security | Tenant isolation, credential hygiene, production deployment, and runbook practices. |
Core endpoints
The LLM Gateway is exposed under your Axiom Cloud base URL:
https://cloud.axiomstudio.ai/rest/v1/llm-gateway
OpenAI-compatible inference endpoints are available below:
/rest/v1/llm-gateway/v1/chat/completions
/rest/v1/llm-gateway/v1/embeddings
Audio endpoints:
POST /rest/v1/llm-gateway/v1/audio/speech text to speech (whole text, buffered or streamed audio)
GET /rest/v1/llm-gateway/v1/audio/speech/realtime streaming text to speech (WebSocket, ElevenLabs)
POST /rest/v1/llm-gateway/v1/audio/transcriptions speech to text (file upload)
GET /rest/v1/llm-gateway/v1/audio/transcriptions/realtime realtime speech to text (WebSocket, ElevenLabs)
GET /rest/v1/llm-gateway/v1/audio/voices voices a speech model can use
Management APIs, such as credentials, providers, analytics, fallback configuration, metrics queries, overview, and audit logs, are exposed under:
/rest/v1/llm-gateway/*
Metrics for Prometheus-compatible scrapers are served through the shared AI gateway metrics endpoint:
/metrics/ai-gateway
Streaming text-to-speech
GET /rest/v1/llm-gateway/v1/audio/speech/realtime upgrades to a WebSocket.
Push text as you have it, for example an LLM reply token by token, and audio
comes back while you are still sending. It serves ElevenLabs speech models;
the gateway opens ElevenLabs' input-streaming socket with the selected
credential, so the provider key never reaches the client. Eleven v3 and v4
models (eleven_v4, eleven_v4_turbo) run on ElevenLabs' text-to-dialogue
socket and the others (eleven_flash_v2_5, eleven_multilingual_v2, ...)
on its text-to-speech socket; the protocol below is the same for both.
Query parameters:
| Parameter | Required | Description |
|---|---|---|
model | Yes | elevenlabs/<model>, for example elevenlabs/eleven_flash_v2_5 |
voice | Yes | Voice name or ID. Built-in models accept only their premade (or operator-listed) voices. |
output_format | No | pcm_8000, pcm_16000, pcm_22050, pcm_24000 (default), pcm_44100, pcm_48000, ulaw_8000 or mp3_44100_128 |
language_code | No | ISO 639 language code |
apply_text_normalization | No | auto, on or off |
seed | No | 0 to 4294967295 |
credential_source | No | built_in uses only operator-enabled built-in models, never the org's own credential |
refusal | No | frame: a refused session (no credits, unknown voice, ...) is upgraded and answered with a {message_type, error, status} frame, since a browser can't read a refused handshake |
Any other parameter is refused. A browser must open the socket from an allowed Axiom Cloud origin and authenticates with the session cookie, like the realtime transcription socket. Agent runtime grants can't open it.
Client frames are JSON text frames:
{"text": "Hello there. "}
{"text": "How can I help?", "flush": true}
{"flush": true}
{"end": true}
A text frame carries 1 to 5,000 characters. flush asks for audio for
everything sent so far without waiting for more text; end says the text is
complete and can be combined with a last text. ElevenLabs buffers text
into natural chunks, so end words with a space or punctuation as an LLM
stream naturally does.
Server frames:
{"message_type": "session_started", "model", "voice_id", "output_format"}once, first.- Binary frames: raw audio in
output_format. Forpcm_*that is signed 16-bit little-endian mono PCM at the format's sample rate. {"message_type": "final", "characters": N}after the last audio, then a normal close.charactersis what the session was billed for.{"message_type": "input_error", "error": "..."}for a frame the gateway can't use; the session stays open.{"message_type": <type>, "error": "..."}then a close for anything that ends the session:insufficient_credits,character_limit_exceeded,input_timeout_exceeded,session_time_limit_exceeded,auth_error(organization access ended), and ElevenLabs failures (provider_auth_error,quota_exceeded,rate_limited,invalid_voice,invalid_request,provider_error). Provider messages are never relayed.
Limits: 50,000 characters and 30 minutes per session, 180 seconds without a
client frame before end, and 25 concurrent sessions per organization per
gateway pod.
Metering: speech is billed per character (Unicode code points) of text sent
to ElevenLabs. Text is paid for before it is sent, so text the balance can't
cover never reaches the provider and the session closes with
insufficient_credits. Built-in models are charged at their speech rate on
the organization's credits: the gateway holds the cost of 1,000 characters
ahead, charges each completed block, and charges only the characters used of
the last one. An organization's own ElevenLabs credential pays one gateway
credit per started 5,000 characters. A session that sends no text is not
charged. Usage records (request type speech_realtime) carry the
characters, so analytics and the ElevenLabs cost reconciliation count them
with buffered speech.
const query = new URLSearchParams({ model: 'elevenlabs/eleven_flash_v2_5', voice: 'Rachel', output_format: 'pcm_24000' });
const socket = new WebSocket(`wss://cloud.axiomstudio.ai/rest/v1/llm-gateway/v1/audio/speech/realtime?${query}`);
socket.binaryType = 'arraybuffer';
socket.onmessage = (event) => {
if (event.data instanceof ArrayBuffer) playPcm16(event.data, 24000); // your audio sink
else console.log(JSON.parse(event.data));
};
socket.onopen = async () => {
for await (const token of llmReplyTokens()) socket.send(JSON.stringify({ text: token }));
socket.send(JSON.stringify({ end: true }));
};
Authentication model
Interactive users authenticate with their Axiom Cloud session. Application examples in this documentation show session-token placeholders because the gateway accepts OpenAI-compatible clients with the session token configured as the API key.
Metrics scraping uses a separate bearer token generated by an operator. Store that token in your monitoring system's secret store and reference it from Prometheus or VictoriaMetrics configuration.