Try approved models in order
Choose a primary model and backups. A fallback happens only for the failure types you allow, such as a rate limit, provider fault, slow start, or context problem.
Multi-provider routing
LLM routing that keeps requests moving
A model can slow down, reject a request, or go offline. LLMRegistry lets you save the path once, call it by one route name, and change the models later without changing your application.
- Route choices
- Direct, cascade, or fixed stages
- Fallback signals
- Rate, fault, speed, and context
- Recorded result
- One request and cost trail
One route name, clear rules
Your application sends a route alias as the model name. The gateway checks the API key, budget, model access, provider connection, and route policy before it contacts a provider.
Pause a failing provider
Circuit breakers watch provider results. Repeated temporary failures open the circuit, hold traffic back, and allow careful recovery checks after a cooldown.
Split a hard request into small jobs
Use up to four fixed stages chosen from Prepare, Plan, Execute, Check, Repair, and Format. Execute is required, and Repair can run once after a failed Check.
What happens to a routed request
The route stays inside the same authorization, spending, and usage boundary as a direct model request.
Check access and money first
LLMRegistry verifies the key, model scope, IP rules, quota, wallet, and route spending limit before provider work starts.
Choose an eligible provider path
The gateway uses the saved order and current provider health. It never adds a model that is outside the route or the key’s allowed model list.
Fail over only when the policy permits it
If a request fails before output begins, the route may try the next candidate. Once a stream has started, an interruption ends that stream instead of mixing two model answers.
Settle one complete record
The request ledger records the chosen model, provider, tokens, cost, timing, and outcome. Prompt content is not needed for the routing trail.
Choose the smallest routing tool that solves the problem
More steps are not always better. Start with a normal cascade, then add racing or fixed stages only when the request needs them.
A good fit
- Your service must survive a provider rate limit or outage.
- You want to change model order without shipping new application code.
- Different stages need different models or limits.
- You need one cost and outcome record for the whole request.
Important limits
- Model racing sends more than one provider request and both attempts may cost money.
- A stream cannot silently switch models after output reaches the user.
- Fixed stages are linear and limited to four, not a general workflow engine.
- Provider availability still depends on your connections and current catalog.
Related LLM gateway controls
Build a route around the outcome you need
Create an organization, connect an approved provider, and keep routing policy outside your application code.
Get started