Two switches must agree
The organization cache policy and the selected route must both be enabled. New routes and new organization policies start with caching off.
Reuse with clear rules
An LLM semantic cache you can control
Many users ask the same question in slightly different words. A semantic cache can return an approved earlier response without making another provider request, while keeping every hit visible in your usage trail.
- Default state
- Off until you enable it
- Cache boundary
- Organization, project, route, and controls
- Provider cost on hit
- Zero
Match meaning without hiding the result
Exact text matching misses questions that mean the same thing. LLMRegistry compares a protected request representation inside a narrow policy boundary, then applies the stricter similarity and expiration rules from the organization and route.
Keep unrelated traffic apart
Organization, project, route, operation, and response-changing controls form the lookup boundary. One project cannot reuse another project’s entry by accident.
See hit, miss, or bypass
Response headers and the request ledger show whether the cache answered, found no match, or stayed out of the way because the request was not eligible.
How a cache decision works
A cache check stays inside the normal authorization and spending flow instead of becoming a hidden shortcut.
Verify the request and cache policy
The gateway checks the API key and route first. Direct model calls, media, embeddings, multimodal chat, and unsupported operations bypass the semantic cache.
Search the correct project boundary
The cache compares the request with recent eligible entries from the same organization, project, route, operation, and request controls.
Return a protected response or call the provider
A qualifying hit decrypts the stored response. A miss calls the normal model path and may save the approved response for later use.
Record what happened
A hit writes a normal request record with its original token counts, a cached outcome, and zero provider cost. It does not disappear from reporting.
Privacy and control are part of the cache
Caching response content is different from the default request path, so the feature is explicit and bounded.
Controls you can use
- Set the similarity threshold and maximum lifetime.
- Use request cache headers to bypass, refresh, or limit age.
- Add project tags for targeted invalidation.
- Clear one project or the whole organization cache with confirmation.
Stored and not stored
- Prompt text is not stored in a cache entry.
- Protected feature vectors and keyed digests support lookup.
- Cached responses are encrypted with authenticated AES-256-GCM.
- A bad or unreadable entry is deleted and treated as a miss.
Related controls
Reuse answers only where it makes sense
Start with caching off, define the safe boundary, and turn it on only for routes whose responses can be reused.
Get started