Process the request
The gateway receives the input, applies authorized routing, sends it to the chosen provider, and streams or returns the provider response.
Clear data boundaries
Know where your AI request data goes
An AI gateway must read a request to send it to a model. The important questions are what it stores, what appears in reports, which optional features keep content, and which provider receives the request.
- Normal request path
- Prompt and response are request-local
- Usage trail
- Operational metadata, not content
- Cache storage
- Explicit and encrypted
The normal gateway path does not become a prompt archive
LLMRegistry processes prompt and completion content so it can deliver the request and response. Normal usage reporting records what happened without storing that text as part of the ledger.
Record the outcome
Operational records can include the request ID, route, model, provider, tokens, cost, timing, cache state, and status—not the prompt or answer text.
Honor the provider boundary
The selected model provider receives the content needed for inference. Its own privacy terms and retention rules still apply.
Optional features can create different storage needs
A feature that reuses an answer cannot work without keeping protected data for that purpose.
Semantic cache starts off
Both the organization policy and the route must allow caching. A normal route does not begin saving reusable answers by accident.
Prompt text stays out of cache entries
The cache uses protected feature vectors and keyed digests for matching instead of saving the original prompt text.
Reusable responses are encrypted
Eligible response content is stored with authenticated AES-256-GCM encryption and a limited lifetime set by policy.
Teams can clear the cache
Authorized users can invalidate one project or clear the organization cache when saved responses should no longer be reused.
Secrets have a separate protection path
Application keys and provider credentials should not appear in reports or normal administration screens.
LLMRegistry API keys
- The full secret is shown once when it is created or rotated.
- Only a SHA-256 digest and a short prefix are stored.
- Lists, audit events, exports, and logs cannot recover the secret.
- Revocation can stop future use.
Provider credentials
- BYOK credentials are encrypted with AES-256-GCM.
- Plaintext exists only while validating, encrypting, or dispatching a request.
- Telemetry receives a safe hint, not the secret.
- Rotation may keep encrypted grace material for 15 minutes.
Make the provider choice part of your privacy review
Gateway controls cannot replace the terms of the company that runs the chosen model.
Build with a data path you can explain
Choose providers carefully, keep optional content storage explicit, and use metadata for operations without turning it into a prompt log.
Get started