CyphrQ
Menu

AI Model Routers Explained: What Gateways See, Store and Route

An AI model router puts one API in front of many models. What Bedrock, Azure, Vertex AI, OpenRouter and LiteLLM see, keep, and where your data goes.

By CyphrQ.ai

Written with AI assistance and human editorial review. Every figure linked to its source.

10 min read

An AI model router, also called a model gateway, is a single API that sits in front of many AI models, so your apps can switch between models from different providers without rewriting code. It also puts one more system in the path of every prompt. Before you adopt one, check four things: what it stores, where it processes data, who holds the keys, and what it logs.

What is an AI model router, in plain terms?

Every AI provider has its own API, its own keys and its own bill. A router gives your apps one front door. The app sends each request to the gateway, the gateway picks a model, passes the request on and returns the answer.

Teams use them for four practical reasons:

  • Switching models without changing application code.
  • One bill and one place to set spending limits.
  • Carrying on when one provider is slow or down.
  • One place to log, filter and control access.

In security terms, a router is a choke point: good for control, and costly if set up carelessly, because every prompt, answer and provider key sits in one place.

What kinds of AI model gateways are there?

There are three broad kinds. The examples below are listed alphabetically within each kind, not ranked. They are described from each vendor's own documentation, checked on 25 October 2026.

Kind Examples Who runs it Who handles your prompts
Cloud AI platform Amazon Bedrock, Azure AI Foundry (now Microsoft Foundry), Google Vertex AI Your cloud provider, inside your cloud account The cloud provider, under your existing cloud contract
Hosted router OpenRouter A separate company between you and the model providers The router, plus whichever provider serves each request
Open-source gateway LiteLLM You, on your own servers You, plus the providers you connect

Cloud AI platforms host models from several providers inside one cloud. AWS says model providers "don't have any access" to the Bedrock accounts that run their models, so they "don't have access to Amazon Bedrock logs or to customer prompts and completions" (AWS). Microsoft says prompts and completions for "Models sold by Azure" are "NOT available to OpenAI or other providers" of those models (Microsoft Learn). Note the scope: that statement covers models Microsoft sells, so check the terms of any other model you deploy. Google says it "won't use your data to train or fine-tune any AI/ML models without your prior permission or instruction" (Google Cloud).

Hosted routers are a separate company in the data path. OpenRouter's documentation says "your prompts are not retained unless you specifically opt in to prompt logging" (OpenRouter). The provider that actually runs the model has its own policy, which is why OpenRouter offers a setting to send requests only to providers with zero data retention.

Open-source gateways run on your own infrastructure, so the extra party is you. LiteLLM's proxy issues "virtual keys" to your apps and needs a Postgres database and a master key, which its docs call "your Proxy Admin key" (LiteLLM). You avoid a new vendor, and you take on the job of securing a server that holds every provider key you own.

Why does an AI model router matter for security and compliance?

Does a model gateway add another party that sees your data?

Usually, yes. Every hop that can read a prompt is a processor to list in your records, contracts and privacy notice. A hosted router adds a company. A cloud platform keeps it to the provider you already have, though each feature you switch on can add storage. A self-hosted gateway adds a server you must patch and monitor.

For what should stay out of prompts whatever the route, see the AI-at-work privacy checklist.

What does "zero data retention" really mean on each platform?

It means different things on each platform, and every vendor lists exceptions.

  • Amazon Bedrock now has a data retention mode. With none, "No request or response data is written to durable storage by AWS or shared with the model provider." Some models require retention for abuse prevention. If you've set none and call one of them, Bedrock blocks the request and returns an error (AWS).
  • Azure AI Foundry runs abuse monitoring. When it flags something, a sample of prompts and completions may be stored for human review. Approved customers can apply for modified abuse monitoring, which turns off that storage and human review, though automated review may still run (Microsoft Learn).
  • Google Vertex AI caches data for "no later than 24 hours", and you can turn caching off. Google also lists features where "Zero data retention may not be possible", including Grounding with Google Search, where it says "There is no way to disable the storage" (Google Cloud).
  • OpenRouter lets you require zero data retention for your whole account or for a single request. It treats in-memory prompt caching as not "retaining" data, and its setting "does not apply to plugins and tools you choose to enable", such as web search (OpenRouter).

Where does an AI model router process your data?

Not always where you'd assume. On Azure, a standard deployment processes prompts within the geography you chose. A "Global" deployment may process them "in any geography where the relevant model sold by Azure is deployed". A "DataZone" deployment keeps processing within a zone such as the EU or the United States. Stored data stays in your chosen geography in all three cases (Microsoft Learn).

On Bedrock, the retention setting applies per Region and "does not propagate to other Regions". When cross-region inference is on, retained data is stored in the Region where the request was processed (AWS). A residency review has to cover every Region a request can reach, not only the one in your config file.

Why are API keys the weakest point of a model gateway?

A gateway holds the keys to every model you pay for. Anyone who steals them gets free AI on your bill, and sometimes your logs too. Both cases in the deeper section began with exposed or stolen credentials, not a flaw in a model. For how secrets leak from developer platforms, see the Hugging Face Spaces case.

What should a model gateway log?

Logs are evidence, and prompt logs are also copies of your data. Record metadata (who, which model, tokens, cost, errors) everywhere. Record prompt content only where you have a reason, with a retention period and access controls to match. Remember that a filter in the gateway is not a fix for prompt injection: it's one layer.

How do cost controls and fallback affect risk?

A stolen key turns straight into a bill. Budgets per key, rate limits and alerts cap the loss. LiteLLM, for example, sets max_budget, budget_duration, tpm_limit and rpm_limit on each virtual key (LiteLLM).

Fallback needs the same scrutiny. If your gateway sends traffic to a second provider when the first fails, that backup must meet the same retention and region rules. Otherwise an outage quietly moves your data somewhere you never approved.

What should you ask before adopting an AI model router?

Six questions, for a vendor or your own platform team:

  1. Which companies can read a prompt on its way to the model, and are they all in our contracts?
  2. Is zero data retention available for the models we need, and which features break it?
  3. In which countries or Regions can a request be processed, including during fallback?
  4. Who can create, rotate and revoke keys, and does every app get its own?
  5. What is logged, for how long, and who can read prompt content?
  6. What spending cap stops a stolen key, and who gets the alert?

If developers are already signing up for hosted routers on personal cards, start with the shadow AI guide. If the gateway feeds agents that act on your systems, read AI agents at work for permission design. More explainers like this live under AI Tools & Trends.

Going deeper: for security and IT teams

Trust boundary. A gateway is a TLS-terminating proxy. It must read the plaintext prompt to route it, so it belongs in the same data classification as the most sensitive prompt it will carry. AWS requires TLS 1.2 for Bedrock, recommends TLS 1.3, and offers FIPS 140-3 endpoints for teams that need validated modules. It also recommends CloudTrail for API activity (AWS).

Threat model. The attacker wants a credential that can call models: a cloud access key, a provider API key or the gateway's admin key. The payoff is free inference, resold access and, if content logging is on, readable prompt history.

In practice: LLMjacking. On 6 May 2024, Sysdig's Threat Research Team reported attackers using cloud credentials stolen from a system running a Laravel version vulnerable to CVE-2021-3129. Their script checked keys for ten AI services, including Bedrock, Azure, Vertex AI and OpenAI. To test access without running a prompt, they called InvokeModel with an invalid parameter: a ValidationException confirmed access. They also queried GetModelInvocationLoggingConfiguration, which Sysdig says helps attackers avoid detection. Sysdig estimated the worst-case cost to the victim "can be over $46,000 per day", assuming Claude 2.x quota limits reached in four Regions.

In practice: Storm-2139. On 27 February 2025, Microsoft's Digital Crimes Unit said the group used "exposed customer credentials scraped from public sources" to reach generative AI services including Azure OpenAI Service. The group "altered the capabilities of these services and resold access to other malicious actors". Microsoft filed suit in the Eastern District of Virginia in December 2024 and seized a website used in the operation in January 2025.

Controls, as documented.

  • Bedrock: set data_retention_mode to none per Region at account or project level. Enforce it with an SCP that denies bedrock-mantle:PutAccountDataRetention, CreateProject and UpdateProject unless the bedrock-mantle:DataRetentionMode condition key equals none. The control plane uses bedrock:PutAccountDataRetention and bedrock:DataRetentionMode. AWS notes there was no console setting at launch, so this is configured through the API or SDK (AWS).
  • Azure: once modified abuse monitoring is approved, confirm it. The resource's JSON view, or az cognitiveservices account show, should list a ContentLogging capability with the value false. The property doesn't appear at all otherwise (Microsoft Learn).
  • Vertex AI: an admin with roles/aiplatform.admin can set disableCache to true on the project's cacheConfig. Keep request-response logging off (the default), and set store = false on Interactions API requests, where the default is true (Google Cloud).
  • OpenRouter: turn on zero data retention at account level, rather than relying on "provider": { "zdr": true } in each request. The request flag can add enforcement but can't remove it. The list of compliant endpoints is published at /api/v1/endpoints/zdr, and endpoints with unverified policies are treated as retaining and training (OpenRouter).
  • LiteLLM: keep the master key for administrators only. Give each app its own virtual key with a models allowlist and a budget. Use key regeneration with a grace_period for rotation; the docs mark key rotation as an Enterprise feature (LiteLLM).

Detection and testing. CloudTrail doesn't record prompts or responses, so content visibility needs model invocation logging (Sysdig). Alert on bursts of InvokeModel calls ending in ValidationException, on GetModelInvocationLoggingConfiguration from unexpected principals, and on per-key spend jumps (LiteLLM exposes spend through /key/info). Test the cap: issue a key with a small budget in a test project and confirm the gateway refuses calls once it's spent.

Key takeaways

  • An AI model router is one API in front of many models. It's a control point and a single point of exposure.
  • Cloud platforms, hosted routers and open-source gateways differ mainly in who else handles your prompts.
  • "Zero data retention" exists on each platform, with listed exceptions. Read them for the models and features you'll use.
  • The documented attacks started with stolen keys. Give each app its own key, set budgets and alert on recon activity.

Sources