Where does LiteLLM run?

On your own infrastructure. LiteLLM and the Privatemode proxy run next to each other, for example on a server of your platform team, and the request to Privatemode leaves that infrastructure encrypted. That is why the example keeps the proxy inside the Compose network. A LiteLLM gateway that someone else operates, such as a hosted service, would see your prompts in plaintext before they reach the proxy, so run both yourself.

Is my data encrypted before it leaves my infrastructure?

Yes, as soon as it reaches the Privatemode proxy. The proxy encrypts every request before it leaves your infrastructure and performs remote attestation to verify the integrity of the Privatemode service. The hops from your clients to LiteLLM and from LiteLLM to the proxy stay inside your own network, so protect them the way you protect the rest of your gateway traffic.

Does Privatemode store or retain data from requests that pass through LiteLLM?

No. Privatemode never learns from your data. All requests are processed using confidential computing, and prompts and responses are not saved once a request completes. Only operational metadata such as token usage, timestamps, and request status is kept for billing and monitoring. LiteLLM can log prompts and responses itself, for example to a database or an observability tool. Those logs live wherever you configure them and are not covered by Privatemode, so check your logging and callback settings for models that carry sensitive data. LiteLLM can keep prompts and responses out of its logs with `turn_off_message_logging`.

Can I combine Privatemode with other providers?

Yes, but be careful with fallbacks and load balancing. LiteLLM can send a request to another deployment when one fails, and a fallback to a conventional provider would process the prompt in plaintext on that provider's servers. Give sensitive workloads a `model_name` that only maps to Privatemode and leave it out of your [fallback rules](https://docs.litellm.ai/docs/proxy/reliability).

What if LiteLLM does not run in Docker?

Start the Privatemode proxy with `docker run -p 127.0.0.1:8080:8080 ghcr.io/edgelesssys/privatemode/privatemode-proxy:latest --apiKey ` and set `api_base: http://localhost:8080/v1` in your LiteLLM config. If LiteLLM runs in a container, put the proxy on the same Docker network, as in the example above.

Which Privatemode models can I add?

Any model from the [model overview](https://privatemode.ai/models). Chat models work as `openai/ ` entries, as shown above. The same proxy also serves embeddings and speech-to-text, so you can add those as further entries in `model_list`.

Integrating Privatemode AI with LiteLLM

Q: Does Privatemode require changes to my LiteLLM setup?

No. Privatemode is one more entry in `model_list`. Your clients, virtual keys, budgets, and the rest of your configuration stay as they are.

LiteLLM is an open-source AI gateway: one OpenAI-compatible endpoint in front of 100+ model providers, with virtual keys, budgets, and logging. Adding Privatemode as one of those models sends the requests that carry sensitive data to open-weight models in an environment whose memory stays encrypted, while every client keeps talking to the same gateway.

Introduction

Keep your gateway, change what happens to the prompts

Encryption at rest, in transit, and in use

The challenge

Every team's prompts meet at the gateway

Platform teams run LiteLLM so that every team reaches models through one endpoint with shared keys, budgets, and logging. That puts customer data, source code, and internal documents from many teams on a single path. With a conventional provider behind it, all of that is processed in plaintext on the provider's servers.

The solution

One more model in your model list

Privatemode exposes an OpenAI-compatible API, so LiteLLM treats it like any other openai/ model. Run the Privatemode proxy next to the gateway and point api_base at it. Every request is encrypted before it leaves your infrastructure and processed in a hardware-isolated environment whose memory stays encrypted. By design, neither the infrastructure provider nor Privatemode can read it. This is verifiable through remote attestation and open-source code.

Benefits

Why use Privatemode AI with LiteLLM?

Just another model in your gateway

End-to-end confidentiality

Learn how

Free tier available

How to get started

How to add Privatemode to LiteLLM

LiteLLM talks to the Privatemode proxy like to any OpenAI-compatible endpoint. The proxy verifies the Privatemode service through remote attestation and encrypts every request before it leaves your infrastructure. The example below runs both with Docker Compose.

Get your API key

If you don't have a Privatemode API key yet, you can generate one for free here.

Add Privatemode to your LiteLLM config

Save this as litellm_config.yaml. The openai/ prefix tells LiteLLM to treat the model as an OpenAI-compatible endpoint, and api_base has to end in /v1. The API key is a placeholder because the Privatemode proxy handles authentication. model_name is the name your applications use.

Run the proxy and LiteLLM

Save this as docker-compose.yml. Put PRIVATEMODE_API_KEY and a long random LITELLM_MASTER_KEY that starts with sk- into a .env file next to it, then start both with docker compose up. The Privatemode proxy stays inside the Compose network, so only LiteLLM can reach it.

Send a test request

Call the gateway with the model name from your config. From here, create virtual keys for your teams, or point any OpenAI-compatible client at http://localhost:4000.