AI and LLMsWritten by GLWS16 min readUbuntu 24.04 LTS / AlmaLinux 9
Commands run on your server unless a section says otherwise. Replace example usernames, addresses and paths with your own values. Keep a working session open when changing remote access.

Why a gateway

If several applications call AI providers directly, each one holds a real provider key, and you cannot see or limit their spending in one place. A gateway fixes that. LiteLLM is a proxy that exposes one OpenAI-compatible API in front of many providers and local models. Applications get their own virtual keys with budgets and model limits. Only the gateway holds the real provider keys, so a leaked app key can be revoked without touching anything else.

You need Docker (see the Docker guide), about 2 GB of memory, and a domain if you publish it.

On the VPS
sudo docker compose version
free -h

Download the quick-start files

This complete example runs the proxy, PostgreSQL for virtual keys and usage, and a private Ollama backend. The local model lets you verify the gateway before buying hosted API access. Allow at least 4 GB RAM for this small model and the stack; larger models need more.

Generate each secret once. Keep .env and the database backup protected and retain the same salt key when restoring. Pin tested image versions before production updates.

Both systems: project and private secrets
mkdir -p ~/litellm && cd ~/litellm
(umask 077; printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\nPOSTGRES_PASSWORD=%s\nUI_USERNAME=guideadmin\nUI_PASSWORD=%s\n' "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" "$(openssl rand -hex 24)" "$(openssl rand -hex 24)" > .env)
sed 's/=.*/=(hidden)/' .env
Both systems: write compose.yaml
cat > compose.yaml <<'EOF'
services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: llmproxy
      POSTGRES_DB: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - database:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U llmproxy -d litellm"]
      interval: 5s
      timeout: 3s
      retries: 20
    restart: unless-stopped
  ollama:
    image: ollama/ollama:latest
    environment:
      OLLAMA_KEEP_ALIVE: 2m
      OLLAMA_NUM_PARALLEL: 1
    volumes:
      - models:/root/.ollama
    restart: unless-stopped
  proxy:
    image: ghcr.io/berriai/litellm:main-stable
    command: ["--config", "/app/config.yaml", "--port", "4000"]
    ports:
      - "127.0.0.1:4000:4000"
    env_file: .env
    environment:
      DATABASE_URL: postgresql://llmproxy:${POSTGRES_PASSWORD}@db:5432/litellm
      STORE_MODEL_IN_DB: "True"
    volumes:
      - ./config.yaml:/app/config.yaml:ro,Z
    depends_on:
      db:
        condition: service_healthy
    restart: unless-stopped
volumes:
  database:
  models:
EOF
Both systems: write config.yaml
cat > config.yaml <<'EOF'
model_list:
  - model_name: my-chat-model
    litellm_params:
      model: ollama/llama3.2:1b
      api_base: http://ollama:11434
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
EOF
chmod 600 config.yaml

Keep it private, then start it

Only the proxy port is published, on loopback. Database and Ollama ports stay private. Start the stack, download the configured model, then wait for readiness; the first startup creates database tables.

Both systems
sudo docker compose config --quiet
sudo docker compose up -d
sudo docker compose exec ollama ollama pull llama3.2:1b
sudo docker compose logs --tail 30 proxy
sudo ss -lntp | grep 4000
Check readiness with the master key; never share its value
set -a; . ./.env; set +a
curl --fail-with-body -sS http://127.0.0.1:4000/health/readiness -H "Authorization: Bearer $LITELLM_MASTER_KEY"
If the port line looks different in your version, edit it by hand so the left side starts with 127.0.0.1:. Do not leave the database port published either.

Reach the admin interface

Keep the SSH tunnel open, visit http://localhost:4000/ui and sign in using UI_USERNAME and UI_PASSWORD from your private .env file. Use the administrator account for key management; applications receive limited virtual keys.

On your computer
ssh -N -L 4000:127.0.0.1:4000 deploy@YOUR_SERVER

Add a model

The example already mounts config.yaml and defines my-chat-model. To use a hosted provider instead, replace the litellm_params below with that provider’s documented model name, add its API key to .env and recreate proxy. Keep provider credentials out of application code.

A dollar budget limits recorded usage only when the provider/model has accurate pricing data; it is not a hard spending guarantee. Local Ollama models have no provider token charge. Add request/token rate limits and provider-side account limits as appropriate.

Config file form (mount it into the container and point the proxy at it)
model_list:
  - model_name: my-chat-model
    litellm_params:
      model: PROVIDER/MODEL_NAME
      api_key: os.environ/PROVIDER_API_KEY
Apply edited configuration
sudo docker compose config --quiet
sudo docker compose up -d --force-recreate proxy
sudo docker compose logs --tail 30 proxy

Issue a limited virtual key

Give each application its own key. Create one through the API with the master key, and set limits: the models it may use, a spending cap and a lifetime. The field names are listed in the project's virtual keys documentation, so check them against your version. The response contains the key, which is shown once, so store it with the application that will use it.

Both systems
set -a; . ./.env; set +a
curl -s http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models":["my-chat-model"],"max_budget":5,"duration":"30d","key_alias":"my-app"}'

Send a test request

Call the gateway the same way you would call any OpenAI-compatible API, but with your own base URL and the virtual key. Any client library that lets you set a base URL works.

Both systems (replace the key and model name)
curl --fail-with-body -sS http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-YOUR-VIRTUAL-KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"my-chat-model","messages":[{"role":"user","content":"Say hello in five words."}]}'

Publish it with HTTPS

Applications on other servers need an HTTPS address. Follow the reverse proxy guide to proxy a name such as ai.example.com to 127.0.0.1:4000, with long read timeouts and proxy buffering off so streamed answers work. Never give applications the master key, and keep the admin interface path off the public site, for example by only allowing /v1 and /chat paths through the proxy.

Nginx location (inside the HTTPS server block)
location /v1/ {
    proxy_pass http://127.0.0.1:4000;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header Connection "";
    proxy_buffering off;
    proxy_read_timeout 600s;
}

Operate it

Keep .env, config.yaml and a protected PostgreSQL dump off this VPS. A database-only backup cannot recover the original salt and provider credentials. Verify a restore with the same image version and test that a saved virtual key still works.

Revoke a leaked key through the admin interface or /key/delete using the master key; then confirm requests with that key fail. Read release notes and back up before updating.

Consistent database backup without exposing the password
cd ~/litellm
set -o pipefail
umask 077
sudo docker compose stop proxy
sudo docker compose exec -T db pg_dump --clean --if-exists -U llmproxy litellm | gzip > litellm-db-$(date +%F).sql.gz
sudo docker compose start proxy
gzip -t litellm-db-$(date +%F).sql.gz
Restore after confirming the database and backup paths
cd ~/litellm
sudo docker compose stop proxy
gunzip -c litellm-db-YYYY-MM-DD.sql.gz | sudo docker compose exec -T db psql -v ON_ERROR_STOP=1 -U llmproxy -d litellm
sudo docker compose start proxy
# Recheck /health/readiness and a request using a saved virtual key.
Update
sudo docker compose pull
sudo docker compose up -d
sudo docker compose logs --tail 30 proxy
RECORDED LAB OUTPUT · AlmaLinux 9
Disallowed model rejected: 403
Restored virtual key still authorizes actual model inference
Revoked key rejected: 401
Recorded real local-model request, model restriction, database restore and key revocation. The upstream was local Ollama; hosted-provider billing and quotas were not exercised.
The database user and name in the dump command come from the Compose file. Check them with grep POSTGRES docker-compose.quickstart.yml and adjust.

Official documentation

These upstream references provide more detail on the commands and workflows in this guide.

KEEP LEARNING

Related GLWS guides

Browse all guides