English EN GitHub
Home Catalogus LiteLLM AI Gateway
Lokale AI & LLM

LiteLLM AI Gateway

LiteLLM is an AI gateway and LLM proxy that unifies access to 100+ Large Language Models (OpenAI, Gemini, Anthropic, Ollama, Azure, Bedrock, HostYourAI) behind a single OpenAI-compatible API format.

ARM64 & AMD64
Docker & Rootless Podman
100% Self-Hosted (Geen Lock-in)
RAM Profiel: Laag
Standaard Poort: Port 4000

Waarom deze configuratie?

Snelle Start (Standalone Docker Compose)

Onderstaand compose-fragment is direct te gebruiken in elke Docker- of Podman-omgeving: 

services:
  litellm:
    container_name: njorddeploy-litellm
    image: "ghcr.io/berriai/litellm:main-stable"
    ports:
      - "4000:4000"
    volumes:
      - "data_root/litellm/config.yaml:/app/config.yaml:ro"
    command:
      - "--config=/app/config.yaml"
    environment:
      - "DATABASE_URL=postgresql://llmproxy:njorddeploy_db_password_123@njorddeploy-litellm-db:5432/litellm"
      - "STORE_MODEL_IN_DB=True"
      - "OPENAI_API_KEY=openai_api_key"
      - "ANTHROPIC_API_KEY=anthropic_api_key"
      - "COHERE_API_KEY=cohere_api_key"
      - "AZURE_API_KEY=azure_api_key"
      - "AZURE_API_BASE=https://YOUR_RESOURCE_NAME.openai.azure.com/"
      - "AZURE_API_VERSION=2023-07-01-preview"
      - "GEMINI_API_KEY=gemini_api_key"
      - "HOSTYOURAI_API_KEY=hostyourai_api_key"
      - "HOSTYOURAI_API_BASE=https://hostyourai.com/api/v1"
      - "AWS_ACCESS_KEY_ID=aws_access_key_id"
      - "AWS_SECRET_ACCESS_KEY=ChangeMeSecurePassword123!"
      - "AWS_REGION_NAME=us-east-1"
    networks:
      - njorddeploy_net
    depends_on:
      - njorddeploy-litellm-db
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.litellm.rule=Host(`litellm.yourdomain.com`)"
      - "traefik.http.routers.litellm.entrypoints=websecure"
      - "traefik.http.routers.litellm.tls.certresolver=default"
      - "traefik.http.services.litellm.loadbalancer.server.port=4000"
    user: "0:0"

  njorddeploy-litellm-db:
    container_name: njorddeploy-litellm-db
    image: postgres:16
    restart: always
    environment:
      - "POSTGRES_DB=litellm"
      - "POSTGRES_USER=llmproxy"
      - "POSTGRES_PASSWORD=njorddeploy_db_password_123"
    volumes:
      - "./data/litellm/postgres_data:/var/lib/postgresql/data"
    networks:
      - njorddeploy_net
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
      interval: 1s
      timeout: 5s
      retries: 10
    user: "0:0"

networks:
  njorddeploy_net:

Start de service direct via de commandline: docker compose up -d

Configuratie & Omgevingsvariabelen

Belangrijkste configuratieparameters en standaardwaarden uit de NjordDeploy component-sjabloon:

Variable Default Value Description
LITELLM_WEB_PORT 4000 The external port for accessing the LiteLLM AI Gateway web interface.
LITELLM_DB_NAME litellm The name of the PostgreSQL database for LiteLLM.
LITELLM_DB_USER llmproxy The username for accessing the PostgreSQL database.
LITELLM_DB_PASSWORD njorddeploy_db_password_123 The password for the PostgreSQL database user.
LITELLM_STORE_MODEL_IN_DB True Set to 'True' to allow adding models to the proxy via the UI, 'False' otherwise.
OPENAI_API_KEY *None* Your OpenAI API Key. Required for using OpenAI models. This key will be available as an environment variable for LiteLLM.
ANTHROPIC_API_KEY *None* Your Anthropic API Key. Required for using Anthropic models. This key will be available as an environment variable for LiteLLM.
COHERE_API_KEY *None* Your Cohere API Key. Required for using Cohere models. This key will be available as an environment variable for LiteLLM.
AZURE_API_KEY *None* Your Azure OpenAI API Key. Required for using Azure OpenAI models. This key will be available as an environment variable for LiteLLM.
AZURE_API_BASE https://YOUR_RESOURCE_NAME.openai.azure.com/ Your Azure OpenAI API Base URL (e.g., https://YOUR_RESOURCE_NAME.openai.azure.com/). Required for Azure OpenAI models.
AZURE_API_VERSION 2023-07-01-preview Your Azure OpenAI API Version (e.g., 2023-07-01-preview). Required for Azure OpenAI models.
GEMINI_API_KEY *None* Your Google Gemini API Key. Required for using Google Gemini models. This key will be available as an environment variable for LiteLLM.
HOSTYOURAI_API_KEY *None* Your HostYourAI API Key for accessing models on HostYourAI.
HOSTYOURAI_API_BASE https://hostyourai.com/api/v1 The API Base URL for your HostYourAI instance or gateway endpoint.
AWS_ACCESS_KEY_ID *None* Your AWS Access Key ID. Required for using AWS Bedrock/Sagemaker models. This key will be available as an environment variable for LiteLLM.
AWS_SECRET_ACCESS_KEY *None* Your AWS Secret Access Key. Required for using AWS Bedrock/Sagemaker models. This key will be available as an environment variable for LiteLLM.
AWS_REGION_NAME us-east-1 Your AWS Region Name (e.g., us-east-1). Required for using AWS Bedrock/Sagemaker models. This key will be available as an environment variable for LiteLLM.
TRAEFIK_HOST litellm.yourdomain.com The hostname for Traefik to route traffic to LiteLLM (e.g., litellm.yourdomain.com).

Ecosysteem & Bedrijfsintegratie

Eenvoudig beheren via NjordDeploy

Installeer, bewaak en update deze service met 1 klik op uw eigen Raspberry Pi of Proxmox server via de NjordDeploy Configurator.

Download NjordDeploy Bekijk GitHub Template Officiële Website Alle 128+ Services