LM Gateway RS

Ultra-lightweight LLM routing gateway written in Rust.
Single binary. No Python. No database. No bloat.

---

What it does

LM Gateway RS sits between your application and your LLM backends, providing a unified OpenAI-compatible interface across any number of local or cloud models. It handles credential management, tier-based routing, and intelligent escalation — so your application stays simple.

---

Why LM Gateway RS?

• Ships as a single static binary — docker run and done
• Zero external runtime dependencies — no Python, no database, no daemon
• Fits on a Raspberry Pi or a $5 VPS
• Can be audited in an afternoon — under 2 000 lines of Rust
• 100% self-hosted — no telemetry, no cloud account, no phone-home

---

Features

• OpenAI-compatible API — drop-in replacement endpoint for any client that speaks /v1/chat/completions
• Tier ladder — define a cheapest→best progression of models, from local Ollama to cloud experts
• Three routing modes:
  • Dispatch — classify intent with a fast local model, forward to the right tier immediately (predictable latency)
  • Escalate — try cheapest tier first; evaluate response quality; escalate only if needed (lowest average cost)
  • Classify — single pre-flight call labels complexity as simple/moderate/complex, then dispatches directly to the appropriate tier (ideal for all-local deployments)
• Ollama-compatible endpoints — GET /api/tags and POST /api/chat let any Ollama client (Home Assistant, Open WebUI, etc.) point at this gateway without modification
• Centralised credential management — backends reference env vars; clients need no API keys
• Live admin UI — dark dashboard at :8081 with real-time traffic log, backend health, and config view
• In-memory traffic log — ring-buffer; zero disk I/O, bounded memory, works on read-only filesystems

---

Quick start

# 1. Copy and edit the example config
cp config.example.toml config.toml
$EDITOR config.toml

# 2. Set secrets via environment variables (never in the config file)
export OPENROUTER_KEY="sk-or-..."

# 3. Run
docker run --rm \
  -v $(pwd)/config.toml:/etc/lm-gateway/config.toml:ro \
  -e OPENROUTER_KEY \
  -p 8080:8080 -p 8081:8081 \
  lm-gateway:latest

Then send a request:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"hint:fast","messages":[{"role":"user","content":"Hello"}]}'

Open the admin UI: http://localhost:8081/

---

Client API (port 8080)

Method	Path	Description
POST	/v1/chat/completions	Route a chat request (OpenAI-compatible)
GET	/v1/models	List available tiers and aliases
GET	/api/tags	List profiles as Ollama "models"
POST	/api/chat	Chat inference — Ollama-compatible; model field = profile name
GET	/healthz	Liveness probe

Use any tier name or alias as the model field:

{
  "model": "hint:fast",
  "messages": [{ "role": "user", "content": "Hello" }]
}

Built-in aliases: hint:fast, hint:cheap, hint:local, hint:cloud, hint:standard, hint:expert

---

Admin API (port 8081)

Method	Path	Description
GET	/	Admin dashboard (web UI)
GET	/admin/health	Gateway health + tier/backend counts
GET	/admin/traffic?limit=N	Recent N requests + aggregate stats
GET	/admin/config	Running config (secrets redacted)
GET	/admin/backends/health	Probe all configured backends

---

Configuration

See config.example.toml for a fully annotated example. A typical setup is under 50 lines.

Key concepts:

Concept	What it is
Backend	A named LLM provider — base URL + optional secret env var
Tier	A named (backend, model) pair in cheapest→best order
Alias	Short name like hint:fast that resolves to a tier
Profile	Routing behaviour: mode, classifier tier, cost ceiling

Environment variable for config path:

LMG_CONFIG=/path/to/config.toml   # default: /etc/lm-gateway/config.toml

---

Use cases

• Homelab / private LLM deployments — route between local Ollama and cloud fallback
• AI agent clusters — serve multiple agents through a single credential-holding gateway
  • Works as-is with ZeroClaw and any OpenAI-compatible agent framework
• Cost optimisation — escalate to expensive cloud models only when local models can't answer
• Development environments — keep all API keys in one place, share across projects

---

Classify mode

Classify mode performs a single, fast non-streaming call to a cheap "classifier" tier before the real request. The classifier responds with a single word — simple, moderate, or complex — and the gateway routes the actual request to:

Label	Tier selected
simple	tiers[0] (cheapest / fastest)
moderate	tiers[n/2] (mid-tier)
complex	tiers[n-1] (most capable)

This gives you predictable, low-latency routing without needing cloud infrastructure. A 1.7b model classifying adds ~50–150 ms on the first call, then the right model answers.

Minimal local config:

[backends.ollama]
provider = "ollama"
base_url = "http://127.0.0.1:11434"

[[tiers]]
name = "local:instant"   # maps to: simple
backend = "ollama"
model = "qwen3:1.7b"

[[tiers]]
name = "local:balanced"  # maps to: moderate
backend = "ollama"
model = "qwen3:8b"

[[tiers]]
name = "local:expert"    # maps to: complex
backend = "ollama"
model = "qwen3:14b"

[profiles.default]
mode       = "classify"
classifier = "local:instant"

---

Ollama compatibility

Two Ollama-format endpoints are always available on the client port:

Endpoint	Purpose
GET /api/tags	Returns configured profiles as Ollama "models"
POST /api/chat	Accepts an Ollama chat request; model name = profile name

Profiles are the public surface. Tiers, aliases, and the classify tier ladder are entirely hidden from Ollama clients. When HA asks "what models do you have?", it sees your profile names — auto, default, or whatever you call them. It never sees the underlying model names.

Home Assistant → GET  lm-gateway-host:8080/api/tags  → {"models": [{"name":"auto:latest"}, ...]}
               → POST lm-gateway-host:8080/api/chat  → model="auto" → classify → right tier

The gateway acts as a drop-in Ollama server. Clients need no awareness of your tier configuration.

---

Building

# Local development (uses platform TLS — schannel on Windows, OpenSSL on Linux)
cargo build

# Production Docker image
# Uses rustls (pure-Rust TLS, no OpenSSL) for a fully static binary.
# Cap RAM for low-memory hosts.
docker build --memory=3g --build-arg CARGO_BUILD_JOBS=2 -t lm-gateway .

The release binary is statically linked and has no runtime dependencies beyond libc.

---

Project layout

src/
├── main.rs          Startup, dual listeners, graceful shutdown
├── config.rs        Config types, TOML loading, validation
├── router.rs        Routing logic (dispatch + escalate + classify modes)
├── traffic.rs       In-memory ring-buffer traffic log
├── error.rs         Unified error type
├── backends/
│   ├── mod.rs       BackendClient enum dispatcher
│   ├── openai.rs    OpenAI / OpenAI-compatible passthrough
│   ├── ollama.rs    Ollama adapter (keyless)
│   └── anthropic.rs Anthropic schema translation
└── api/
    ├── mod.rs       Router assembly
    ├── health.rs    GET /healthz
    ├── client.rs    POST /v1/chat/completions, GET /v1/models,
    │                GET /api/tags, POST /api/chat (Ollama compat)
    ├── admin.rs     Admin endpoints
    └── admin_ui.html Single-page admin dashboard

---

Design principles

• One job — route LLM traffic. Nothing else.
• No magic — config is a single TOML file; behaviour is deterministic and auditable.
• Small surface — no database, no agent, no scheduler. A process that starts fast and uses < 10 MB RAM at idle.
• Transparent — config endpoint redacts secrets; traffic log captures routing decisions.
• Upstream-friendly — clean Rust, idiomatic error handling, documented public API surface.

---

License

AGPL-3.0 — see LICENSE.