Skip to main content

09b · liter-llm — The Model Gateway

liter-llm is the multi-provider gateway the pack routes model traffic through. It is a vendored Rust workspace at tools/liter-llm, installed as a single binary at ~/.local/bin/liter-llm.

This reference preserves the gateway contract recorded on 2026-07-30. The pinned tools/liter-llm source and configured installation own the current CLI, schemas and model inventory. This page is not fresh installed-runtime evidence; verify changed gateway source at the final local integration boundary.

It is a server, not a completion CLI​

This is the single most important fact about it, and the one earlier docs got wrong.

$ liter-llm --help
LiterLLM proxy server and MCP tool server

Commands:
api Start the OpenAI-compatible proxy server
mcp Start the MCP server exposing LLM operations as tools
help Print this message or the help of the given subcommand(s)

Two subcommands. That is the entire CLI.

There is no liter-llm complete, no mcp-call, no list_models subcommand. Earlier revisions of several skills called liter-llm complete --model medium and paired it with 2>/dev/null || echo "{}". Because the guard only checked that the binary existed, the call failed, the error was swallowed, and the caller silently produced empty results — for the entire life of those scripts.

Shell callers should speak OpenAI REST to the gateway. The pack provides a helper that reports failures instead of swallowing them:

. "${CLAUDE_PLUGIN_ROOT}/shared/scripts/lib/kbd-model-resolve.sh"
out="$(kbd_complete "$(kbd_resolve_role critic)" "$SYSTEM" "$USER" 2048)" || {
echo "model call failed (see message above)" >&2
}

kbd_complete captures the HTTP status explicitly (curl -s without -f exits 0 on 4xx/5xx), distinguishes 401/403 from other failures, and fails loudly on a non-JSON body or an empty message — because a silent empty completion reads as "the model had nothing to say".

liter-llm api — the OpenAI-compatible proxy​

$ liter-llm api --help
Start the OpenAI-compatible proxy server

Options:
-c, --config <CONFIG> Path to config file (default: auto-discover liter-llm-proxy.toml)
--host <HOST> Override bind host [default: 0.0.0.0]
-p, --port <PORT> Override bind port
--tokio-worker-threads <N>
--tokio-max-blocking-threads <N>

The gateway listens on loopback only​

liter-llm's own default bind host is 0.0.0.0, every interface. Started that way, the gateway is reachable from the whole local network, and the only thing between another machine and every provider key in the config is the master key. The proxy config file has no host setting, so the address can only be chosen on the command line.

The pack's service wrapper, shared/scripts/liter-llm-api-launch.sh, therefore passes the host explicitly and defaults it to loopback:

exec "$LITER_LLM_BIN" api --host "${LITER_LLM_HOST:-127.0.0.1}" \
--config "$HOME/.config/liter-llm/liter-llm-proxy.toml"

Every consumer in the pack (kbd_complete, the adversarial-review scripts, the protocol-eval grader) runs on the same machine and reaches the gateway through localhost, so loopback is all they need.

When to change it. Set LITER_LLM_HOST=0.0.0.0 in the service's environment only when another machine really has to use this gateway, and only on a network you trust. Prefer an SSH tunnel or a private overlay network to a wide bind. The variable must be in the LaunchAgent's EnvironmentVariables: launchd does not read a shell profile.

When starting it by hand, pass the host yourself. A bare liter-llm api --config … uses the upstream default and binds every interface:

liter-llm api --host 127.0.0.1 --config "$HOME/.config/liter-llm/liter-llm-proxy.toml"

Check what is listening (macOS and Linux):

lsof -nP -iTCP:4000 -sTCP:LISTEN

127.0.0.1:4000 is loopback. *:4000 means every interface. Use the port from your proxy config if it is not 4000. A service that was already running keeps its old bind until it is restarted with the new wrapper (bash scripts/install-mcp-services.sh --restart).

Routes​

The recorded proxy contract places /v1/* routes behind Bearer authentication. Health routes remain separate; configured reachability is not an authorized completion.

GroupRoutes
Chat & completion/v1/chat/completions, /v1/responses, /v1/responses/{id}, /v1/responses/{id}/cancel
Models/v1/models
Embeddings & rerank/v1/embeddings, /v1/rerank
Audio/v1/audio/speech, /v1/audio/transcriptions
Images/v1/images/generations
Documents/v1/ocr
Moderation/v1/moderations
Search/v1/search
Files/v1/files, /v1/files/{id}/content
Batches/v1/batches, /v1/batches/{id}, /v1/batches/{id}/cancel
Realtime/v1/realtime
Health (unauthenticated)/health, /healthz, /readyz, /health/liveness, /health/readiness
Spec/openapi.json

OpenAI-compatible clients can target the configured base URL and credentials. Check the specific route and payload supported by both client and gateway; an OpenAI-shaped API is not a guarantee of every SDK feature.

liter-llm mcp — the MCP tool server​

$ liter-llm mcp --help
Start the MCP server exposing LLM operations as tools

Options:
-c, --config <CONFIG> Path to config file
--transport <TRANSPORT> Transport mode: stdio or http [default: stdio]
--host <HOST> Host for HTTP transport [default: 127.0.0.1]
--port <PORT> Port for HTTP transport [default: 3001]

The 22 tools​

Enumerated live from tools/list on 2026-07-30:

cancel_batch cancel_response chat create_batch create_file
create_response delete_file embed file_content generate_image
list_batches list_files list_models moderate ocr
rerank retrieve_batch retrieve_file retrieve_response
search speech transcribe

The chat tool is chat. Earlier documentation in this repository described four tools that do not exist:

Documented beforeReality
completeDoes not exist. Use chat.
healthDoes not exist. Probe GET /v1/models.
streamDoes not exist. Streaming is a chat parameter.
get_cost, create_api_key, set_rate_limit, set_budget, cache_*Not tools. These are config sections ([budget], [rate_limit], [cache], [[keys]]).

list_models is a genuine MCP tool — but not a CLI subcommand. From a shell, use GET /v1/models.

Configuration​

liter-llm-proxy.toml. Every struct uses serde(deny_unknown_fields), so a typo is a hard parse error rather than a silently ignored key.

Sections​

SectionPurpose
[server]bind host/port
[general]master_key, default_timeout_secs, max_retries, cost tracking, tracing
[[models]]named model → provider + endpoint + key
[[aliases]]pattern-matched routing (pattern, api_key, base_url)
[[keys]]virtual API keys with per-key model allowlists and RPM
[rate_limit]rpm, tpm
[budget]global_limit, per-model limits, enforcement mode
[cache]response caching — max_entries, ttl_seconds, backend
[files]file storage
[health]probe interval and probe model
[cooldown]failure cooldown duration
[mcp]stdio_trust_local and stdio key binding
[security]outbound_policy, outbound_allowlist

[[models]] — the exact schema​

Fields are exactly these. Anything else is a parse error:

[[models]]
name = "kbd-judge" # what callers send as "model"
provider_model = "openai/gpt-5.6-sol" # registry prefix + model id
api_key = "${OPENAI_API_KEY}" # literal or ${VAR}
base_url = "http://localhost:8181/v1" # optional endpoint override
timeout_secs = 60 # optional
fallbacks = ["kbd-critic"] # optional, tried in order

fallbacks declares alternate routes. Record the actual provider/model returned; a fallback does not preserve distinct-model review or budget limits automatically.

A minimal working config​

Three things are mandatory and were all missing from the config this repo shipped before 2026-07-30 — the result being a gateway that could not serve a single request:

[general]
# REQUIRED. Every /v1/* route is behind an unconditional Bearer check, so a config with
# no master_key and no [[keys]] answers 401 to EVERYTHING, /v1/models included.
master_key = "${LITER_LLM_MASTER_KEY}"

[security]
# REQUIRED for any localhost base_url. The default deny_private REFUSES loopback.
outbound_policy = "off"

[[models]]
name = "kbd-judge"
provider_model = "openai/gpt-5.6-sol"
api_key = "sk-proxy-local"
base_url = "http://localhost:8181/v1"

Config discovery — pass --config​

ProxyConfig::discover() walks the current directory upward looking for liter-llm-proxy.toml. It never searches $HOME. Always pass an absolute path:

liter-llm api --config ~/.config/liter-llm/liter-llm-proxy.toml
liter-llm mcp --transport stdio --config ~/.config/liter-llm/liter-llm-proxy.toml

Without it, liter-llm mcp does not load zero models — it fails to start, because [mcp] stdio_trust_local lives in the config it was never given:

$ liter-llm mcp --transport stdio
Error: stdio MCP transport requires authentication configuration; set either
`mcp.stdio_key_id` or `mcp.stdio_trust_local = true`

Environment interpolation​

${VAR} only. There is no ${VAR:-default}, and an unset variable expands to the empty string rather than erroring — which surfaces much later as an unexplained 401. The pack's tooling verifies every referenced variable is set:

bash skills/process/liter-llm-bridge/scripts/configure-models.sh check
bash scripts/check-model-config.sh

Providers​

The pinned registry (tools/liter-llm/schemas/providers.json) owns available providers. Model references are prefix/model-id:

provider_model = "moonshot/kimi-k2.5"
provider_model = "minimax/MiniMax-M2.5"
provider_model = "dashscope/qwen3-coder-plus"
provider_model = "zai/glm-4.7"
provider_model = "openai/gpt-5.6-sol"

Any OpenAI-compatible endpoint that is not in the registry is reachable by keeping a routable prefix and overriding base_url — which is also how subscription coding plans are reached. See 09a · Adversarial Review.

What we expose, and why​

The pack does not use all 22 tools. What it actually depends on:

CapabilityUsed by
chat / /v1/chat/completionsadversarial-review judge, pmpo-evolver extraction & dreaming, learn-grade
/v1/modelsgateway reachability probe, configure-models.sh verify
[[models]] + base_urlpointing a logical role at any provider without touching a script
fallbacksrole resilience
[security] outbound_policypermitting the local openai-proxy on loopback
[[keys]], [rate_limit], [budget], [cache]available, not currently wired by the pack

The last row matters: virtual keys, rate limits, budgets, and response caching are real features of the gateway you can enable yourself in liter-llm-proxy.toml. No pack skill configures them today.

Managing it​

S="${CLAUDE_PLUGIN_ROOT}/skills/process/liter-llm-bridge/scripts/configure-models.sh"

bash "$S" check # state; changes nothing
bash "$S" repair # add ONLY missing mandatory pieces (merges, never clobbers)
bash "$S" add-provider kimi # local-proxy|kimi|minimax|qwen|glm|glm-coding|kimi-coding
bash "$S" verify # GET /v1/models must be 200, then one real completion per role
bash "$S" migrate # retire the legacy config.toml

There is no liter-llm config, doctor, or validate subcommand. The only way to validate a config is to start the server and watch it fail — which is exactly what verify automates.

Relationship to a subscription proxy​

A separately operated subscription proxy may expose an OpenAI-compatible endpoint and use locally owned session credentials. The pack does not make those credentials portable authority or guarantee a fixed model list. Configure an authorized base_url explicitly and confirm the actual selected model and reasoning controls.

liter-llm api is the multi-provider gateway and authenticates its /v1/* routes. The pack's native API service template is macOS-only; binary support on another platform does not provide that service definition. See Service operations for owners and install paths.

See also​