Skip to main content

09a · Adversarial Review & Model Configuration

The single most important quality gate in the pack, and the one that is easiest to have running in a state where it does nothing.

Adversarial review dispatches a fresh-context LLM judge with an explicit mandate to find problems in a KBD artifact or change diff. The rule it exists to enforce is one sentence long:

The model that produced the work is never the model that reviews it.

That is not a stylistic preference. A critic that shares the producer's blind spots cannot see the producer's blind spots. If Claude writes a plan and Claude reviews it, the review will systematically miss exactly the class of error Claude is prone to — and it will return PASS, which reads like evidence of quality.

The failure this page exists to prevent​

Every one of the first eight stored reviews in this repository recorded:

{
"isolation_mode": "harness-native",
"judge_model": "harness-subagent (claude, parent-session family)",
"verdict": "PASS"
}

Claude reviewing Claude, eight times, all passing. The pipeline ran, wrote artifacts, and reported success. It had never once reached a second model.

It failed silently for five compounding reasons — worth knowing, because each is a trap you can re-enter:

  1. Three parallel config surfaces, none authoritative. project.json → model_policy was declared as policy_source by five skills and read by none.
  2. A schema mismatch. The file the code read used a flat [aliases] table; liter-llm's real schema is an [[aliases]] array. The parse silently yielded nothing and the dispatcher sent the literal string "frontier" as a model id.
  3. The collision guard never fired. Every packet carried producer_model: "unknown", so candidate != producer passed trivially.
  4. isolation_mode was a hardcoded literal. It said "liter-llm" regardless of what answered, so a self-grade was indistinguishable from a real cross-model review.
  5. The gateway config could not serve a request — see Mandatory sections below.

Everything on this page is designed so that when it is broken, it says so.

Architecture​

The judge receives only the review packet — the diff or artifact, its acceptance criteria or phase goals, the file tree, and blocking constraints. It never sees the producing session's conversation. Isolation is structural, not honour-system.

Two files, one gateway​

Configuration is split so that neither file is a script, and neither requires editing one:

FileOwns
~/.prometheus/kbd/models.tomlrole → model name (KBD owns)
~/.config/liter-llm/liter-llm-proxy.tomlname → provider + base_url + ${KEY} (liter-llm owns)

Adding a provider edits liter-llm's file. Repointing a role edits models.toml. This split is the whole point: a previous session "fixed" model routing by editing pack scripts inside a plugin cache — edits the next install destroys and git never sees.

# ~/.prometheus/kbd/models.toml
[gateway]
candidates = ["http://localhost:8181/v1", "http://localhost:4000/v1"]

[roles]
# The producer. Intentionally NOT a [[models]] entry — it is the harness itself and is
# never dispatched. It is named so the collision check has something concrete to compare
# against instead of the literal "unknown".
generator = "kbd-frontier"
critic = "kbd-critic" # MUST differ from generator
judge = "kbd-judge"

Resolution order​

Implemented once, in shared/scripts/lib/kbd-model-resolve.sh. AWS-CLI convention, highest wins:

explicit argument
> PROMETHEUS_KBD_<ROLE>_MODEL e.g. PROMETHEUS_KBD_JUDGE_MODEL
> ~/.prometheus/kbd/models.toml [roles]
> .kbd-orchestrator/project.json model_policy
> built-in default kbd-judge / kbd-critic / kbd-frontier

preflight-models.sh prints which layer supplied each role, so "where did that model come from?" is never a guess:

{
"status": "ok",
"gateway": "http://localhost:8181/v1",
"roles": {
"judge": { "model": "kbd-judge", "source": "~/.prometheus/kbd/models.toml" },
"critic": { "model": "kbd-critic", "source": "~/.prometheus/kbd/models.toml" }
},
"distinct_models": 2,
"config_defects": []
}

Three contracts that will bite you​

All three were simultaneously broken in the config this repo shipped before 2026-07-30, which is why the judge could not reach a second model. Each produces a failure that points away from its cause.

1. /v1/* requires a Bearer token unconditionally​

A config with no [general] master_key and no [[keys]] answers 401 to everything, /v1/models included. The symptom looks like a bad credential in the caller.

[general]
master_key = "${LITER_LLM_MASTER_KEY}"

2. deny_private blocks loopback​

[security].outbound_policy defaults to deny_private, which refuses any localhost / 127.0.0.1 base_url with OutboundForbidden. The symptom looks like a dead service.

[security]
outbound_policy = "off"
# or, if this host also reaches untrusted networks:
# outbound_policy = "allowlist"
# outbound_allowlist = ["http://localhost:8181"]

3. liter-llm never searches $HOME​

ProxyConfig::discover() walks the current directory upward looking for liter-llm-proxy.toml. It does not look in ~/.config. Always pass an absolute --config:

liter-llm api --config ~/.config/liter-llm/liter-llm-proxy.toml
liter-llm mcp --transport stdio --config ~/.config/liter-llm/liter-llm-proxy.toml

Without it, liter-llm mcp does not merely load zero models — it fails to start, because [mcp] stdio_trust_local lives in the config it was never given.

Configuring providers​

S="${CLAUDE_PLUGIN_ROOT}/skills/process/liter-llm-bridge/scripts/configure-models.sh"

bash "$S" check # report state; change nothing
bash "$S" repair # add ONLY the missing mandatory pieces
bash "$S" add-provider glm-coding # local-proxy|kimi|minimax|qwen|glm|glm-coding|kimi-coding
bash "$S" verify # live 1-token completion per role
bash "$S" migrate # retire the legacy config.toml

repair merges and never clobbers — a hand-added [[models]] entry survives a re-run.

Provider reference​

liter-llm has first-class providers for all of these; use the real registry prefixes.

Providerprovider_modelbase_urlkey var
openai-proxy (local)openai/gpt-5.6-solhttp://localhost:8181/v1none — sk-proxy-local placeholder
Kimi / Moonshotmoonshot/kimi-k2.5https://api.moonshot.ai/v1MOONSHOT_API_KEY
MiniMaxminimax/MiniMax-M2.5https://api.minimax.io/v1MINIMAX_API_KEY
Qwen / DashScopedashscope/qwen3-coder-plushttps://dashscope-intl.aliyuncs.com/compatible-mode/v1DASHSCOPE_API_KEY
Z.ai GLMzai/glm-4.7https://api.z.ai/api/paas/v4ZAI_API_KEY

Coding plans need a different endpoint​

The *-coding-plan identifiers in liter-llm's catalog are metadata, not routable providers — provider_model = "kimi-for-coding/k3" will not route. Keep a routable prefix and override the endpoint:

[[models]]
name = "kbd-glm"
provider_model = "zai/glm-5.2"
base_url = "https://api.z.ai/api/coding/paas/v4" # coding path, NOT /api/paas/v4
api_key = "${ZAI_API_KEY}"

Z.ai's documentation is explicit that a Coding Plan must use /api/coding/paas/v4 or the subscription quota is not drawn. add-provider glm-coding writes this for you.

OAuth-only subscription plans (MiniMax/Qwen portals, some Kimi coding plans) expose Anthropic-shaped endpoints that an OpenAI-REST caller cannot use. The wizard detects and refuses these rather than writing config that 404s later.

Secrets​

Never in the TOML. Keys live in ~/.prometheus/kbd/secrets.env (0600, outside any repo) and are referenced as ${VAR}:

set -a; . ~/.prometheus/kbd/secrets.env; set +a

liter-llm supports ${VAR} only — no ${VAR:-default} — and expands an unset variable to "", which surfaces much later as an unexplained 401. Both configure-models.sh check and scripts/check-model-config.sh verify every referenced variable is actually set.

Running a review​

SKILL_DIR="${CLAUDE_PLUGIN_ROOT}/skills/process/adversarial-review"

# 0. Load the gateway credential and DECLARE THE PRODUCER.
# Without KBD_PRODUCER_MODEL the collision check passes trivially.
set -a; . ~/.prometheus/kbd/secrets.env; set +a
export KBD_PRODUCER_MODEL="claude-opus-5"

bash "$SKILL_DIR/scripts/preflight-models.sh"
bash "$SKILL_DIR/scripts/build-review-packet.sh" --mode diff --phase "$PHASE" \
--target "$CHANGE_ID" --out ".../packet.json"
bash "$SKILL_DIR/scripts/dispatch-judge.sh" --mode diff \
--packet ".../packet.json" --out ".../findings.json"
bash "$SKILL_DIR/scripts/check-findings-sycophancy.sh" \
--findings ".../findings.json" --counter-key "adv-review-$CHANGE_ID"

Run evidence and review only after all planned phase production is complete. Stage artifacts can be prepared earlier, but an available review command does not authorize early review. Two packet modes:

  • --mode diff — post-implementation, in kbd-execute's QA gate. Reviews a change's diff against its acceptance criteria, after refine-validate passes and before archive.
  • --mode artifact — reviews a prepared assessment.md, analysis.md or plan.md artifact at the authorized completed-production boundary.

Reading the artifact​

This is how you tell a real review from theatre.

FieldMeaning
isolation_moderest-gateway:<url> — the endpoint that actually served it
cross_model_checkverified-distinct · same-model-collision · unverified-producer-unknown
judge_model / producer_modelthe compared pair
verdictBLOCK iff any CRITICAL finding

A healthy review:

{
"verdict": "BLOCK",
"judge_model": "kbd-judge",
"producer_model": "claude-opus-5",
"isolation_mode": "rest-gateway:http://localhost:8181/v1",
"cross_model_check": "verified-distinct"
}

unverified-producer-unknown means the packet carried no producer_model, so the guarantee could not be enforced — export KBD_PRODUCER_MODEL. harness-native means a native fallback was selected. Record its actual model family. Same-family review remains an independent-QA gap, even with fresh context.

Fallback chain​

Fallback dispatch and acceptance are separate. Report degradation; an unavailable independent judge leaves certification pending:

  1. REST gateway — packet isolation with the returned route and actual producer/judge evidence. A reachable endpoint or different alias alone does not prove a distinct model family.
  2. Harness-native subagent — when no gateway answers (exit 3). Prompt is exactly the mandate plus the packet, nothing else. Recorded as harness-native.
  3. Skip with warning (exit 4) — no judge available at all.

There is no liter-llm complete. The binary ships only api and mcp — it is a proxy server. Earlier revisions called that non-existent subcommand; because the guard only checked that the binary existed, the failure surfaced as "liter-llm unavailable" rather than as the CLI-contract mismatch it was. See 09b · liter-llm.

Troubleshooting​

Start here:

bash scripts/check-model-config.sh

Run this diagnostic only at the completed-phase boundary. Inspect its actual status and source provenance; a drift report does not establish who edited a copy or authorize overwriting it. Correct the owned source, preserve local changes and update through the verified installation path. The reported model or isolation_mode is configuration evidence until an actual review runs.

SymptomCauseFix
HTTP 401 on everythingno master_keyconfigure-models.sh repair
OutboundForbiddendeny_private vs localhostset outbound_policy
MCP server won't startmissing --configpass an absolute path
status: config_brokensee config_defectsconfigure-models.sh repair
cross_model_check: unverified-producer-unknownno producer declaredexport KBD_PRODUCER_MODEL
JUDGE_MODEL_COLLISIONjudge == producer, no alternativeadd a second provider
Unset ${VAR} → later 401liter-llm expands unset to ""source secrets.env

Anti-theatre gate​

The judge's own report is screened by sycophancy-correction before it is surfaced. A zero-finding report must carry checked_classes — the failure classes the judge examined and why each does not apply. A clean report with no due-diligence trail is rejected as theatre, not accepted as praise.

See also​