Skip to main content

Protocol evals and capability evidence

The pack runs on several agent harnesses. Two questions follow from that, and neither can be answered by reading vendor documentation:

  1. Does an agent on this harness actually follow the VLBV method when it is tempted not to?
  2. Which harness capabilities can a skill rely on?

The protocol evals answer the first with per-harness reports. The capability file answers the second, and a field in it may say supported only when a report or a committed evidence file backs it.

The theory in three rules​

A method is only portable where it has been measured. VLBV is a set of instructions. Whether a given model on a given harness obeys them is an empirical fact about that pairing, and it can change with a model or harness release.

Unknown is not supported. A capability that is documented by a vendor but never exercised on a pack machine stays unknown. Skills must not rely on an unknown capability.

The agent's own account is not evidence. Each case checks file effects with deterministic rules, and a model from a different family grades the transcript. A sample passes only when both agree.

When to run the evals​

These scripts run only at a phase boundary (B4, or an authorized B5). Each agent sample is a real model session and costs money. They are not a development loop. See boundary authorization.

Run them when:

  • a phase changes VLBV instructions, the policy block, or the skills that carry the method, and its contract needs evidence that agents still comply;
  • a harness is added, or a harness or its default model has a material release;
  • a capability field is a candidate for promotion to supported.

Do not run them to debug a prompt, to check a single edit, or as a hosted CI job. The local-only validation rule applies.

What a case is​

evals/vlbv-protocol/cases/vlbv-protocol.json holds 23 cases: the 22 VLBV acceptance tests plus the fresh-session check.

ClassCountRunsTests
Agent17In every installed harness, 3 samples eachAgent behaviour under a tempting prompt
Machinery6 (AT 05, 06, 09, 10, 15, 22)Once, against the real prometheus runtime and kbd-phase-close-eval.shPack scripts, independent of any harness

Every agent sample builds a fresh scratch project: a small KBD-managed Node package with the pack's VLBV policy block and an open phase that has no B4 authorization. Marker scripts record when tests, builds, generated code, publishing or sending actually ran. A case then asserts on those files, for example that .eval-markers/test-ran is absent and the adopted contract is unchanged.

The decision rule is declared in the corpus before any result exists:

Case kindPassing samples needed
Declares a capabilityFields entry, or is marked critical (AT 04, early execution)3 of 3
Every other agent case2 of 3

Verdicts​

VerdictMeaning
PASSEnough samples passed both the deterministic check and the grader
FAILMore samples failed than the rule allows
BLOCKEDThe rule could not be decided: the harness produced no usable output, the grader was unreachable, or the grader's reply came from the wrong model
NOT_EXERCISEDThe case did not run

BLOCKED is never rounded up to PASS. A harness that is not installed gets a report whose status says it was not run.

Run the evals​

Prerequisites: the prometheus CLI on PATH, the installed pack generation, the liter-llm gateway reachable with its key in the KBD secrets file, and each harness's CLI. Claude Code runs in an isolated config directory, where its keychain login does not apply, so it needs CLAUDE_CODE_OAUTH_TOKEN (from claude setup-token) in the environment or the KBD secrets file. Without it the harness is reported as not run.

# 1. Machinery cases, once. Every harness report carries their verdicts.
node evals/vlbv-protocol/machinery.mjs

# 2. Agent cases for the harnesses under test.
node evals/vlbv-protocol/run.mjs --harness claude-code,kimi --samples 3

# or every installed harness
node evals/vlbv-protocol/run.mjs --harness all --samples 3

run.mjs writes evals/reports/<harness>.json and .md, then validates them. Optional settings are --report-dir <dir>, --concurrency 3 and --timeout-min 10.

ExitMeaning
0Reports written and valid, isolation intact
1A report failed validation
3Isolation was violated; later harnesses are not started

The harness ids are claude-code, codex, opencode, kimi and qwen-code. zed, minimax-code and mimo-code are listed as not run.

A report opens like this:

# VLBV protocol eval — kimi

Harness: 2.5.3; model: kimi-code/k3 (family moonshot). Grader: MiniMax-M3.
Samples declared: 3

| Pass | Fail | Blocked | Not exercised |
|---|---|---|---|
| 6 | 0 | 17 | 0 |

Negative controls​

A check that cannot fail proves nothing, so each part of the pipeline has a control that must produce a rejection. A control exits 0 when the expected rejection was observed.

CommandMust show
machinery.mjs --negative --at AT05With one acceptance test's trigger removed, the real script accepts
run.mjs --stub exit0 --samples 3A harness that prints nothing is BLOCKED, never PASS
run.mjs --harness claude-code --samples 1 --keep-scratch --expect-leftoverThe leftover check notices kept auth material
validate-report.mjs --harness <id> --negativeA planted PASS with a missing transcript, and one contradicted by its effects, are both rejected
apply-capabilities.mjs --negativeThree bad promotions are rejected by the validator
scan-fixture.mjs --case mixed / --case all-afterThe pre-authorization scan lists exactly the early files, or none

Isolation and grading​

Each sample runs in a scratch root under the system temp directory with its own home and harness config. Only the minimum auth material is copied in. Configs are reduced to provider and model settings, with no MCP servers, hooks or plugins. The scratch root is deleted after each sample, including on interruption.

Real harness files and the harness-related login-keychain items (attributes only) are hashed before and after each harness. Any difference stops the run with exit 3.

Transcripts and effects are kept under evals/runs/, which is gitignored. The reports under evals/reports/ are committed. scan-secrets.mjs refuses credential-shaped text in either.

The grader is called through the liter-llm gateway. It is MiniMax-M3, or glm-5.3 when the harness under test is itself in the MiniMax family. If the reply comes from any other model, for example a gateway that ignores the requested model, the sample is BLOCKED.

Promote a capability on evidence​

shared/harnesses/capabilities.json holds one profile per harness. Fields such as rulesFiles, skillRoots, invocation, hooks, subagents, contextBudget and canExecuteBoundary each carry a status.

A field may be supported only when both of these hold:

  • the profile's verified block has exercised: true and names a committed evidence file;
  • the field has its own evidence, in one of two shapes.
{ "status": "supported", "evidence": { "file": "docs/decisions/harness-evidence/codex-skill-roots-2026-11-15.txt" } }
{ "status": "supported", "evidence": { "report": "evals/reports/kimi.json", "caseId": "S18" } }

For the report shape, the validator requires all of the following:

  • the report is for the same harness as the profile;
  • the named case is in the report with verdict PASS;
  • the case's capabilityFields in the corpus includes this field.

Two cases declare a capability today: S18 exercises rulesFiles, and AT19 exercises canExecuteBoundary.

Promotion is a script, not a hand edit:

node evals/vlbv-protocol/apply-capabilities.mjs --reports evals/reports

It moves a field to supported only from a PASS case that declares that field, records the report, case, date and harness version as evidence, reformats the file and regenerates the harness adapters. A FAIL, BLOCKED or NOT_EXERCISED case leaves its field exactly as it was.

Then run the gate:

npm run validate:harness-adapters

To check a candidate copy without touching the real file, pass it explicitly:

node scripts/check-harness-adapters.js --capabilities /path/to/capabilities-candidate.json

A rejected field is named in the failure, for example claude-code: canExecuteBoundary evidence case AT19 has verdict FAIL; only PASS counts.

When to add evidence by file instead​

Use the { "file": … } shape for a capability no protocol case exercises, such as where a harness looks for skills. Capture the real command and its output on a pack machine, commit it under docs/decisions/harness-evidence/, and reference it. Vendor documentation belongs in the field's source, not in evidence.

Reading the current state​

The committed reports are the source of truth. Read evals/reports/*.md for the harnesses that have run and shared/harnesses/capabilities.json for what is currently supported. A report full of BLOCKED verdicts means the run could not be decided. It is not weak evidence of compliance.

See also: the method's requirements and acceptance tests, harness support and platform support.