Skip to main content

Installation, doctor, and recovery

The installer is strict by default. It builds or accepts the independently pinned Exec release binary, checks its version, stages it on the destination filesystem, applies and verifies the platform signature, atomically replaces the destination, reads back the installed hash/version/signature, and writes a mode-0600 installation receipt. Any failure restores the prior binary or removes the failed first install.

Build and install locally​

Finish the phase's production changes and local integration gates before installing. The installer invokes the canonical reproducible build helper by default:

bash scripts/install-prometheus-exec.sh
prometheus-exec --version

scripts/build-prometheus-exec.sh stages a workspace, uses the crate's pinned toolchain and remaps source paths. A plain Cargo release build may embed a checkout path and cannot be assumed to match the certified hash. Keep the source build, expected version and the config/prometheus-exec-binary.json hashes aligned through the authorized final certification process; do not bypass a mismatch. Exec's version is independent of the skill pack and other services.

The certified build hash​

The installer refuses to install a binary whose SHA-256 differs from a committed value. That gate is what lets a receipt say which exact bytes produced a result. It only works if the same source gives the same bytes everywhere, so the build is made reproducible and the hash is pinned per platform.

One hash per target​

A release binary for Apple Silicon and one for Intel are different files. A single hash can match only the architecture that certified it, so the manifest pins one value per Rust host target:

{
"expectedVersion": "prometheus-exec 1.7.0",
"expectedBuildSha256ByTarget": {
"aarch64-apple-darwin": "<64 hex characters>",
"x86_64-apple-darwin": "<64 hex characters>"
}
}

The installer asks rustc -vV for the host target from inside the crate directory, because rust-toolchain.toml is resolved from there and an x86_64 toolchain on Apple Silicon builds x86_64 binaries. It then reads that target's entry. When rustc is absent it falls back to the kernel's architecture. A manifest that still has only the older single expectedBuildSha256 field keeps working: that value is used for every target.

Keep one map entry per line. The manifest is read with awk, not a JSON parser.

Check what a machine would require without building or installing:

bash scripts/install-prometheus-exec.sh --dry-run
[dry-run] would build crates/prometheus-exec --release via scripts/build-prometheus-exec.sh
[dry-run] would require source sha256:fe2f2c5e… (target x86_64-apple-darwin), then verify, stage, sign, atomically install, and read back …/prometheus-exec

The installation receipt records the same target beside the expected, built and installed hashes.

What makes the build reproducible​

The helper removes three machine-specific inputs:

InputWhy it changes the hashWhat the helper does
Checkout pathrustc embeds absolute source paths in panic locations and file!()Remaps the repository, Cargo home and build directory with --remap-path-prefix
Path dependencies outside the crateCargo hashes an out-of-workspace dependency's absolute path into every symbolBuilds in a staged workspace that contains the crate, substrate/exec-* and wit/
The rust-src componentWhen it is installed, rustc rewrites standard-library paths from /rustc/<commit>/… to the local toolchain directory, which embeds the home directoryRemaps the toolchain's source directory back to /rustc/<commit>

The third one is easy to trigger without noticing. Installing rust-analyzer adds rust-src, and from then on a plain build produces a different binary from the same source. With the remap in place, the set of installed toolchain components no longer affects the hash.

Flags are passed through CARGO_ENCODED_RUSTFLAGS and RUSTFLAGS is unset, so ambient compiler flags cannot alter the certified binary.

When to recertify​

Recertify when one of these changed on purpose:

  • source under crates/prometheus-exec, substrate/exec-* or wit/;
  • the crate's Cargo.lock;
  • the pinned channel in crates/prometheus-exec/rust-toolchain.toml;
  • scripts/build-prometheus-exec.sh itself.

Do not recertify to make a mismatch go away. A mismatch with none of those changes means the build is not reproducible on that machine, and that is the defect to find.

Print the hash for the machine you are on:

bash scripts/build-prometheus-exec.sh --print-hash

Write the value to that machine's entry in expectedBuildSha256ByTarget. A hash is specific to its target, so every pinned target must be recertified, each on a machine of that target, and all entries updated in the same commit. Until that is done, the installer on the other architecture correctly refuses the build.

A no-op Cargo build proves nothing about a pin. If Cargo reports the crate as fresh, the hash you read belongs to an earlier build. The integration test has a real-build case that copies the working tree to a second directory, builds there from scratch and requires the pinned hash:

PROMETHEUS_EXEC_TEST_REAL_BUILD=1 bash scripts/tests/install-prometheus-exec.test.sh

Run it at the phase boundary, with no other Cargo build active on the machine.

Overrides​

VariableUse
PROMETHEUS_EXEC_EXPECTED_SHA256Require this hash instead of the manifest's. For a controlled one-off, such as installing an approved prebuilt binary
PROMETHEUS_EXEC_SOURCE_BINInstall this file instead of building
PROMETHEUS_EXEC_HASH_MANIFESTRead the pin from another manifest file
PROMETHEUS_EXEC_EXPECTED_VERSIONRequire a different --version string

Exporting PROMETHEUS_EXEC_EXPECTED_SHA256 in a shell profile defeats the gate on that machine: any build it names installs, whatever the committed pin says. Keep it for single commands.

Validate the checked documentation examples against the installed binary and active signed generation using disposable state:

npm run check:docs-exec-examples
npm run docs:examples

The first command checks source syntax, expected outputs, relationship semantics, and documentation drift. The second exercises the existing local certification driver and removes its temporary state on exit. Hosted workflows do not run these product examples.

Inspect service installation without changing the machine:

bash scripts/install-prometheus-exec-service.sh --dry-run

The shipped service definition is macOS-only; a binary build on another platform does not provide a matching managed service. On macOS, the service installer creates a private identity if absent, renders the checked LaunchAgent, validates it with plutil, installs it atomically, and optionally loads ai.prometheus.exec. Use --no-load when only the definition should be installed.

Non-mutating doctor​

prometheus-exec doctor \
--socket "$HOME/.prometheus/run/prometheus-exec.sock" \
--state-dir "$HOME/.prometheus/exec" \
--identity "$HOME/.prometheus/exec/identity.json" \
--plugin-root "$HOME/.prometheus/plugins/prometheus-skill-pack" \
--service-definition "$HOME/Library/LaunchAgents/ai.prometheus.exec.plist" \
--mcp-schema ./docs/reference/api/prometheus-exec.mcp.json \
--exclude remote-queue \
--format json

Doctor reads binary identity, the service definition, the LaunchAgent label's loaded state, socket health/readiness and permissions, receipt identity, Tier P/W availability, signed component trust, ledger/receipt/CAS reconciliation, MCP schema parity, and an optional remote queue. A supplied service definition fails diagnosis if its LaunchAgent is not loaded. Exclusions are applied before check construction. Diagnosis does not install, start, stop, compile, consume, contact excluded services, or rewrite state.

Logs and common failures​

The LaunchAgent writes standard output and error under the configured Prometheus log directory. Check the JSON doctor result first; it preserves individual failed checks instead of translating empty/error output into success.

Review Security and trust boundaries before changing identity, component trust, socket, or sandbox settings.

SymptomMeaningAction
source hash mismatch for target <triple>; expected …, got …The built binary is not the certified one for this architectureIf exec source or the toolchain changed on purpose, recertify every pinned target. Otherwise find what made the build differ
no valid certified build hash for target '<triple>'The manifest has no entry for this machine's targetRun build-prometheus-exec.sh --print-hash on this machine and add the entry through the certification process
/health works and /ready is 503Process is live but a required subsystem is not readyRead the failed readiness subsystem and service log
component-unauthorizedBytes, hash pin, active generation, or rollback state does not authorize the componentVerify the signed plugin generation and component descriptor
request_hash_conflictOne request ID was reused with different canonical contentGenerate a new request ID or restore the original request
artifact_not_foundCAS does not contain the requested digestReconcile the run/receipt pins; do not fabricate an artifact
Tier P backend unavailableThe supported OS sandbox was not found or certifiedDo not enable a direct-process fallback
remote queue omittedRemote diagnosis was not selected or was excludedThis does not affect local health or offline verification

Recovery and rollback​

After a crash, restart the same identity, state directory, socket, and plugin root. Reconciliation recovers logged receipts, requeues never-spawned work, and marks post-spawn interrupted work terminal without re-execution.

Binary backups are content-addressed under the install backup directory. Plugin rollback is separate and atomic:

node scripts/install-plugin-generation.js --rollback
node scripts/install-plugin-generation.js --verify

The pointer switch moves payload, component metadata, skill/component indexes, stable dispatchers, and target receipt authority together. It never rewrites a signed immutable generation.

Next: Platform and evidence status.