Module eval 

Evaluation harness — domain model + scorers (foundation layer).

Defines eval cases/suites/scores/results and a Scorer contract with rule-based scorers. This module performs no IO and runs nothing on its own; the suite loader, runner, persistence, and CLI surface are separate changes.

Re-exports

pub use targeted::ContextEfficiencyProvider;

pub use targeted::RoutingProvider;

pub use targeted::SkillActivationProvider;

Modules

cli

Eval harness CLI surface (EH5).

targeted

Targeted eval providers (CH-17): skill-activation accuracy, routing accuracy, and context-strategy efficiency.

Structs

Contains

1.0 when output contains the case’s expected substring, else 0.0.

EvalCase

A single evaluation case: an input and an optional expected output.

EvalResult

The scores for one case in a run (persisted to a file by EH4).

EvalSuite

A named collection of eval cases (loaded from a golden suite file by EH2).

ExactMatch

1.0 when output equals the case’s expected, else 0.0 (0.0 if no expected).

JsonValid

1.0 when output parses as valid JSON, else 0.0.

LlmJudge

An LLM-as-judge scorer. Holds the provider it calls (captured at construction) since Scorer::score only receives (case, output).

NonEmpty

1.0 when output is non-empty after trimming, else 0.0.

PatternMatch

Literal substring/anchor match (no regex dependency — covers the common contains/format checks; a true-regex scorer is a later option).

RegressionEntry

One scorer’s regression verdict vs a baseline.

RegressionReport

The full regression report for a run vs its baseline.

Runner

Runs an EvalSuite through a CompletionProvider and scores each case.

Score

A normalized score in the range 0.0–1.0 produced by a Scorer.

Sycophancy

Quality scorer: higher value = less sycophantic (1.0 - sycophancy_score), derived from the existing rule-based sycophancy detector. Uses a default detector config so eval scoring is independent of runtime gating.

Enums

PatternMode

How a PatternMatch matches its literal pattern against the output.

ScorerSpec

A declarative scorer entry in a suite file. Serde-tagged by type (snake_case), e.g. { "type": "pattern_match", "pattern": "ERROR", "mode": "starts_with" }.

Traits

CompletionProvider

Produces a model output for an eval case’s input. Implemented by the CLI over Orchestrator::chat_non_streaming; stubbed in tests.

Scorer

Maps a (case, output) pair to a normalized Score.

Functions

build_scorers

Build the scorers for a suite: its declared ScorerSpecs, or — when it declares none — the default set from default_scorers. provider backs any llm_judge scorers.

compare

Compare a current summary to a baseline. A scorer regresses when its mean drops below the baseline by more than threshold. With no baseline entry for a scorer, it is not a regression (a run can establish a baseline).

default_scorers

Default scorer set when a suite declares none: quality scorers always, plus expected-based scorers when every case carries an expected output.

load_baseline

Load a suite’s baseline summary, or None if it does not exist.

load_suite

Load an EvalSuite from a golden file, choosing the parser by extension (.json, .yaml, .yml). Returns an error (never panics) on a missing file, an unparseable body, or an unknown extension.

save_baseline

Save a suite’s baseline summary, creating dir if needed.

save_results

Write a run’s results to <dir>/<suite>-<ts>.json, creating dir if needed.

summarize

Mean score per scorer across all results in a run.

Type Aliases

ScoreSummary

Per-scorer mean score (0.0–1.0) over a run. BTreeMap for deterministic order.