Skip to main content

Module targeted

Module targeted 

Source
Expand description

Targeted eval providers (CH-17): skill-activation accuracy, routing accuracy, and context-strategy efficiency.

Unlike the general starter suite (which exercises real LLM completions through OrchestratorCompletionProvider), these three suites test deterministic runtime decision code paths directly — no model call, no API key required. Each provider seeds a small, fixed fixture once, then complete() interprets one JSON-encoded case input per call and returns the actual decision the real runtime code made, as a string the existing Contains/ExactMatch scorers can grade unchanged.

Structs§

ContextEfficiencyProvider
Exercises strategy_for_model (CH-05’s model-aware Auto resolution) — a pure function, no fixture needed. Case input is a JSON object {"effective_context_tokens": <u32>}; the returned completion describes the resolved strategy and its key parameter.
RoutingProvider
Exercises ModelRouter::route (CH-03/CH-09’s real capability-filter + health-aware selection) against a small fixed fixture registry seeded with real catalog providers. Case input is a JSON object with a requirements field (deserialized as RouteRequirements) and an optional trip_health_for provider id (simulates a CH-03 cooldown before routing). The returned completion is the selected "provider/model" string, or "none".
SkillActivationProvider
Exercises SkillService::match_skills (the real keyword matcher CH-08 instruments) against a small fixed set of fixture skills. Case input is the raw query text; the returned completion is the comma-joined matched skill_ids (or "none" when nothing matched).