Expand description
Targeted eval providers (CH-17): skill-activation accuracy, routing accuracy, and context-strategy efficiency.
Unlike the general starter suite (which exercises real LLM completions
through OrchestratorCompletionProvider), these three suites test
deterministic runtime decision code paths directly — no model call, no
API key required. Each provider seeds a small, fixed fixture once, then
complete() interprets one JSON-encoded case input per call and returns
the actual decision the real runtime code made, as a string the existing
Contains/ExactMatch scorers can grade unchanged.
Structs§
- Context
Efficiency Provider - Exercises
strategy_for_model(CH-05’s model-awareAutoresolution) — a pure function, no fixture needed. Case input is a JSON object{"effective_context_tokens": <u32>}; the returned completion describes the resolved strategy and its key parameter. - Routing
Provider - Exercises
ModelRouter::route(CH-03/CH-09’s real capability-filter + health-aware selection) against a small fixed fixture registry seeded with real catalog providers. Case input is a JSON object with arequirementsfield (deserialized asRouteRequirements) and an optionaltrip_health_forprovider id (simulates a CH-03 cooldown before routing). The returned completion is the selected"provider/model"string, or"none". - Skill
Activation Provider - Exercises
SkillService::match_skills(the real keyword matcher CH-08 instruments) against a small fixed set of fixture skills. Case input is the raw query text; the returned completion is the comma-joined matchedskill_ids (or"none"when nothing matched).