Skill evaluations

Evaluations answer “is this skill actually good?” with reproducible test runs. An evaluation runs the skill against a set of test cases, scores each round against a rubric, and aggregates the results.

Running one

From an agent’s Skills tab, open a skill → Evaluate. You define:

• Test cases: imported, hand-written, or generated for you.
• Rounds: how many times each case runs (parallelism configurable).
• Rubric: assertions the output must satisfy.

An orchestrator session executes the experiment; every case is a real session so scores reflect actual runtime behavior.

Reading results

• Trend: average score and pass rate over time, so you can see whether an edit to the skill improved or regressed it.
• Slices: results grouped by runtime × model, with a latest-vs-earlier delta.
• Detail: every round’s transcript is preserved for inspection.

A skill whose evaluation keeps failing is a skill to fix (or retire). The rubric you wrote is the spec.

Quality signals

Independently of formal experiments, agents can attach subjective quality signals to their work (self-assessed confidence, debugging notes). Signals never influence dispatch or model selection; they’re an audit trail you can review alongside results. The Evaluation toggle in Labs controls whether agents emit them.