Evaluation and evolution

Name what the evidence measures

Historical quiz results are observations of recorded performance. Learner feedback and inferred conversation signals are proxies. Synthetic learners and model judges can expose behavioral weaknesses, but they do not establish improved learning for people. Retention, transfer, and learning gain remain unknown unless the corresponding measurements exist.

The historical bench command summarizes records. Teaching-bench executes fresh training cases with the active revision. Auto-improve proposes a bounded teaching skill and evaluates it before activation. These are different operations with different claims.

Run a bounded experiment

Teaching-bench runs the training split only and never activates a revision. Auto-improve evaluates an incumbent and candidate on paired validation and an independently gated sealed holdout. Model-backed evaluation requires an authenticated provider and consumes provider usage.

Use an independently authored case pack to define an experiment. A positional topic is not an experiment specification. Force bypasses the cooldown only; it cannot bypass a failed gate or reopen consumed holdout evidence.

keating teaching-bench --cases ./independent-cases.json
keating auto-improve --cases ./independent-cases.json

Keep proposal and evaluation authority separate

The actor receives the learner conversation prefix and teaching instructions. The judge receives the fixed case and actual execution. The maintainer and proposer work from training evidence and indexed knowledge, not held-out conversations or their answer criteria.

The activation gate requires both validation and holdout to pass. The documented gate includes at least six independent families, no family regression, no failed critical candidate criterion, mean improvement of at least 0.05, and a one-sided paired sign-test p-value at most 0.05. Missing executions or judgments invalidate the comparison. These are behavior-benchmark rules, not a human-study power analysis.

Preserve revisions, failures, and exhausted evidence

Teaching procedures and maintained knowledge snapshots are content-addressed. Experiments persist their verdict before changing the active revision pointer. New sessions use an accepted revision; running sessions retain the one they started with. Failed proposals remain inspectable, and useful maintained knowledge can survive a rejected proposal.

A release holdout is consumed before execution, including when a run later rejects or crashes. Renaming case IDs or changing numbers does not create independent evidence. Renewal needs genuinely new tasks and families, authored without exposing them to the proposal loop.

Measure learner outcomes directly

Learning checks distinguish precheck, immediate, delayed, and transfer stages and record whether work was unassisted, assisted, or unknown. This creates a place to collect the measurements that a conversation transcript cannot supply. A recorded score still needs careful interpretation: assistance, missing follow-up, task comparability, and study design affect the conclusion.

Legacy parameter evolution and prompt-quality scoring remain diagnostic. They do not silently replace source prompts or authorize the bounded teaching-revision gate. Report local deterministic tests, hosted execution, client behavior, and human-learning evidence separately.

keating learning-check start fractions
keating learning-check list
keating learning-check show <check-id>