Beyond Evals Lab
An executable proof-of-concept for practical assurance of agentic systems. Tests, controls, verification, validation, trajectory analysis, monitoring, and offline evals — as separate, composable mechanisms.
pnpm install && pnpm testNo API keys required. Rule-based agent, fully deterministic.
The Thesis
Agent assurance is not equivalent to agent evaluation. A production agent needs several distinct mechanisms — tests, controls, verification, validation, trajectory analysis, and monitoring — and assurance combines these pieces without pretending they are interchangeable.
Deterministic ≠ Test
Deterministic code can be an eval grader. Probabilistic code can be a test. The distinction is about what question is being answered, not the nature of the code.
Probabilistic ≠ Eval
A model-based grader is probabilistic but its output is still evidence about one run — not a population estimate. The context determines the epistemic role.
Verification is per-execution
Given run-123, verification returns claims about that specific execution. Evaluation estimates behavior across a distribution. Same checker, different question.
No single quality score
Outcome, verification, validation, trajectory, controls, and efficiency are reported separately. Composite scores are explicitly experimental if they exist at all.
Experiments
Seven experiments, each runnable with a single command. Understand each assurance concept by observing it in action — no theory required.
Tests are not the opposite of evals
Vitest deterministically establishes properties of implementation components. The eval harness runs tasks sampled from an empirical task distribution. Some eval graders are deterministic. Determinism does not distinguish tests from evals.
pnpm testpnpm evalVerify one run
`verifyRefundOutcome()` establishes evidence for a specific execution. This is verification — claims about what happened in one run.
pnpm demo:successEvaluate many runs
The same verifier now contributes to `verified_outcome_rate`. Same function, different epistemic role. Verification = one run. Evaluation = population estimate.
pnpm evalCorrect outcome, unacceptable path
The agent produced the correct refund but attempted a prohibited action first. Outcome-only evaluation would have missed this.
pnpm demo:trajectory-failureVerification is not validation
The system correctly executed a refund it should never have chosen. Verification asks 'was this executed correctly?' Validation asks 'should this have been done at all?'
pnpm demo:validation-failureGuardrails are not verification
A control asks: May this action execute? A verifier asks: What actually happened? Neither replaces the other.
pnpm demo:control-blockProduction creates future evals
Production traces become the empirical task distribution for future evaluations. The human curation step is preserved deliberately.
pnpm demo:feedback-looppnpm traces:mineArchitecture
The runtime pipeline separates concerns that most frameworks conflate. Controls run before mutation, verification after, and the eval harness reuses the same verifiers — just with a different question.
Runtime Pipeline
parse → control → execute → mutate → trace Agent never calls tools directly. Runtime mediates every interaction.
Offline Eval
dataset → run → verify → grade → aggregate Same verifyRefundOutcome() used in runtime and offline eval.
Current Metrics
From pnpm eval across 20 cases in the core dataset. No single aggregate score — each dimension
is reported independently.
across 22 test files
deterministic claims per run
business-intent rules
over 20 eval cases
path analysis
Outcome PASS / Trajectory FAIL
Important: The disagreements are more informative than any aggregate pass rate. 8 cases where outcome passes but trajectory fails, 3 cases where verification passes but validation fails. These are the cases that demonstrate why outcome-only evaluation is insufficient.
Quick Start
No API keys required. The default agent is rule-based and fully deterministic.
Key Design Decisions
Same verifier function, two roles — runtime assurance and eval grading
Claim-oriented verification — structured evidence per claim, not a boolean
No aggregate quality score — dimensions reported separately
Controls + verification are complementary — prevent vs. detect
Human curation in the feedback loop — observed behavior ≠ expected behavior
Conceptual Map
Deterministic/probabilistic describes the evidence mechanism. Tests/verification/evals/monitoring/controls describe how the evidence is being used. These are different axes.
| Mechanism | Scope |
|---|---|
| Test | implementation |
| Control | pre-action |
| Verification | one run |
| Trajectory | one run |
| Validation | scenario/system |
| Monitoring | production |
| Eval | population |
| Assurance | system-level |