Progressive, Uninterrupted Eval Building: A Daily Workflow for Agent-Driven Development

Research date: Aug 31, 2026. Workflow applies to Codex CLI and any coding agent with a scriptable non-interactive mode; the file formats are tool-neutral. For how evals interact with persistent memory systems, see the companion study Evals and Agent Memory.

TL;DR

An evaluation suite should be built as a by-product of daily agent-assisted work, not as a separate project. The pattern has three rules:

Capture-first: every time the agent gets something wrong — or you have to correct it — append one JSONL row to the golden set. Cost target: under a minute, zero context switch.
Checkpoint runs: run a tiny smoke suite at natural boundaries (end of task, before commit) and the full suite in CI or nightly. Never run the full suite inside your interactive flow.
Agent-maintained: the agent appends cases, runs the suites, fixes the harness, and distills weekly. The human reviews diffs, not every row.

This mirrors the pattern OpenAI documents for testing agent skills: define success, start with a small prompt set, grow it from real failures, grade with deterministic checks on traces (codex exec --json) and a rubric pass (--output-schema), and let failures drive coverage (https://developers.openai.com/blog/eval-skills).

Why dedicated eval sessions fail

The conventional approach is to schedule "eval work": set aside a day, collect a few hundred cases, build a harness, run it once, and file the results. In practice:

The eval work competes with shipping and loses.
A golden set built in a weekend goes stale in two weeks, because the product and prompts changed.
Nobody re-runs a suite that lives outside the normal commit flow.
The results, when they exist, are a snapshot, not a living regression signal.

The insight that fixes this: agent sessions are already recorded. Every task, every wrong answer, every correction is raw material for a golden set. The only new habit is capture — and capture is one line appended to a file.

Rule 1 — Capture without stopping

Capture triggers are things that happen naturally during a working day:

Trigger	Example	Case type
Wrong or incomplete output	Agent answers a support question without the required refund window	Functional
Corrected behavior	You say "no, ask before deleting" and the agent redoes it	Behavioral
Hallucinated fact or citation	Agent cites a file that does not exist	Functional
Broken tool use	Agent runs a command against the wrong directory	Process
Should-not-trigger	Agent rewrites unrelated code when asked a narrow question	Negative control
Refused valid task	Agent declines to do something it is allowed to do	Behavioral
Security / red-team miss	Agent leaks a tool name or accepts a prompt-injection bait	Safety
Ambiguous instruction	The prompt is genuinely unclear; success needs definition	Coverage

Each capture is one line appended to evals/cases.jsonl:

{"id": "c-0231", "input": "Dispute a charge on my account", "expected": "Must mention Transaction History, Dispute button, and 60-day window", "rubric": ["refund_window", "steps"], "category": "billing", "source": "session-2026-08-31-auth", "date": "2026-08-31", "must_pass": true}

The expected field does not need to be a full reference answer. For most cases it is a rubric: facts that must appear, things that must not, or an LLM-judge instruction. That is what keeps capture fast.

To make capture a habit, put it in the agent's durable instructions. In AGENTS.md:

## Eval capture
When you produce a wrong, incomplete, or unsafe result for a user request — or the user corrects you — append one JSONL case to evals/cases.jsonl (schema below). Keep it to one line. If a similar case already exists, skip it.

Two capture shortcuts make this even cheaper:

Correction-as-case: when you correct the agent mid-task, the corrected transcript is a golden case. Ask the agent to add it.
A slash-command or skill: a tiny eval-capture skill that prompts for the input, expected behavior, and category, then appends the JSONL row.

Rule 2 — Run at checkpoints, not in the interactive path

Split the suite into two tiers so the fast feedback never blocks the interactive flow:

Tier	Size	When it runs	Time budget
Smoke set	10–20 cases	End of task, before commit	Seconds
Full set	50–500+ cases	CI on every PR, nightly full run	Minutes

The runner follows the official agent-eval pattern:

For each prompt, run codex exec --json --full-auto "<prompt>" and save the JSONL trace.
Run deterministic checks over the trace's command_execution events: did it run the expected command, create the expected file, avoid the forbidden one? Every command is inspectable, so regressions are explainable.
For qualitative requirements, run a read-only second pass with --output-schema so the agent returns a structured rubric (overall_pass, score, per-check notes) that the harness can diff across runs.

Tooling: Promptfoo is the recommended runner — OpenAI is winding down its hosted Evals product and officially recommends Promptfoo for continuing and extending evaluation workflows (https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo). There is also a Promptfoo Codex plugin for building and maintaining eval suites with Codex. DeepEval is a reasonable Python-native alternative.

The gate philosophy matters more than the tool: a regression blocks the merge; an absolute score does not. If the suite score dips only because a case is stale, that is a case-editing task, not a merge blocker.

Rule 3 — Let the agent maintain the harness

Keep the eval infrastructure as normal repo code the agent owns:

evals/
  cases.jsonl            # the golden set, grown daily
  prompts-smoke.csv      # the 10-20 smoke prompts (id, should_trigger, prompt)
  rubric.schema.json     # --output-schema for qualitative grading
  run-smoke.sh           # fast tier
  run-full.sh            # CI tier
  artifacts/             # traces and graded outputs (gitignored)

The agent can scaffold this on request, append cases, run tiers, and fix the runner when a case fails because of harness bugs rather than behavior. The human's job is review: weekly, look at the diff of cases.jsonl, approve or reject, and prune duplicates.

Weekly distillation

Once a week, hand the agent the accumulated captures and ask it to produce a proposed revision:

merge near-duplicate cases and keep the more precise one;
upgrade recurring one-off failures into stable cases;
re-balance categories so no area dominates;
flag cases whose expectations are no longer correct;
report pass-rate trends by category since the last run.

This is the same "propose, don't directly mutate" pattern the blog already uses for repository memory (Local Repository Memory for Coding Agents): the agent drafts, the human approves.

Growth cadence

Cadence	Action	Effort
Per task	Capture failures/corrections as JSONL rows	< 1 minute
Per commit	Run the smoke tier; fix regressions before merging	Seconds
Per PR (CI)	Full suite; block on regressions only	Automated
Weekly	Distill cases, re-balance categories, review pass rates	15 minutes
Monthly	Prune stale cases, review coverage by category, tune rubrics	30 minutes

The suite grows at the speed of real work — a few cases a day — and stays small enough to remain meaningful. A golden set of a few hundred high-provenance cases beats a few thousand scraped ones.

Golden-set hygiene

Provenance: every case carries source (session id, ticket, prompt id) and date. You must be able to trace a case back to the event that created it.
Negative cases: keep should_trigger: false / must-not cases. False positives are the most common silent regression.
Dedupe at write time: if a nearly identical case exists, update it instead of appending.
No secrets: never capture credentials, tokens, or customer data in cases. Redact before appending.
Stable ids: ids never change; edits are new versions. Score history stays comparable.
Must-pass vs informational: must_pass: true blocks merges; informational cases only trend in the weekly report.

Anti-patterns

Saving up captures for a monthly "annotation day" — capture is only cheap when it happens at the moment.
Building a huge golden set up front — it goes stale and never gets maintained.
Running the full suite inside the interactive session — it teaches you to stop running it.
Hand-editing a spreadsheet of cases — JSONL in git gives you diffs, review, and CI.
Treating a score as a gate without knowing which cases changed — the per-case trace is the actual artifact.

Sources

OpenAI, "Testing Agent Skills Systematically with Evals": https://developers.openai.com/blog/eval-skills
OpenAI, "Moving from OpenAI Evals to Promptfoo": https://developers.openai.com/cookbook/examples/evaluation/moving-from-openai-evals-to-promptfoo
OpenAI, "Build an Agent Improvement Loop with Traces, Evals, and Codex": https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop
Companion: Evals and Agent Memory — Codex Memories and Hermes Agent