How Evals Meet Agent Memory: Codex Memories and Hermes Agent's “Grows With You”

Research date: Aug 31, 2026. Primary sources: official Codex memory documentation, the Hermes Agent README (Nous Research), Mem0's technical writeup on Hermes memory, and the cited memory benchmarks. For the daily workflow this study supports, see Progressive, Uninterrupted Eval Building.

TL;DR

Memory and evals are the two halves of the same growth loop:

Memory accumulates — facts, preferences, session history, reusable skills.
Evals verify — whether the accumulated knowledge actually makes the agent better.

Hermes Agent's claim to be "the agent that grows with you" is real in mechanism — persistent memory files, full-text session search, autonomous skill creation and improvement, user modeling — but growth without measurement is vibes. An eval harness is the instrument that turns "the agent remembers more" into "the agent is measurably better and did not regress." The two systems feed each other: memory supplies eval cases, eval results become memory, and the memory system itself is a thing to evaluate.

Part 1 — The two memory systems studied

Codex local memories

Official documentation (https://developers.openai.com/codex/memories): Codex can convert useful context from eligible prior chats into local memory files under ~/.codex/memories/, including summaries, durable entries, recent inputs, and supporting evidence.

Key design facts:

Background generation: memories are generated after a chat has been idle long enough, not at session end; short-lived or active sessions are skipped.
Redaction: secrets are redacted from generated memory fields.
Chat-level control: /memories in an interactive session decides whether the chat can use existing memories or contribute to future ones.
Config flags: memories.generate_memories, memories.use_memories, memories.extract_model, memories.consolidation_model, and rate-limit guards for the background pass.
Role separation: the docs are explicit that memories are a helpful recall layer, not the source of truth — required team guidance belongs in AGENTS.md or checked-in documentation.

Hermes Agent (Nous Research)

Hermes is an open-source personal AI agent built by Nous Research, branded as "the self-improving AI agent" and "the agent that grows with you" (https://github.com/NousResearch/hermes-agent). Its learning loop, per the README: creates skills from experience, improves them during use, nudges itself to persist knowledge, searches its own past conversations, and builds a deepening model of the user across sessions.

Memory architecture, per the README and Mem0's technical writeup (https://mem0.ai/blog/how-memory-works-in-hermes-agent-(and-how-to-improve-it)):

Layer	What it is	Detail
Working memory	Current session context	Rendered into the system prompt
Long-term curated memory	`MEMORY.md` (2,200 char cap) + `USER.md` (1,375 char cap)	Short declarative entries separated by `§`; a usage gauge is shown to the model; at ~80% the agent must consolidate before adding more
Session archive	SQLite `state.db` with FTS5 full-text search	Every CLI session logged; a `session_search` tool retrieves and summarizes past transcripts on demand
Procedural memory	Skills system	Agent-created skills from experience, self-improved during use; compatible with the agentskills.io open standard
User model	Honcho dialectic user modeling	Optional external layer

Two deliberate design choices matter for eval work:

Frozen snapshot for prefix caching: at session start, MEMORY.md and USER.md are pasted into the system prompt and frozen for the rest of the session. Writes hit disk immediately but only become visible to the model next session. The trade is stable prompt prefixes (cheap cached inference on long sessions) at the cost of not seeing your own newest memory mid-session.
Budget forces curation: a tiny character budget means memory writes are a scarce operation; the model must consolidate before adding. That is effectively a built-in write-quality pressure, but it is not measured — nothing scores whether consolidation lost or corrupted information.

External memory providers (Mem0, LanceDB, TencentDB Agent Memory) bolt on unbounded, semantic, per-user, cross-machine storage. Mem0's integration, for example, adds server-side fact extraction in a background thread, semantic search, and per-user isolation.

Part 2 — The eval-memory interface: four directions

1. Memory → eval cases

Memory is the raw material for a golden set. Every session archived by Hermes (state.db) or summarized by Codex memories is a candidate source of eval cases:

Failures: a session where the agent got something wrong is a case with the corrected behavior as the expected outcome.
Corrections: a user correction is the highest-quality case source — the user literally specified the expected behavior.
Recurring topics: if session search shows the same question or failure three times, that deserves a stable case.

In the daily workflow (companion page), this is the capture pipe: instead of manually remembering "we had that bug last Tuesday," the agent can session_search for it, summarize the transcript, and propose JSONL cases. Codex memories produce chat summaries that are likewise worth reviewing for case-worthy misses — but only after human review, because generated summaries are not ground truth.

2. Eval results → memory

The reverse direction is just as important:

Lessons learned: regression findings ("the billing prompt regressed after the model swap") become durable memory entries that the agent injects into future sessions.
Rules that must always apply: eval findings that are now policy belong in AGENTS.md, not memory — the official Codex docs make exactly this point.
Procedural memory: the eval harness itself is a skill. Hermes' skills system is procedural memory: "how to run the smoke suite, grade with the rubric, and append cases" becomes a skill the agent can invoke and improve.

This is where "grows with you" becomes concrete: the agent remembers what it learned from evals and encodes how to run evals as a skill — as long as a human reviews the promotion of any memory or skill change.

3. Evals of the memory system itself

A memory system is an AI component and needs its own eval dimensions:

Dimension	Question	How to measure
Write quality	Does it extract the right facts, dedupe, and avoid contradictions?	Precision/recall against human-curated facts; contradiction-detection suites; duplicate rate
Retrieval	Does it return the right memories when needed?	Recall@k, MRR/nDCG on benchmark sessions
Consolidation	Does compression lose or corrupt information?	Round-trip fidelity, conflict resolution, fact retention after merge
Application	Does memory actually improve task outcomes?	Task accuracy with memory on/off; token savings vs full-context
Safety	Can injected content poison memory?	Injection scan coverage, redaction checks, provenance tracking

Public benchmarks in this space:

LoCoMo — very long-term conversational memory: multi-session conversations (~300 turns / ~9K tokens on average per conversation), question answering, event summarization, and multi-modal dialogue generation (https://arxiv.org/abs/2402.17753).
LongMemEval (ICLR 2025) — long-term memory retrieval across five categories, with the S variant holding ~48 sessions per question (~115K tokens each) (https://arxiv.org/abs/2410.10813).
MemoryAgentBench — incremental multi-turn memory testing; four core competencies including accurate retrieval, updates, and conflict resolution (https://arxiv.org/abs/2507.05257).
AMA-Bench — long-horizon memory in realistic agentic settings with a causality graph; the strongest system (AMA-Agent) reaches ~57% accuracy, ~11 points above baselines (https://arxiv.org/abs/2602.22769).
AgentMemBench — taxonomy of five memory-management strategy families, scoring both quality and efficiency (https://arxiv.org/abs/2608.00009).
Mem0 memory-benchmarks — an open-source suite that runs LoCoMo, LongMemEval, and related benchmarks across memory backends (https://github.com/mem0ai/memory-benchmarks).

Reported results also quantify the efficiency argument: memory-augmented retrieval can cost on the order of ~7K tokens per retrieval call on LoCoMo versus ~26K for full-context injection, which is the economic reason memory systems exist (Mem0, "State of AI Agent Memory 2026": https://mem0.ai/blog/state-of-ai-agent-memory-2026).

For a personal agent like Hermes, the practical version of "eval the memory" is a periodic audit skill: export MEMORY.md/USER.md, run duplicate and contradiction checks, spot-check retrieval with questions from real sessions, and re-score a small fixed set of "does the agent remember last week's facts" questions.

4. Trust and safety at the interface

Hermes scans memory writes for injection attacks and data-leakage patterns before persisting (README security notes; Mem0 writeup).
Codex redacts secrets from generated memory fields, but the official docs still tell users to review memory files before sharing.
The blog's existing memory pattern (companion page) adds a human gate: agents propose changes to rules/decisions/lessons; humans approve. Raw transcripts never auto-promote to instructions.

Eval cases themselves are a security surface: a golden set captures real prompts, so it must never contain credentials or customer data, and its expected outputs must be treated as untrusted until reviewed.

Part 3 — Hermes: claim vs. mechanism

What "grows with you" actually is. The README describes a closed learning loop: the agent creates skills after complex tasks, improves skills during use, nudges itself to persist knowledge, searches its own conversation history, and models the user across sessions. It also exposes "research-ready" batch trajectory generation and trajectory compression for training future tool-calling models. So the growth is a combination of:

accumulation (memory entries, session archive, skills),
curation (budget-forced consolidation, skill improvement),
recall (FTS5 session search, semantic providers),
user modeling (Honcho),
data flywheel (trajectory generation).

Where evals fit. None of that is self-verifying. The same evaluation gap that hits enterprises — half of surveyed organizations shipped an agent that passed internal evals and then failed a customer (VentureBeat Pulse research, July 2026) — applies a fortiori to a personal agent with no eval gate at all. Concretely, for a Hermes-style setup:

Make eval-running a skill: "run the smoke suite, grade with the rubric, report per-category scores."
Let memory accumulate cases: every correction and failure becomes a candidate JSONL row, distilled weekly with human approval.
Track a score history: growth is only defensible if the suite score trends up and regressions are caught.
Use trajectory generation for eval data: batch trajectories from real sessions can seed the golden set, with human review.

Honest limits.

"Grows with you" is mostly accumulation and curation within fixed budgets. A 2,200-character memory is deliberately lossy; consolidation is a compression step with no built-in fidelity score.
Cross-session consistency is only as good as retrieval. Substring-match addressing works at ten entries and degrades at fifty; that is precisely why Hermes ships external semantic providers.
Memory growth is not competence growth. An agent that remembers your preferences can still regress on the actual task — which is why the eval suite, not the memory file, is the scoreboard.

Sources

OpenAI, "Memories" (official Codex/ChatGPT documentation): https://developers.openai.com/codex/memories
Nous Research, Hermes Agent README: https://github.com/NousResearch/hermes-agent
Mem0, "How Memory works in Hermes Agent (and how to improve it)": https://mem0.ai/blog/how-memory-works-in-hermes-agent-(and-how-to-improve-it)
LoCoMo: https://arxiv.org/abs/2402.17753
LongMemEval: https://arxiv.org/abs/2410.10813
MemoryAgentBench: https://arxiv.org/abs/2507.05257
AMA-Bench: https://arxiv.org/abs/2602.22769
AgentMemBench: https://arxiv.org/abs/2608.00009
Mem0 memory-benchmarks suite: https://github.com/mem0ai/memory-benchmarks
Mem0, "State of AI Agent Memory 2026": https://mem0.ai/blog/state-of-ai-agent-memory-2026
VentureBeat Pulse, "The agent evaluation gap" (July 2026), via Predictive Analytics World
Companion: Progressive, Uninterrupted Eval Building