Coding Agent CLIs: Is the Loop Just the Loop? — Benchmarks, Implementations, and What Actually Differs
Research date: Aug 2, 2026 · All facts linked to verified URLs · Scope: CLI agents (Codex, Gemini CLI, OpenCode, Claude Code, Terminus 2, etc.)
Two questions drive this review:
- Is there a benchmark that tests the agent CLI only (not the LLM), e.g. Gemini CLI vs OpenCode vs Codex on the same coding problem with the same model?
- Is the basic iteration loop (user query → LLM query → tool call → observe → repeat) the same across all coding agent CLIs, making the basic use case not worth comparing?
Short answers (full evidence below): 1. No official benchmark isolates the CLI from the model — every leaderboard scores agent+model combinations. But Terminal-Bench's leaderboard happens to contain same-model rows across CLIs, its harness (Harbor) makes a controlled CLI-only run possible, and community tests have pinned the same model across CLIs (up to ~30x token spread, 2x+ task-performance gaps). 2. Yes — every official source (OpenAI, Anthropic, Google) documents the same loop: prompt+context → model inference → parse tool calls → execute tools → append results → re-query → repeat. Differences live in the harness engineering around the loop (context management, sandboxing, tool schemas, token economics), not in the loop itself.
Part 1 — Benchmark Landscape: Is There a CLI-Only Benchmark?
1.1 Verdict: no, none exists as a maintained benchmark
Every major leaderboard scores the combination of an agent (harness) + a model, and submitters choose both — so scores cannot be attributed to the CLI alone:
| Benchmark | URL | What it measures | CLI isolated? |
|---|---|---|---|
| Terminal-Bench (TB) 2.x | https://www.tbench.ai/ · repo https://github.com/harbor-framework/terminal-bench | agent + model pairs on 89 realistic terminal tasks (v2.0); rank, agent, model, effort, accuracy, cost | No — model and effort are self-chosen per submission |
| SWE-bench Verified | https://www.swebench.com/ | agent + model combos, % resolved, agent dropdown | No |
| Terminal-Bench 3.0 (ex-Frontier-Bench) | https://frontierbench.ai | same agent+model ranking model | No |
Note: the canonical Terminal-Bench repo moved from swe-bench/Terminal-Bench (404) to harbor-framework/terminal-bench; the site is tbench.ai, not terminal-benchmark.com (dead).
1.2 The closest thing: same-model rows on the Terminal-Bench leaderboard
Because TB allows any agent to be submitted with any model, the official leaderboard contains rows where the same model is run through different CLIs (uncontrolled — different dates/effort, so informal):
| Model (held same) | CLI A | CLI B | Gap (TB 2.1) |
|---|---|---|---|
| GPT-5.5 | Codex CLI 83.1% | Terminus 2 78.0% | +5.1 |
| Gemini 3 Pro | Terminus 2 73.9% | Gemini CLI 65.8% | +8.1 |
| Opus 4.7 | Claude Code 68.9% | Terminus 2 66.1% | +2.8 |
TB 2.0, same model across 6 CLIs (Claude Opus 4.5): Droid 63.1 / Letta Code 59.1 / Terminus 2 57.8 / Goose 54.3 / Claude Code 52.1 / OpenHands 51.9 / OpenCode 51.7. GPT-5.2: Droid 64.9 / Codex CLI 62.9 / Mux 60.7 / Terminus 2 54.0. (Source: leaderboard https://www.tbench.ai/leaderboard/terminal-bench/2.1 and /2.0)
Takeaway: the harness demonstrably matters — same model, spread of ~5–10 points across CLIs — but this is not a controlled benchmark.
1.3 The enabler: Harbor
Harbor (https://github.com/harbor-framework/harbor, docs https://harborframework.com/docs/agents) is the harness behind TB and accepts any agent/model combination: harbor run --dataset X --agent <cli> --model <same-model>. It supports Terminus-2, Claude Code, Codex CLI, Gemini CLI, Grok Build, OpenHands, Mini-SWE-Agent, plus custom agents — i.e. the CLI-only methodology doesn't exist as a leaderboard, but it is runnable yourself with one command.
1.4 The mirror image: holding the harness constant (mini-SWE-agent)
mini-SWE-agent (https://mini-swe-agent.com/latest/) was built so the model is the variable: "a system that puts the language model (rather than the agent scaffold) in the middle of our attention" on SWE-bench (bash only). Its SWE-bench leaderboard ranks models within a fixed harness — the exact inverse of a CLI-only benchmark. Implication: the field standardized on "fix harness, vary model," and never shipped the opposite.
1.5 Academic studies with model held constant (scaffold ablations)
| Study | URL | Finding |
|---|---|---|
| SWE-Search (Meta) | https://arxiv.org/abs/2410.20285 | MCTS scaffolding gives ~23% relative improvement vs a standard agent "across five models" |
| icat-agent | https://arxiv.org/abs/2606.25514 | Adaptive multi-agent scaffolding beats SWE-agent, mini-SWE-agent and Claude Code "while using the same underlying models" (+3.6–8.4% SWE-bench Verified) |
| Confucius Code Agent | https://arxiv.org/abs/2512.10398 | 59% on SWE-bench Pro "under identical repositories, model backends, and tool access" |
| ContextBench | https://arxiv.org/abs/2602.05892 | 4 frontier LLMs × 5 coding agents: "sophisticated agent scaffolding yields only marginal gains in context retrieval" — the "Bitter Lesson of coding agents" |
| Failure as a Process | https://arxiv.org/abs/2607.09510 | 3,843 trajectories, 7 models × 3 scaffolds on Terminal-Bench; failure modes are scaffold-dependent |
| Meta-Harness artifact | https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact | single evolved harness: 76.4% on TB 2.0 with Claude Opus 4.6 — harness choice alone is worth ~15+ points at fixed model |
These are one-off studies, not maintained benchmarks — but they are the only controlled "harness-only" data that exists.
1.6 Community same-model tests (informal but controlled)
| Test | URL | Setup | Result |
|---|---|---|---|
| systima.ai CLI benchmark | https://systima.ai/blog/claude-code-vs-opencode-token-overhead (+ repo https://github.com/systima-ai/agentic-coding-tools-comparison) | Claude Code vs OpenCode, same model pinned (claude-sonnet-4-5, re-run on Fable 5), same machine, same tasks |
Both pass every scored task (identical outcomes); but 3.7x more tokens per passing run for Claude Code (~33K vs ~7K system prompt+schemas; 27 tools/99,778 chars vs 10 tools/20,856 chars); costs converge on multi-step tasks because Claude Code batches parallel tool calls while OpenCode does one per turn |
| HN: token bloat across 6 harnesses | https://news.ycombinator.com/item?id=49135257 (details: /item?id=49136112) | Same model (GPT 5.6), 10 identical agentic tasks in a VM, 6 harnesses | Claude Code 5.07M tokens … OpenCode 1.56M … smol 173K — ~30x spread from the harness alone, same model, same tasks |
| HN: same model forked across CLIs | https://news.ycombinator.com/item?id=46991089 | GLM-4.7 (same model) run through forked Codex CLI, Gemini CLI, Mistral Vibe, OpenCode, same benchmark | Mistral Vibe 0.35 vs Codex 0.15 — 2x gap, same model, attributed to 5 harness decisions: file-edit tools, command sandboxing, context management, error handling, cross-session memory |
| Ask HN: Claude Code vs OpenCode | https://news.ycombinator.com/item?id=46772249 (quote: /item?id=46773694) | Same model by assumption | "The output is 'identical' if you're using the same model. Consider them separate front ends really." — differences listed as UX-only (session forking, permissive tools, modes, model choice) |
1.7 Same-task comparisons, model NOT held constant (for context)
- DeployHQ: identical deployment tasks in Claude Code / Codex / Gemini CLI — https://www.deployhq.com/blog/comparing-claude-code-openai-codex-and-google-gemini-cli-which-ai-coding-assistant-is-right-for-your-deployment-workflow (Codex fastest, no questions; Claude Code most defensive; Gemini grounds via search)
- pikvue: 50 tasks on a 200+ file project, 90/79/76% pass rates — https://pikvue.com/claude-code-vs-codex-cli-vs-gemini-cli-2026-best-ai-coding-agent-for-terminal-compared/ — quote: "For simple, single-file tasks, all three perform similarly"; differences concentrate on complex multi-file tasks (low-rigor blog; treat numbers as illustrative)
- Reddit SWE-bench on a real Rails codebase: https://old.reddit.com/r/ClaudeAI/comments/1qxr7vs/gpt53_codex_vs_opus_46_we_benchmarked_both_on_our/ — GPT-5.3+Codex ~0.70 quality at <\(1/ticket vs Opus 4.6 ~0.61 at ~\)5/ticket (model+CLI confounded)
Part 2 — Implementation Analyses: Is the Basic Loop the Same?
2.1 Yes — the canonical loop is identical across all CLIs
Every official source describes the same structure, which is a direct descendant of ReAct (https://arxiv.org/abs/2210.03629 — thought → action → observation, interleaved):
user query → prompt+context assembly
→ model inference (with tool schemas)
→ parse tool calls
→ execute tools (sandboxed/approved)
→ append results to context
→ re-query → repeat
→ until assistant emits a final message / exit condition
| Source | URL | What it says |
|---|---|---|
| OpenAI — "Unrolling the Codex Agent Loop" (M. Bolin, Jan 2026) | https://openai.com/index/unrolling-the-codex-agent-loop/ | "At the heart of every AI agent is something called 'the agent loop'": input → prompt (instructions/tools/input layers) → Responses API inference → tool call → execute → append output → re-query → repeat until assistant message ends the turn. Statelessness, prompt caching via exact prefix, auto-compaction (/responses/compact) |
| OpenAI — "A Practical Guide to Building Agents" (official PDF) | https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf | "Every orchestration approach needs the concept of a 'run,' typically implemented as a loop"; "This concept of a while loop is central to the functioning of an agent"; exit conditions = final-output tool, no-tool-call response, max turns |
| Anthropic — "Building effective agents" | https://www.anthropic.com/research/building-effective-agents | Agents "are typically just LLMs using tools based on environmental feedback in a loop"; augmented-LLM building block; ground truth from tool-call results at each step |
| Anthropic — "Getting started with loops" (official) | https://claude.com/blog/getting-started-with-loops | Loops = "agents repeating cycles of work until a stop condition is met"; basic agentic loop = gather context → act → check → repeat |
| Anthropic — "How Claude Code works" (official docs) | https://code.claude.com/docs/en/how-claude-code-works | Same agentic loop; Claude Code as an "agentic harness"; auto-compaction; permissions/checkpoints instead of OS sandboxing |
| Google — Gemini CLI architecture | https://google-gemini.github.io/gemini-cli/docs/architecture.html | input → prompt construction → API → tool request → user approval for mutating tools → execute → result back to API → final response; conversation-state management |
| OpenAI — "Harness engineering" | https://openai.com/index/harness-engineering/ | The loop in practice ("Ralph Wiggum loop"), context and AGENTS.md design |
| Simon Willison — "Designing agentic loops" | https://simonwillison.net/2025/Sep/30/designing-agentic-loops/ | Coding agents = tools in a loop; differences are sandboxing / YOLO mode / permission policies, not the loop |
| Simon Willison — guide chapter "How coding agents work" | https://simonwillison.net/guides/agentic-engineering-patterns/how-coding-agents-work/ | "LLM + system prompt + tools in a loop… that's most of what it takes"; a basic tool loop is "a few dozen lines of code" |
| ReAct (academic ancestor) | https://arxiv.org/abs/2210.03629 | Interleaved reasoning + acting |
| Reflexion (loop + memory) | https://arxiv.org/abs/2303.11366 | Act → observe → reflect → episodic memory |
| Gergely Orosz — "How Codex is built" | https://newsletter.pragmaticengineer.com/p/how-codex-is-built | The agent loop is "something every AI agent uses, not just Codex"; covers compaction, sandboxing-by-default (deliberate adoption tradeoff), Rust vs Claude Code's TS |
| ByteByteGo — "How OpenAI Codex works" | https://blog.bytebytego.com/p/how-openai-codex-works | "The model is a component, the agent is the system"; loop, quadratic prompt growth, caching fragility |
| Mario Zechner — Pi coding agent | https://mariozechner.at/posts/2025-11-30-pi-coding-agent/ | A <1K-token prompt + 4-tool harness was competitive on TB 2.0 vs Claude Code/Codex with native models — the "model does the work" hypothesis; also documents OpenCode's prompts are cut-down copies of Claude Code's |
| claude-trace (reverse engineering) | https://simonwillison.net/2025/Jun/2/claude-trace/ | HTTP interception reveals the dispatch_agent subagent tool — i.e. a nested second loop (subagent with fresh context) |
⚠️ Fact-check of popular claims: "The Boring Agent Loop" is not a real Simon Willison essay (404 on the guessed URL; his actual essays are the two above). "XCodeAgent" (arXiv 2502.13157) does not exist as a paper (that ID is a biostatistics paper); the correct XCode context is xCodeEval (https://arxiv.org/abs/2303.03004). The "Commander" benchmark does not exist (commanderbenchmark.com unreachable). The claimed paper "A Large-Scale Study on the Effect of Agent Scaffolding" (arXiv 2505.19414) returns zero results — the real scaffold study is SWE-Search (https://arxiv.org/abs/2410.20285).
2.2 Where the harness differs around the identical loop
| Layer | Codex CLI | Claude Code | Gemini CLI | OpenCode |
|---|---|---|---|---|
| Model binding | Responses API-bound (but --oss) |
Anthropic-bound (multi-model within Anthropic) | Multiple backends | Fully provider-agnostic (any provider via config) |
| Context management | /responses/compact, ZDR constraints |
auto-compaction, subagent context isolation | 1M-token window + caching | compaction agent, session/thread model |
| Sandboxing | OS sandbox (macOS sandbox-exec; Windows sandbox) | permission modes + checkpoints (no OS sandbox) | per-tool user approval + --sandbox runtime |
permission rules (ask/allow/deny) |
| Terminal emulation | plain subprocess Bash tool | plain subprocess Bash tool | node-pty PTY with terminal-state serialization (interactive commands) | plain subprocess Bash tool |
| Tool protocol | Responses API function calling; custom JSON-RPC App Server (MCP "too thin") | Anthropic tool-use blocks; MCP | MCP | MCP first-class |
| System prompt payload | ~8.5K tokens (measured) | ~6–10K tokens | ~10–13K tokens | small (cut-down Claude Code prompt); <1K for Pi |
| Tool surface | measured 27 tools / 99,778 chars schemas (Claude Code) vs 10 tools / 20,856 chars (OpenCode) — systima.ai rig | — | — | — |
Sources: OpenAI harness posts https://openai.com/index/unrolling-the-codex-agent-loop/ · https://openai.com/index/unlocking-the-codex-harness/ · https://openai.com/index/introducing-codex/ · Gemini PTY post https://developers.googleblog.com/en/say-hello-to-a-new-level-of-interactivity-in-gemini-cli/ · OpenCode docs https://opencode.ai/docs/ + https://opencode.ai/docs/agents/ · system-prompt dissection https://codex.danielvaughan.com/2026/04/19/system-prompts-compared-codex-gemini-claude-code/ · capability matrix https://hidekazu-konishi.com/entry/cli_coding_agents_comparison.html · wren.wtf critique https://wren.wtf/shower-thoughts/stop-using-opencode/
System-prompt "DNA" differences (from the dissection post): Gemini = proactive autonomy ("fulfill implied follow-up actions"); Claude Code = restraint ("don't add features beyond what was asked"); Codex = per-model prompts + anti-slop rules. All converge on: anti-verbosity, context-file hierarchies (AGENTS.md / CLAUDE.md / GEMINI.md), tool-over-shell, no auto-push.
2.3 Where the loops differ: the subagent / nested loop
The only structural loop difference found: Claude Code adds a nested dispatch_agent subagent loop (fresh context), OpenCode has build/plan primary agents + subagents via a Task tool with steps (max agentic iterations before forced text-only response), Codex's harness has thread/turn/item primitives with approval pauses (https://openai.com/index/unlocking-the-codex-harness/). These are hierarchy additions, not a different base loop.
Part 3 — Executive Summary
Q1: Is there a benchmark testing the CLI only? — No official one; the field never shipped it.
- All maintained leaderboards (Terminal-Bench, SWE-bench Verified, Frontier-Bench/TB3) score agent+model combinations; the mirror-image standard (mini-SWE-agent) holds the harness fixed to test models.
- Best available evidence that the CLI matters at fixed model: (a) uncontrolled same-model rows on the TB leaderboard (~3–8 point spreads across CLIs), (b) scaffold-ablation papers holding models constant (SWE-Search +23% relative; Meta-Harness +~15 points at fixed model), (c) community tests pinning the same model across CLIs (2x task-performance gap; up to 30x token spread).
- If you want a true CLI-only comparison today: run Harbor yourself —
harbor run --dataset [email protected] --agent <cli> --model <same-model>(https://harborframework.com/docs/agents).
Q2: Is the basic iteration loop the same? — Yes, structurally identical.
- The loop (prompt → inference → tool call → execute → append → repeat) is documented identically by OpenAI, Anthropic, and Google, descends from ReAct, and no source documents a different base loop.
- "The output is 'identical' if you're using the same model" (HN) and "both harnesses passed every scored task" (systima, same model) hold for basic/single-file use cases; pikvue: "For simple, single-file tasks, all three perform similarly."
- So: for the basic use case, you do not need to compare — the loop is the loop. What differs and is worth comparing at fixed model: token economics (3.7x–30x), system-prompt/tool-schema size (1K→13K tokens, 10 vs 27 tools), caching stability, batching behavior (parallel vs serial tool calls), sandboxing quality, permission friction, and proactiveness (Gemini's "fulfill implied follow-ups" vs Claude Code's restraint).
- Differences concentrate on complex multi-file tasks and failure modes (Failure as a Process: failures are scaffold-dependent), not the naive query.
Bottom line
Same model, same loop, same answers to simple questions — but the harness around the loop is worth 5–15 points on hard terminal tasks and 2x–30x on tokens. For the naive use case: don't benchmark the CLIs, pick by UX, cost, and model access. For hard tasks: benchmark with the model pinned (Harbor makes this easy), because the scaffold is a first-class variable.
References (all verified)
Benchmarks & harnesses - Terminal-Bench: https://www.tbench.ai/ · https://github.com/harbor-framework/terminal-bench · https://www.tbench.ai/leaderboard/terminal-bench/2.1 · https://frontierbench.ai - Harbor: https://github.com/harbor-framework/harbor · https://harborframework.com/docs/agents - SWE-bench Verified: https://www.swebench.com/ · mini-SWE-agent: https://mini-swe-agent.com/latest/ - SWE-Search: https://arxiv.org/abs/2410.20285 · icat-agent: https://arxiv.org/abs/2606.25514 · Confucius Code Agent: https://arxiv.org/abs/2512.10398 · ContextBench: https://arxiv.org/abs/2602.05892 · Failure as a Process: https://arxiv.org/abs/2607.09510 · Meta-Harness: https://github.com/stanford-iris-lab/meta-harness-tbench2-artifact
Agent-loop analyses - https://openai.com/index/unrolling-the-codex-agent-loop/ · https://openai.com/index/unlocking-the-codex-harness/ · https://openai.com/index/introducing-codex/ · https://openai.com/index/harness-engineering/ · https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf - https://www.anthropic.com/research/building-effective-agents · https://claude.com/blog/getting-started-with-loops · https://code.claude.com/docs/en/how-claude-code-works - https://google-gemini.github.io/gemini-cli/docs/architecture.html · https://developers.googleblog.com/en/say-hello-to-a-new-level-of-interactivity-in-gemini-cli/ - https://opencode.ai/docs/ · https://opencode.ai/docs/agents/ - https://simonwillison.net/2025/Sep/30/designing-agentic-loops/ · https://simonwillison.net/guides/agentic-engineering-patterns/how-coding-agents-work/ · https://simonwillison.net/2025/Jun/2/claude-trace/ - https://newsletter.pragmaticengineer.com/p/how-codex-is-built · https://blog.bytebytego.com/p/how-openai-codex-works · https://mariozechner.at/posts/2025-11-30-pi-coding-agent/ - ReAct: https://arxiv.org/abs/2210.03629 · Reflexion: https://arxiv.org/abs/2303.11366
Same-model community comparisons - https://systima.ai/blog/claude-code-vs-opencode-token-overhead · https://github.com/systima-ai/agentic-coding-tools-comparison - https://news.ycombinator.com/item?id=49135257 · https://news.ycombinator.com/item?id=46991089 · https://news.ycombinator.com/item?id=46772249 · https://news.ycombinator.com/item?id=46773694 - https://www.deployhq.com/blog/comparing-claude-code-openai-codex-and-google-gemini-cli-which-ai-coding-assistant-is-right-for-your-deployment-workflow - https://pikvue.com/claude-code-vs-codex-cli-vs-gemini-cli-2026-best-ai-coding-agent-for-terminal-compared/
Capability/system-prompt analyses - https://codex.danielvaughan.com/2026/04/19/system-prompts-compared-codex-gemini-claude-code/ · https://hidekazu-konishi.com/entry/cli_coding_agents_comparison.html · https://wren.wtf/shower-thoughts/stop-using-opencode/
Official repos - https://github.com/openai/codex · https://github.com/google-gemini/gemini-cli (→ Antigravity CLI, https://developers.googleblog.com/an-important-update-transitioning-gemini-cli-to-antigravity-cli/) · https://github.com/anomalyco/opencode · https://github.com/anthropics/claude-code