lagindicator

Same Model, Different Behavior: What Coding Agent CLIs Change Around the LLM

Agent Dev Tools Updated: September 5, 2026 Research date: August 2, 2026

Research article on why the same LLM behaves differently in different coding agent CLIs. All URLs verified; prompt quotes from official repos.

A research article on why the same LLM behaves differently in different coding agent CLIs. Research date: Aug 2, 2026. All URLs verified; prompt quotes are from the actual prompt files in the official repos.

TL;DR

Run the same frontier model in Codex CLI, Claude Code, Gemini CLI, or OpenCode and you get materially different behavior — different levels of proactiveness, different verbosity, different testing habits, different willingness to plan. The agent loop is identical (see the companion article agent-loop-pattern.md); what differs is everything the harness puts around the model:

  1. The system prompt — content (identity, autonomy stance, verbosity rules, risk frameworks) and size (from ~350 tokens in Pi to ~6,000+ in Gemini CLI).
  2. Per-model prompts — Codex ships a different prompt per model generation; OpenCode ships 14 model-specific prompt files.
  3. Memory injection — AGENTS.md / CLAUDE.md / GEMINI.md hierarchies, auto-memory, skills, environment context.
  4. Inference parameters — thinking-token budgets and reasoning effort are harness-set knobs that swing same-model benchmark scores by 5-10+ points.
  5. Sub-agent orchestration — hidden dispatch_agent loops (Claude Code), named sub-agents (Gemini CLI), child sessions (OpenCode), or none (Pi).
  6. Context management mandates — whether the prompt tells the model to treat context as a precious resource, and how compaction/exit conditions are enforced.

Evidence is both direct (people editing one line of a system prompt and changing the behavior of the same model) and quantitative (benchmarks holding the model constant and varying only the harness).

Part 1 — Evidence: the harness changes the model’s behavior

1.1 Direct evidence: prompt edits change same-model behavior

Gemini CLI — one line changes everything. Google Developer Advocate Daniela Petruzalek documented that the Gemini CLI system prompt contains the line Proactiveness: Fulfill the user's request thoroughly, including reasonable, directly implied follow-up actions. — and identified it as “the source of 80% of my problems”: the same model would delete files when asked a rhetorical question (“why did you add this file?”). She fixed the behavior by editing the prompt file and overriding the system prompt via the undocumented GEMINI_SYSTEM_MD environment variable (https://danicat.dev/posts/20250715-gemini-cli-system-prompt/). Same model, different prompt line → different behavior. (Notably, the current Gemini CLI prompt in the repo has since rewritten this mandate to “Take reasonable liberties to fulfill broad goals while staying within the requested scope” — the prompt evolves, and behavior evolves with it: https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/prompts/snippets.ts)

Claude Code — prompt churn changes model behavior. Mario Zechner (creator of the Pi agent), who prefers running Claude models, complains: “The system prompt and tools also change on every release, which breaks my workflows and changes model behavior” (https://mariozechner.at/posts/2025-11-30-pi-coding-agent/). He maintains cchistory, a version-diff tool for Claude Code’s prompt: https://cchistory.mariozechner.at.

Anthropic’s own numbers — orchestration alone is worth 90%. In building Claude’s Research feature, Anthropic found that a multi-agent system with Claude Opus 4 as lead agent + Claude Sonnet 4 subagents outperformed single-agent Claude Opus 4 by 90.2% on their internal research eval — same model family, different harness architecture. They also report that “a prompt tweak might boost success rates from 30% to 80%,” that token usage explains 80% of performance variance, and that “small changes to the lead agent can unpredictably change how subagents behave” (https://www.anthropic.com/engineering/built-multi-agent-research-system).

1.2 Quantitative evidence: benchmarks with the model held constant

Aider polyglot leaderboard — harness settings move same-model scores. Aider’s benchmark holds the harness fixed and varies models; but rows for the same model under different harness-set parameters show the lever sizes (https://aider.chat/docs/leaderboards/):

Same model, different harness settingsPass rateWell-formed edits
claude-sonnet-4, 32k thinking tokens61.3%97.3%
claude-sonnet-4, no thinking56.4%98.2%
gemini-2.5-pro-06-05, 32k thinking83.1%99.6%
gemini-2.5-pro-06-05, default thinking79.1%100%
gpt-5, reasoning high88.0%91.6%
gpt-5, reasoning medium86.7%88.4%
gpt-5, reasoning low81.3%86.7%
o3, reasoning high81.3%94.7%
o3, default76.9%93.8%
o3 (high) + gpt-4.1 architect/editor split78.2%100%

Thinking-token budgets and reasoning-effort are harness-controlled parameters (the CLI decides them, or the user via CLI flags). Same model, different settings → 5-15 point swings, plus visible edit-format adherence differences.

SWE-Search — the same five models, different scaffold, +23%. SWE-Search applied an MCTS scaffold on top of standard open-source agents and demonstrated “a 23% relative improvement in performance across five models compared to standard open-source agents without MCTS” (https://arxiv.org/abs/2410.20285).

mini-swe-agent — >74% SWE-bench Verified with a 100-line agent and no tools besides bash. The SWE-bench team’s minimal agent deliberately has “no tools other than bash — it doesn’t even need to use the tool-calling interface of the LMs,” a completely linear history, and subprocess.run for every action. The README frames it as “a system that puts the language model (rather than the agent scaffold) in the middle of our attention” — and it still scores >74% on SWE-bench Verified and “beats Claude Code and Codex on DeepSWE” (https://github.com/SWE-agent/mini-swe-agent). The same team’s original SWE-agent paper had claimed the interface (tool design) was the key to agent performance; mini is the explicit retraction of that claim as models improved (https://arxiv.org/abs/2405.15793).

Terminal-Bench — minimalism is competitive. Pi (a ~350-token system prompt, four tools, zero safety rails) ran a Terminal-Bench 2.0 submission with Claude Opus 4.5 and placed competitively against Codex, Cursor, Windsurf and others (https://www.tbench.ai/, results: https://gist.github.com/badlogic/f45e8f6e481e5ab7d3a50659da84edaa). Terminal-Bench’s own Terminus 2 agent — which just hands the model a tmux session, no file tools at all — “holds its own against agents with far more sophisticated tooling” (Zechner, https://mariozechner.at/posts/2025-11-30-pi-coding-agent/).

1.3 The prompts are public, and they differ measurably

The strongest resource comparing them directly is Daniel Vaughan’s dissection of Codex CLI vs Gemini CLI vs Claude Code vs Pi (https://codex.danielvaughan.com/2026/04/19/system-prompts-compared-codex-gemini-claude-code/), cross-checked against the actual source files:

PromptLocationEst. tokens
Codex CLI, GPT-5.1 full promptcodex-rs/core/gpt_5_1_prompt.md in https://github.com/openai/codex~6,000
Codex CLI, compact prompt (fine-tuned Codex models)gpt-5.2-codex_prompt.md~1,900
Gemini CLI, assembledhttps://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/prompts/snippets.ts~5,000-6,000
Claude Code, corebundled in npm @anthropic-ai/claude-code; not published by Anthropic~3,000-5,000
Pihttps://github.com/badlogic/pi-mono/blob/main/packages/coding-agent/src/core/system-prompt.ts~350
OpenCode, per model14 files in https://github.com/anomalyco/opencode/tree/dev/packages/opencode/src/session/prompt (anthropic.txt 8.2KB, gemini.txt 15.4KB, codex.txt 7.4KB, gpt.txt, kimi.txt, meta.txt, …)varies by model

Measured total initial payload (prompt + tool schemas + context, from actual API traces): Codex CLI ~8,497 input tokens for a minimal session (measured by Alex Fazio on Codex CLI v0.114.0, https://gist.github.com/alexfazio/359c17d84cb6a5af12bac88fa1db9770); Gemini CLI ~13,055 tokens for two trivial math queries (https://github.com/google-gemini/gemini-cli/issues/3784); Pi <1,000 tokens. Community collections of extracted prompts: https://github.com/asgeirtj/system_prompts_leaks (61.9k stars), https://github.com/badlogic/cchistory.

Part 2 — The levers, one by one

2.1 System prompt content: the persona

Identity lines from the prompts (verified):

CLIIdentityAutonomy stance
Codex CLI“You are GPT-5.1 running in the Codex CLI, a terminal-based coding assistant.”“Persist until the task is fully handled end-to-end within the current turn whenever feasible… assume the user wants you to make code changes”
Gemini CLI“You are Gemini CLI, an interactive CLI agent specializing in software engineering tasks.”“Fulfill the user’s request thoroughly” (older); today “Take reasonable liberties to fulfill broad goals while staying within the requested scope”
Claude Code“You are an interactive agent that helps users with software engineering tasks.”“Don’t add features, refactor code, or make ‘improvements’ beyond what was asked.”
Pi“You are an expert coding assistant operating inside pi, a coding agent harness.”No autonomy instructions at all — “delegated to the model’s training”

Behavioral consequences documented by users: Gemini CLI’s older proactiveness line made it perform “directly implied follow-up actions” — deleting files mid-question (https://danicat.dev/posts/20250715-gemini-cli-system-prompt/); Codex’s prompt fights “AI slop” explicitly (“Avoid collapsing into ‘AI slop’…”); Claude Code’s prompt enforces YAGNI (“Three similar lines of code is better than a premature abstraction”) and a structured risk taxonomy (“Measure twice, cut once”). Full philosophy comparison in https://codex.danielvaughan.com/2026/04/19/system-prompts-compared-codex-gemini-claude-code/.

Divergences with direct behavioral effect (per Vaughan’s comparison + primary files):

DimensionCodex CLIGemini CLIClaude CodePi
Verbatim vs concise outputtiered rules (tiny/medium/large changes)“<3 lines of text output per response”“If you can say it in one sentence, don’t use three.”silence (nothing to reinforce verbosity)
Testing stance“Don’t add tests to codebases with no tests”“ALWAYS search for and update related tests” + “Validation is the only path to finality”nothing in core promptnothing
Risk/approval behavioradapts to approval mode; dirty-worktree rules“Do not take significant actions beyond the clear scope… ask for confirmation first”“Executing Actions with Care” section; confirm destructive opsnone (YOLO by default)
Recovery rules—3-strike rule: after 3 failed fix attempts, stop and propose a different approach——
Context economy(harness-level compaction)“your own context window is your most precious resource”; context-efficiency mandate with worked examplesauto-compactionno compaction at all (issue #92)

2.2 Per-model prompts

2.3 Memory and context-file hierarchies

CLIContext fileBehavior
Codex CLIAGENTS.md (cross-tool standard, 60,000+ repos, 25+ tools)hierarchical root→leaf; direct instructions override; 32 KiB cap
Claude CodeCLAUDE.md + auto-memory (.claude memory files)“Compact Instructions” section survives compaction
Gemini CLIGEMINI.md + hierarchical memory (global ~/.gemini → project → subdirs)context precedence order; cannot override Core Mandates
PiAGENTS.md; SYSTEM.md can fully replace the promptbecause the default prompt is tiny, a good SYSTEM.md “shifts the agent’s personality more dramatically than in any other tool”

Memory is a behavioral lever in both directions: project files teach conventions, and Gemini CLI even maintains a per-project private memory index (~/.gemini/tmp/<hash>/memory/MEMORY.md) that the agent can edit itself (visible in https://github.com/google-gemini/gemini-cli/blob/main/packages/core/src/prompts/snippets.ts). Portability matters too: AGENTS.md works across 25+ tools; CLAUDE.md/GEMINI.md lock you in.

2.4 Inference parameters (harness-set knobs)

  • Thinking-token budgets and reasoning effort move same-model scores by 5-15 points (aider data above). The CLI (or its config) decides defaults: Codex sets per-model defaults; Gemini CLI historically allowed --thinking-tokens; aider surfaces them as flags.
  • Different CLIs hitting the same provider API even send different request shapes: Codex uses the Responses API statelessly with exact-prefix prompts for cache hits; Pi’s pi-ai layer must paper over provider differences (thinking traces as <thinking> blocks, signed event-stream blobs, no tool-call streaming from Google) (https://mariozechner.at/posts/2025-11-30-pi-coding-agent/).

2.5 Sub-agent orchestration

CLIMechanismVisibility
Claude Codedispatch_agent tool, concurrent, stateless, read-only sub-agents; depth-limited“zero visibility into what that sub-agent does” (Zechner); prompts unpublished (https://simonwillison.net/2025/Jun/2/claude-trace/)
Gemini CLInamed sub-agents in the prompt (codebase_investigator, cli_help, generalist, browser_agent) + “strategic orchestrator” mandatedelegated work “consolidated into a single summary”
OpenCodesubagents = child sessions via Task toolsession files visible
Pinone — “no dedicated sub-agent tool”; spawning itself via bash is the workaroundfull output observability

Anthropic’s research post is the canonical argument that orchestration is a performance lever, not just an architecture choice: subagents “provide compression” and “separation of concerns,” and the multi-agent layout beat single-agent Opus 4 by 90.2% (https://www.anthropic.com/engineering/built-multi-agent-research-system).

2.6 Loop-level behavior mandates

  • Exit conditions: Pi’s agent loop has “no max steps or similar knobs… The loop just loops until the agent says it’s done” (https://mariozechner.at/posts/2025-11-30-pi-coding-agent/); OpenCode forces text-only summarization when steps is exhausted (https://opencode.ai/docs/agents/); Claude Code stops auto-compacting with a thrashing error to avoid infinite loops (https://code.claude.com/docs/en/how-claude-code-works).
  • Compaction philosophy: automatic summarization (Claude Code), automatic /responses/compact (Codex), hidden compaction agent (OpenCode), none (Pi — Zechner reports cramming “hundreds of exchanges” into one session without it).
  • Context-efficiency in the prompt itself: only Gemini CLI’s prompt teaches the model context economics (estimating-context-usage heuristics, parallel reads to save turns, wait_for_previous sequencing rules) — behavior shaping via instructions, not just harness mechanics.

Part 3 — Which levers matter most

LeverMagnitude of effectEvidence
System prompt content (autonomy/verbosity/testing stance)Highest, most user-visiblePetruzalek’s one-line fix; Vaughan’s philosophy table; prompt sources
Sub-agent orchestrationVery high on hard/long-horizon tasksAnthropic +90.2%; token-usage-80%-of-variance
Inference parameters (thinking budget)High (5-15 pts on same benchmark)aider leaderboard rows
Prompt size / token overheadMedium-high economics; critical for local 32K modelsFazio ~8.5K baseline; gemini-cli #3784 13K; Pi <1K
Per-model promptsMedium; avoids wasted tokens, tunes behaviorCodex compact prompts; OpenCode 14 files
Memory/context-file hierarchyMedium; compounds across sessionsAGENTS.md/CLAUDE.md/GEMINI.md ecosystems
Exit conditions/compactionLow-medium; mostly cost & robustnesspi issue #92; opencode steps

Practical takeaways for users: the fastest behavioral lever is the system-prompt override each tool exposes — Codex model_instructions_file (https://github.com/openai/codex/blob/main/codex-rs/config/src/config_toml.rs), Gemini GEMINI_SYSTEM_MD / GEMINI_WRITE_SYSTEM_MD (https://danicat.dev/posts/20250715-gemini-cli-system-prompt/), Claude Code CLAUDE_CODE_SIMPLE=1 (https://codex.danielvaughan.com/2026/04/19/system-prompts-compared-codex-gemini-claude-code/), Pi SYSTEM.md — plus writing AGENTS.md/CLAUDE.md/GEMINI.md, which every vendor ranks above their own defaults in precedence.

Caveats: no public benchmark isolates model × CLI in a controlled matrix (the earlier benchmark review reached the same conclusion: all leaderboards score agent+model combos). Pi’s Terminal-Bench run used a different model (Claude Opus 4.5) than its competitors, so it proves minimalism is viable, not that harness alone explains the gap. The cleanest model-held-constant evidence remains aider (settings), SWE-Search (+23% across five models), and Anthropic’s internal evals.

Conclusion

The loop is the same; the model can be the same; the CLI is the difference. Every coding agent CLI is a prompt + memory + orchestration + inference-settings wrapper around a general model, and each wrapper encodes a bet about what a good software engineer is: Codex bets on persistence and anti-slop aesthetics, Gemini on orchestration and validation (“Validation is the only path to finality”), Claude Code on restraint (“don’t add features beyond what was asked”), Pi on the model’s own RL training (“there does not appear to be a need for 10,000 tokens of system prompt”), and OpenCode on copying the winner and letting per-model prompts do the tuning. The system prompt, in Vaughan’s phrase, “is the constitution of a coding agent” — and like constitutions, they reveal what their authors fear most: Codex fears blandness, Gemini fears inaction, Claude Code fears overreach, Pi fears the constitution itself.

References (all verified Aug 2, 2026)

Primary prompt sources

Behavioral evidence

Benchmark / quantitative

Prompt archaeology / extraction