Codex and Blender: the agent isn't clicking — it's running bpy (plus the Sep-2026 map of agents that control your computer)

Research date: 10 Sep 2026. Every claim below carries the URL it was verified against on that date; community/secondary-sourced items are labeled as such and collected in the caveats. Companion articles: computer-use-landscape-2026-09-07.md (full computer-use baseline), codex-mcp-server-pattern.md.

TL;DR

• The Codex↔Blender demos are not screenshot-driven modeling. OpenAI’s own walkthrough of the house that appears in the GPT-6 Astra launch video says the scene was built through the Blender Python API (bpy), that “Astra ran scripts through Blender’s own executable in background mode, using its --python option”, and that Computer Use was used to inspect the result: “Astra also opened and inspected the scene in Blender through computer use, checking the application alongside the saved renders.” (Architectural visualization with Astra, Sept 2026)
• Programmatic actuation, visual verification. That is also the shape of the official Blender MCP server (built by the Blender project): MCP client ⇄ stdio ⇄ blender-mcp ⇄ TCP socket ⇄ Blender add-on, whose central tool is execute_blender_code, plus *_for_cli twins that run in a background Blender process. It ships screenshot tools too — but for looking, not clicking (blender.org/lab/mcp-server).
• Codex can consume any MCP server (STDIO + Streamable HTTP, ~/.codex/config.toml), so a Codex↔Blender loop needs zero screenshots (Codex MCP docs).
• There is no OpenAI-first-party Blender connector. The request openai/codex#22149 is open with 0 comments and no maintainer reply; Blender appears nowhere in OpenAI’s docs or the 90+ Codex plugins. The official connector is Blender’s, and because it is MCP it is client-neutral — Anthropic says so explicitly.
• Vendors all give the same rule: if an app exposes an API/MCP/plugin, use it; use pixels only when you must see. OpenAI: “prefer that structured integration… Choose Computer Use when ChatGPT needs to inspect or operate the app visually”; and for GPT-6 Astra “we recommend code execution” over the computer tool.
• Why: grounding is nearly solved (Astra ScreenSpot-Pro 92.7%) but long-horizon desktop control is not (OSWorld 2.0 ≈ 72.6% partial, ~40 min/task), and the GUI channel needs OS consent and the foreground. The programmatic channel is cheaper and replayable — but it executes arbitrary generated code inside your app, which is why Blender tells you to run it in a VM.

1. The answer, in OpenAI’s own words

The Astra launch post shows a Blender→Unreal house with no explanation of the mechanism (GPT-6 Astra). The developer blog for the same project does explain it:

“I asked Astra in Codex to turn that brief into an editable 3D scene in Blender.”

“Astra built an editable scene through the Blender Python API (bpy). It created the architecture, joinery, furniture, planting, materials, lights, and cameras.”

“For the cinematic studies, Astra ran scripts through Blender’s own executable in background mode, using its --python option. Those scripts placed the cameras, chose lenses, and rendered the views. Astra also opened and inspected the scene in Blender through computer use, checking the application alongside the saved renders.” — developers.openai.com/blog/architectural-visualization-with-astra

It even self-corrects from renders: “Astra produced views of the rooms and close studies of the materials, then used them to find and repair problems. One example was the steel sink: its surface normals… made a flat part of the bowl look pinched.”

So the honest one-liner: the agent writes Python; Blender executes it; the agent looks at pictures to judge the result. Any “AI clicks around in Blender” framing is describing the inspection step, or a deliberately GUI-only tutorial (e.g. bettertoken.ai, whose prompt literally says “Do not generate the geometry with a Python script” — reseller marketing page).

2. What Codex’s Computer Use actually is

It is a plugin — an MCP server plus a skill — inside the ChatGPT/Codex desktop app (docs):

• See: screenshots. Act: synthetic mouse/keyboard. macOS requires Screen Recording + Accessibility (the privacy entry is named “Codex Computer Use”); Windows “runs on the active desktop. It can’t operate in the background… expect ChatGPT to move the pointer, type, and take over the foreground.”
• Consent: per-app approvals with an “Always-allowed apps” list; admins can restrict app access; file/shell actions still follow the task’s sandbox and approval settings.
• Launch framing (April 2026, “Codex for (almost) everything”): “Codex can now use all of the apps on your computer by seeing, clicking, and typing with its own cursor… helpful for iterating on frontend changes, testing apps, or working in apps that don’t expose an API” (openai.com). Blender exposes one.
• Real-world friction: on Windows, Computer Use can refuse the Blender window over an owner mismatch — “window id <HWND> no longer belongs to blender.4.5; current owner is blender.4.5” — and in one test captured a window listed as File Explorer whose screenshot actually contained Blender (openai/codex#36335, open, 31 Jul 2026).

3. Blender’s programmatic surface

3.1 Official Blender MCP server (Blender Lab)

Page: blender.org/lab/mcp-server · Source: projects.blender.org/lab/blender_mcp (created 2026-02-21; v1.0.0 released 2026-04-27; v1.0.2 corrective release 2026-09-08; 24 Gitea stars on 2026-09-10). Announced by Blender itself in the Q1 2026 Lab report: “some exploration of LLM powered tools has started, with an implementation of an MCP connector for Blender.”

MCP Client  ⇐ MCP/stdio ⇒  blender-mcp  ⇐ TCP socket ⇒  Blender Add-on

Requires Blender 5.1+; add-on installed by dragging its link into Blender (twice — once to add the Blender Lab repository, once to install); a .mcpb bundle exists for clients that accept it.

Selected tools (full list in readme_tools.rst):

Note what is absent: no click_at, no drag, no press_key. Control is code; pixels are input. The Blender page is also blunt about the risk:

“The MCP server will execute LLM generated code in Blender without any guards… recommended to use a virtual machine, or a system without access to sensitive information.”

and about scope: “Blender does not have any built-in functionality for connecting to LLMs.” Blender’s policy line, after the Anthropic-donation backlash: “No generative AI functionality is currently available or planned to be integrated in Blender.” (AI policies, 1 May 2026)

3.2 The community server most people install

ahujasid/blender-mcp — 28,084★, created 2025-03-07, pushed 2026-09-07. Same shape (add-on opens a socket server inside Blender, JSON over TCP, port 9876; MCP server bridges to it), plus asset integrations (Poly Haven, Sketchfab, Poly Pizza, Hyper3D Rodin, Hunyuan3D) and an “Execute any Python code in Blender” tool. Codex is a documented client:

[mcp_servers.blender]
command = "uvx"
args = ["blender-mcp"]

or codex mcp add blender -- uvx blender-mcp. Its own warning: “ALWAYS save your work before using it”; opt-in BLENDER_MCP_SAFE_MODE=1.

4. Wiring Codex → Blender takes three lines

Codex (app, CLI and IDE extension) shares one MCP config (docs): STDIO or Streamable HTTP servers, bearer/OAuth, ~/.codex/config.toml (or project .codex/config.toml), codex mcp add, /mcp to list. So:

1. Install the Blender add-on (community or Blender Lab) and start its server from Edit → Preferences → Add-ons → BlenderMCP.
2. Add it to Codex (~/.codex/config.toml or Settings → MCP servers → Add server).
3. Ask for a scene. The agent calls execute_blender_code, reads structured summaries back, and only opens a PNG when it needs to judge how something looks.

Anthropic’s Claude path is the same idea, shipped productized: “The Blender developers have created an MCP connector, which is now officially available for Claude… And because the connector is built on MCP, it is accessible to other LLMs in addition to Claude” (Claude for Creative Work, 28 Apr 2026; setup in Claude Academy).

5. Why there is no OpenAI Blender connector

• openai/codex#22149 “Blender Connector / MCP-style integration for Codex” — opened 2026-05-11, open, 0 comments, no maintainer reply, 10 👍: “I recently saw Anthropic introduce the Blender Connector for Claude… Are there any plans for official Blender integration support? Would MCP-style tool connectivity be the recommended direction?”
• No “blender” in OpenAI’s docs index or cookbook; none among the 90+ Codex plugins announced with computer use. OpenAI’s only Blender artifact is a use-case blog post.
• Meanwhile Anthropic became a Blender corporate patron (28 Apr 2026).

This is absence of evidence, not a vendor statement — but it matches the strategy: OpenAI ships a generic computer-use + MCP substrate; the connectors are the app vendors’ job (or the community’s).

6. Four channels — and how to spot them in a demo

OpenAI’s own selection rule for 1/3 vs 4: “If the target app exposes a dedicated plugin or MCP server, prefer that structured integration for data access and repeatable operations. Choose Computer Use when ChatGPT needs to inspect or operate the app visually” (docs); and on the API: “For GPT-6 Astra, we recommend code execution. The computer tool remains supported as an alternative.” For 3 vs 1: use programmatic calling when control flow is predictable and code can return a smaller structured result; keep direct calls for “writes or approval-sensitive actions… to preserve a clear authorization boundary” (PTC guide).

The tell in any demo: if the artifacts are .py/.blend and renders, it’s channel 1–2. If the artifact is a screen recording, it’s channel 4.

7. The numbers behind “prefer the API”

• Grounding ≈ solved: GPT-6 Astra ScreenSpot-Pro (no tools) 92.7%.
• Long-horizon control is not: OSWorld 2.0 (v2026.08.08, offline set, partial score) Astra 72.6% vs GPT-5.6 Sol 65.7%, Claude Fable 5 70.2% — and “Astra achieves higher computer-use performance in about 47% less time per task… roughly 40 minutes per task vs ~75 minutes” (openai.com/index/gpt-6-astra). ⚠️ Vendor-reported partial scores on an offline subset; do not compare with the OSWorld 2.0 paper’s binary numbers without the metric.
• Safety: internal computer-use safety benchmark (lower better) Astra 2.4% (1.8% with AutoReview) vs Sol 22.0%, Fable 5.1 9.5%, Fable 5 18.3%, Opus 5 11.5%.
• 3D-as-code is strong: BenchCAD (rebuild 3D objects from multi-view renders by generating CAD code) Astra 95.9% vs Sol 83.3%, Fable 5.1 84.3%, Opus 5 82.1%.
• Failure modes differ by channel. GUI: window/permission mismatches (codex#36335). Code: procedural over-fragmentation — a community benchmark of “codex… using MCP and computer use” reports a chair built from “137 mesh objects and 1,492 vertices, which suggests a procedural assembly of simple parts rather than a compact mesh” (codex-blender-bench, community source).

8. The landscape: agents that control the computer

8.1 Every vendor now documents a tool ladder — and says start at the top

The vendors say this in writing:

• OpenAI: “prefer that structured integration… Choose Computer Use when ChatGPT needs to inspect or operate the app visually”; for GPT-6 Astra “we recommend code execution” (computer-use docs, API guide).
• Anthropic: “Claude will reach for the most precise tool first, starting with connectors to services like Slack or Google Calendar. When there isn’t a connector, Claude can directly control your browser, mouse, keyboard, and screen”; and “working through your screen is slower than using a direct integration” (Dispatch and computer use, 23 Mar 2026). Its newer browser tool “adds page structure, so agents target web elements more reliably than with pixels alone” (20 Aug 2026).
• AWS, in the plainest sentence anyone has published on this: “Computer-use agents and APIs solve fundamentally different problems. When an API exists, agents should — and do — use it. But the majority of enterprise software — legacy ERP systems, thick-client applications, proprietary tools — simply doesn’t have API access.” (The Register, 6 May 2026)
• Microsoft doesn’t ship a pixel product; it ships a registry: MCP on Windows / the Windows On-device Agent Registry (ODR) — “a secure, manageable interface to discover and use agent connectors”, where “MCP servers are contained in a separate environment by default and can only access approved resources, limiting vulnerability to threats like cross-prompt injection attacks”, plus default Windows connectors including a File Explorer MCP server (docs, last updated 4 Jun 2026); and App Actions (URI/COM activation) — Windows’ answer to Apple’s App Intents.
• Apple likewise: App Intents, on-screen awareness, and MCP in Xcode 27 — no third-party screen-driving (newsroom, 8 Jun 2026).
• Google is the one major vendor with no published “prefer an API” sentence in its computer use docs — I searched the full page. Its substitute is safety machinery: a per-action intent field, safety_decision ∈ {allowed, require_confirmation, blocked}, and opt-in screenshot prompt-injection detection (default false) (ai.google.dev).

8.2 Creative and 3D apps: the connector wave is entirely programmatic

NVIDIA’s SIGGRAPH 2026 write-up states the trend plainly:

“Leading creative applications are opening Model Context Protocol (MCP) connections that let AI agents work inside the tools where scenes, shots, timelines, assets and edits come to life — while creators stay in control.” — NVIDIA blog, 20 Jul 2026

Its roll-call, same source: Adobe (creative agent across Firefly/Express/CC plus an Adobe Express Developer MCP Server), Affinity by Canva (AI Connector for Claude, MCP-based), Blender (“offers a lightweight MCP server through Blender Lab… a strong example of how open creative tools can become agent-accessible without changing the creative center of gravity”), Boris FX Silhouette (FX Scripting API as first-class MCP tools; interactive online mode plus offline headless instances), Foundry Griptape (MCP orchestration across Blender and Nuke), SideFX Houdini 22 (MCP via the new APEX Script workflow, initially rigging), Unreal Engine (connect AI clients to the editor through MCP). Anthropic’s version is the 28 Apr 2026 connector drop. Not one of these integrations is screenshot-driven.

8.3 Pixels are being absorbed into the programmatic namespace

The neatest 2026 pattern: computer use is increasingly an MCP server. Codex’s Computer Use is literally “an MCP server + a skill” you toggle in Plugins. AWS WorkSpaces gives agents “a managed MCP endpoint that provides governed access to desktop tools such as screenshots, mouse control, and text input”. The popular community Windows server is UIA-tree-based and vision-optional (“doesn’t rely on any traditional computer vision techniques”; ~7k★). trycua/cua (~22.5k★) calls the genre “Computer-Use 2.0… an agent moving between code, APIs, and graphical interfaces within the same task.”

And the newest structural piece — WebMCP, a Draft Community Group Report dated 9 September 2026, edited by Microsoft and Google engineers:

“The WebMCP API enables web applications to provide JavaScript-based tools to AI agents… Web pages that use WebMCP can be thought of as Model Context Protocol servers that implement tools in client-side script instead of on the backend.” — webmachinelearning.github.io/webmcp

Its repo argues the case explicitly in anti-pixel terms (“UI Disintermediation & Context Loss”), and it is already live: supported in ChatGPT Desktop (“Site tools”, shipped 25 Aug 2026), origin trials in Chrome 149 and Edge 150, experimental in Brave; no Firefox/Safari support yet. That is Blender’s story repeated for the web: give the agent tools, not pixels.

8.4 What is new since the 7 Sep 2026 baseline

• Meta Muse (8 Sep 2026) — a personal agent on “Muse Secure VM, a dedicated secure computer with its own browser”, with a separate Sentinel agent that must approve anything reaching the internet (about.fb.com) — the sandbox-per-agent pattern, productized.
• Claude in Chrome GA and a built-in browser in Cowork (both 26 Aug 2026).
• ChatGPT desktop app for Linux in preview (11 Aug 2026) — and “Some features, including Computer Use, aren’t yet available in the Linux preview”.
• Codex Apple Messages plugin (20 Aug 2026) — reads/chats/sends through Messages, “sends messages only after you approve” — again programmatic, not GUI.
• Unity official plugin for Claude Code (10 Sep 2026) — press-reported.
• MCP spec is still 2026-07-28; “stateless MCP” is press-reported and not yet a published release.

8.5 What the channels actually cost

• Long-horizon GUI work is still the weak spot, and the failures are cognitive. OSWorld 2.0’s own page: under binary completion at 500 steps, the best model “completes only 20.6% of tasks at a 54.8% partial score”; agents fail because they “lose track of constraints, miss information that arrives mid-task, guess rather than ask the user, and skip verification” — not because they can’t click (osworld-v2.xlang.ai). Vendor numbers in the 62–73% range are partial scores on different releases/budgets — never quote them next to 20.6%. For a like-for-like binary comparison, Simular’s own chart (28 Aug 2026) puts Sai at 28.25% binary vs 23.14% for GPT-5.6 Sol.
• Pixels cost tokens. Reflex’s benchmark: “a browser-use vision agent needed half a million tokens to click on a dropdown menu”, concluding agents can be “45 times more expensive than using an API” — a single-scenario vendor benchmark that AWS publicly disputes (The Register, 6 May 2026).
• Mix shift, measured at scale. The UK AI Security Institute analysed 177,436 MCP tools (Nov 2024 → Feb 2026): “the share of action tools in monthly downloads rose from 24% to 65%…, driven by growth in computer use and browser automation tools”, and warns that consequential actions increasingly happen “rather than through restricted, secure API integrations” (AISI, 26 Mar 2026). The same study: MCP servers with payment execution grew from 46 to over 1,200 in a year, and 55% of MCP servers published in Jan 2026 show AI assistance in their creation (Claude Code accounts for 66% of those) — the connector ecosystem is now largely agent-written.
• Security posture differs by channel. Pixel channels need screenshot injection classifiers (Anthropic runs them automatically and steers the model to ask for confirmation; Google’s is opt-in, default false). Structured channels trade that for containment: Windows ODR sandboxes servers; WebMCP gives “each tool invocation… a safety review before it runs”, “tied to its originating page and tool registration” — while opening a new text-level injection surface in tool descriptions and names.

Synthesis: the frontier labs ship generic perception + actuation; app vendors ship the connectors; the community fills the gaps; and the pixel channel is being standardized into just another MCP endpoint. Blender is the template case — an open app that became agent-accessible by exposing a protocol next to its Python API, without putting AI inside the product (its policy still says “No generative AI functionality is currently available or planned to be integrated in Blender”). Agents that “control your computer” are converging on controlling your applications' APIs, and reaching for the mouse only where no API exists. Per-OS substrate, mobile coverage, the protocol stack and the full benchmark table are in the companion baseline computer-use-landscape-2026-09-07.md (research date 7 Sep 2026).

9. Caveats

• openai.com blocks scripted fetches (403); the launch-page texts were read via a text mirror of the canonical URLs — descriptions are OpenAI’s words, but dates (Codex computer use: macOS 16 Apr 2026, Windows May 2026; Astra: 3–4 Sep 2026) are secondary/community-reported. The arch-viz post exposes no machine-readable date.
• OSWorld 2.0 figures are vendor partial scores on an offline subset, not paper binary scores.
• bettertoken.ai (API-key reseller) and vagon.io (cloud-PC vendor) are marketing sources; their value here is that they paraphrase primaries I read directly.
• The Medium walkthrough (23 Jul 2026) is fetch-blocked and was read via a mirror; “headless” there is inference, not a quote.
• Vendor self-published: 3D-Agent, Mixar (a fork of Blender with an in-app agent).
• Unverified, HN-only: a claim that the community Blender MCP maintainer’s GitHub account was compromised (HN 49238028, 10 Aug 2026) — worth checking before uvx blender-mcp.
• Never quote OSWorld numbers without the metric. Paper: 20.6% binary / 54.8% partial (best model, 500 steps). Vendor claims of 62–73% are partial scores on other releases and step budgets; Astra’s 72.6% in §7 is a partial score on the offline 2026.08.08 set.
• The 45× / 500k-tokens figure is one vendor benchmark on one browser scenario, publicly disputed by AWS; treat it as directional.
• Google: its computer-use docs contain no “prefer an API/MCP” statement — absence of evidence, not a claim.
• WebMCP is a Draft Community Group Report, not a W3C Standard; support is uneven (ChatGPT Desktop yes, Chrome 149 / Edge 150 origin trials, no Firefox/Safari).
• Press-reported only (no vendor page resolved): Unity plugin for Claude Code (10 Sep 2026); Unreal 5.8 MCP; Adobe’s own posts for 28 Apr / 6 Aug 2026. Meta Muse is vendor-verified at about.fb.com.
• Houdini: the NVIDIA blog (20 Jul 2026) is the primary source for MCP-in-Houdini-22; there is no official SideFX docs page for it (careful — sidefx.com/docs/.../mcp.html is Houdini’s movie-conversion CLI, unrelated to MCP), and trade coverage describes it as a Labs-bound sneak peek focused on APEX Script rigging rather than a shipping core feature.
• Claude Fable 5.1 / Mythos 5.1 (1 Sep 2026) are press-indexed only — no canonical URL or benchmark numbers were verified, so none are cited here.
• “No official OpenAI↔Blender connector” is absence of evidence across docs, cookbook, plugin list and issue tracker, not a vendor denial.