lagindicator

Computer Use in Sep 2026: GUI Agents for Desktop, Phone and Pad — Features, Limits per OS, and Where the Community Is Building

Agent Dev Tools

A dated (7 Sep 2026) map of the 'computer use' category — AI agents that read a screen and drive mouse/keyboard/touch on real machines. Covers the frontier products (Anthropic, OpenAI, Google, Microsoft, Apple), the open-source stack and benchmarks, desktop OS-by-OS coverage (Windows, macOS, Linux/Ubuntu), phone & pad use (Android vs iOS, what it takes to run one, Google Artemis deep-dive), the hard limits that stop it from controlling 'everything', and where the community is building (MCP/A2A/AG-UI, SDKs, agentic-OS).

Research date: 7 Sep 2026. Every claim below carries the URL it was verified against on that date; community/third-party-sourced numbers are labeled “community-reported”; items that could not be independently verified are collected in the caveats section. Star counts and repo dates are live GitHub API snapshots from 7 Sep 2026.

TL;DR

  • What “computer use” is: a model loop that perceives a UI (screenshots and/or the OS accessibility tree) and acts by emitting synthetic mouse/keyboard/touch actions that your own client executes. Frontier labs turned it from an Oct-2024 research demo into a GA category by mid-2026 (Anthropic toolset GA 20 Aug 2026, OpenAI computer tool + ChatGPT-desktop Computer Use, Google Gemini computer-use environments browser/mobile/desktop).
  • Can it control everything? No. Every OS is “read with consent, act with consent”: TCC/UAC/Wayland/Android gates, auth walls (passkeys, 2FA, password fields), CAPTCHA, DRM screens. Agents control “everything” only inside a sandbox they are handed — never silently on your real machine.
  • Desktop: Windows and macOS have the deepest products (OpenAI ChatGPT Computer Use on both, Anthropic macOS dispatch, Microsoft Windows-11-VM tooling). Linux/Ubuntu has no first-party vendor driver — it is the open-source commons and the sandbox OS agents run inside.
  • Phone/pad: Android is agent-friendly (Google’s open-source Artemis, “agentic Android” on Galaxy S26 preview); iOS/iPadOS is closed — no third-party screen-driving agent exists, only dev-time XCUITest/WebDriverAgent tooling and Apple’s sanctioned App Intents path. Pads: Android tablets reuse Android tooling; iPads have no sanctioned path.
  • Direction: protocol stack consolidation (MCP spec 2026-07-28, A2A v1.0 now under the Linux Foundation-directed Agentic AI Foundation, AG-UI for agent↔app), computer use as a standard tool category in every agent SDK, and OS vendors embedding agents (Microsoft “first agentic OS”, Google killing standalone Project Mariner on 4 May 2026, ChatGPT workspace agents, Claude Cowork) — so GUI-driving becomes the fallback escape hatch for legacy/cross-app work.

1. What “computer use” is, and how it differs from “browser use”

A computer-use (CUA) system is an agent loop where the model is given screen state and emits UI actions that run on a real OS:

Feature set of the category: perception (screenshots — ~1,000–1,800 input tokens each on Claude — plus accessibility tree/DOM), actuation (synthetic input; Android agents use adb shell input), grounding parsers (OmniParser V2, UI-TARS, Qwen-VL, OS-Atlas) for models without native vision, planning/state-tracking, human-in-the-loop gates, and sandboxing.

Browser use vs computer use. Anthropic’s docs: “For tasks that stay inside webpages, the browser use tool is the closer fit… it doesn’t need a full desktop environment” (https://platform.claude.com/docs/en/agents-and-tools/tool-use/browser-use-tool). Computer use exists because much software is not a webpage — native/legacy/cross-app workflows. Browser-only orgs are only starting to expand; the browser-use org’s desktop move is a brand-new, Windows-only, experimental windows-harness (created 23 Aug 2026; PrintWindow screenshots + Windows UI Automation + synthetic input; https://github.com/browser-use/windows-harness). For the browser-harness/browser-use side of the story see the separate browser-harness research.

2. The landscape map — frontier vendors

Anthropic. Computer use is a GA client toolset (computer_toolset_20260801, no beta header) on the Claude API and Google Cloud/Vertex (beta on AWS Bedrock / Microsoft Foundry); models include the claude-fable-5.1 / mythos-5.1 / opus-5 / sonnet-5 line and claude-opus-4.8; not in Claude Managed Agents (https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool). GA of computer use + a new browser-use tool + Skills/Files APIs was announced 20 Aug 2026 (https://claude.com/blog/computer-use-skills-api-files-api). The reference environment is a Linux container (Xvfb + Mutter/Tint2 + Firefox + LibreOffice; https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo). Desktop-adjacent: Claude Code (macOS/Windows/Linux-beta + IDE/desktop/web/iOS/Android, https://code.claude.com/docs/en/overview); Claude in Chrome is GA (https://claude.com/claude-in-chrome); since 23 Mar 2026 Claude Code and Claude Cowork can fall back to driving your real screen when no tool exists, incl. Dispatch where a task is assigned from the Claude iOS/Android app and Claude operates your Mac (https://claude.com/blog/dispatch-and-computer-use). Safety: automatic screenshot prompt-injection classifiers steer Claude to ask for confirmation (docs above). Pricing: token-usage only; screenshots dominate cost.

OpenAI. Changelog lineage: computer-use-preview + built-in computer-use tool in the Responses API (11 Mar 2025) → GPT-5.4 mini with built-in computer use (17 Mar 2026) → GPT-5.5 (24 Apr 2026) → GPT-6 Astra (3 Sep 2026) where docs recommend code-execution (PyAutoGUI/Playwright) over the pixel tool (https://developers.openai.com/api/docs/changelog). Sandboxing is the developer’s job; Agents SDK supports containerized sandbox agents (https://developers.openai.com/api/docs/guides/agents/sandboxes). ChatGPT desktop “Computer Use” (macOS + Windows, ChatGPT Work/Codex, via an MCP-server plugin + skill) drives macOS apps, Windows apps, browsers and iOS simulator flows; Windows is foreground-only with a per-app allowlist; macOS supports “locked use” and needs Screen Recording + Accessibility grants (https://learn.chatgpt.com/docs/computer-use). Operator (cloud browser agent) folded into ChatGPT as “ChatGPT agent” on 17 Jul 2025 (https://openai.com/index/introducing-operator/). GPT-5.6 Sol: $4/M input, $20/M output (promo); no separate computer-use surcharge.

Google. Gemini API Computer Use tool on Gemini 3.x with the Interactions API; examples use Gemini 3.8 Flash (https://ai.google.dev/gemini-api/docs/computer-use). Project Mariner was shut down on 4 May 2026, its tech folded into Gemini Agent / Google AI Mode / Chrome Auto-Browse (https://en.wikipedia.org/wiki/Project_Mariner https://www.digitaltrends.com/computing/google-pulls-the-plug-on-project-mariner-the-ai-agent-that-browsed-the-web-like-a-human/). Google’s current general agent Antigravity is a managed agent in a remote Linux sandbox — web/files/code only, not desktop-GUI (https://ai.google.dev/gemini-api/docs/antigravity-agent). ADK 2.0 ships an official computer-use toolset over Playwright/Chromium (https://adk.dev/integrations/computer-use/).

Microsoft. No first-party “Windows computer-use model”; the play is open tooling + OS + cloud-PC infrastructure: OmniParser V2 (Feb 2025; 39.5% ScreenSpot-Pro; MIT; Jul-2026 YOLOv9-E detector) + OmniTool driving a Windows 11 VM with your model of choice (https://github.com/microsoft/OmniParser https://www.microsoft.com/en-us/research/articles/omniparser-v2-turning-any-llm-into-a-computer-use-agent/); Windows Agent Arena (Windows-11-VM benchmark platform, https://github.com/microsoft/WindowsAgentArena); Magma research model (UI + robotics, https://github.com/microsoft/Magma); Windows 365 for Agents (CUAs on Cloud PCs, Build 2026, https://github.com/microsoft/windows-365-for-agents); the Windows AI stack explicitly says Agent Launchers are “not… for background services or silent automation” (https://learn.microsoft.com/en-us/windows/ai/agent-launchers/).

Apple. No pixel/screenshot-level computer-use product exists on Apple’s developer site. Apple’s story is intent-based: App Intents (per-app opt-in actions, https://developer.apple.com/documentation/appintents), View Annotations “on-screen awareness”, Foundation Models, Shortcuts NL automation. No verifiable Apple computer-use strategy (absence of evidence).

Startups (community-reported). Aggregator coverage repeatedly names Claude Cowork, Manus Desktop, Perplexity Computer as the leading third-party desktop agents of 2026 (e.g. https://andrew.ooo/answers/best-ai-desktop-agents-2026/), and AskUI as agentic UI testing across web/desktop/OS-level (https://www.askui.com/blog-posts/claude-vs-openai-operator-vs-askui). Treat feature claims as community-reported — several could not be vendor-verified this session.

3. The open-source & research stack

“Eyes” — GUI grounding models & screen parsers

ProjectWhat it isKey numbers (as cited)Source
UI-TARS / 1.5 / 2 (ByteDance)Open-weights GUI-grounded models; UI-TARS-1.5-7B is the de-facto grounding model inside open agentsUI-TARS-2: 47.5 OSWorld / 50.6 WAA / 73.3 AndroidWorld / 88.2 Online-Mind2Web (self-report)https://github.com/bytedance/UI-TARS arXiv 2509.02544
OmniParser V2 (Microsoft)Pure-vision screen parser (region detector + OCR/captioner)39.5% ScreenSpot-Pro at releasehttps://github.com/microsoft/OmniParser
Qwen-CUA (xLang/Alibaba)397B-A17B MoE native CUA — screenshots only, no a11y tree86.2 OSWorld-Verified / 18.5 OSWorld-2.0 binary (Max: 87.6 / 21.2)arXiv 2608.02352, https://github.com/xlang-ai/Qwen-CUA
Qwen2.5/3-VL“Interactive visual agent… operating computers and mobile devices”—arXiv 2502.13923, arXiv 2511.21631
OS-AtlasCross-platform GUI grounding corpus (13M elements)—arXiv 2410.23218
ShowUI / ShowUI-π2B VLA, screenshot grounding75.1% zero-shot screenshot groundingarXiv 2411.17465, arXiv 2512.24965
2026 entrants (ScaleCUA, MAI-UI, EvoCUA, CUA-Suite)Larger open CUA effortsScaleCUA +10.7 ScreenSpot-Pro (arXiv 2509.15221); EvoCUA 56.7% OSWorld (arXiv 2601.15876); UI-Vision: foundation action models fail ~60% of professional desktop tasks (arXiv 2603.24440)arXiv above

Agent frameworks, drivers, sandboxes

  • Simular Agent S — strongest complete open stack (Apache-2.0, ~12.2k★): S1→S3 drives real macOS/Windows/Linux desktops via screenshots + grounding model + OS a11y APIs; first framework past ~72% human on OSWorld (72.6% best-of-n) (https://github.com/simular-ai/Agent-S). Hosted Sai claims 73% on OSWorld 2.0 — conflicts with the paper’s numbers (Section 4).
  • trycua/cua — MIT background MCP/CLI driver (click/type/scroll + a11y-tree inspection without stealing focus) on macOS/Windows/Linux; sandbox fleets for Linux, Windows 11, macOS, Android (https://github.com/trycua/cua).
  • ByteDance Agent TARS (formerly UI-TARS-desktop) — multimodal agent stack with a native GUI agent for local/remote computer & browser (~38.9k★; https://github.com/bytedance/UI-TARS-desktop).
  • browser-use windows-harness — Windows-only, experimental, 2 weeks old (23 Aug 2026): PrintWindow screenshots + Windows UIA + synthetic input (https://github.com/browser-use/windows-harness).
  • LangChain LangGraph CUA — official open CUA implementation (https://github.com/langchain-ai/langgraph-cua-py).
  • Pivoted/stalled pioneers: Open Interpreter is now a Rust coding agent (https://github.com/openinterpreter/openinterpreter); OS-Copilot dormant since Sep 2024; Self-Operating Computer (2023–24) idle since Sep 2025; OpenAdapt refocused on verified demo→program compilation (https://github.com/OpenAdaptAI/OpenAdapt). Anthropic’s original computer-use-demo repos now 404 — the pattern survives as container quickstarts + Cua-style sandbox fleets.

Benchmarks & reliability — what the numbers actually say

BenchmarkWhat it measuresReference points (as cited)Source
OSWorld 1.0 (2024)369 desktop tasks (Ubuntu/Windows/macOS)Humans 72.4%; best agent 12.2% (2024) → frontier ~70–80%, open models 47–69% by early 2026arXiv 2404.07972
Windows Agent ArenaWindows-11 VM tasks at scaleUI-TARS-2 50.6; Agent S3 50.2→56.6https://github.com/microsoft/WindowsAgentArena
OSWorld 2.0 (Jun 2026)108 real long-horizon workflows (~318 tool calls vs ~30 in v1)Paper-verified: best (Claude Opus 4.8, 500 steps) 20.6% binary / 54.8% partial; GPT-5.5 ~13%. Vendor claims conflict (Sai 73%, GPT-5.6 Sol 62.57%) — read any number with its metric/releasearXiv 2606.29537, https://github.com/xlang-ai/OSWorld-V2 https://www.simular.ai/articles/sai-tops-osworld-2-0
Efficiency studiesSteps vs humans, tool use, chained workBest agents take 2.7–4.3× more steps; chain completion ~31%arXiv 2506.16042, arXiv 2510.24563, arXiv 2606.21654

Takeaways: grounding (finding the element) is largely solved; long-horizon reliability is not — OSWorld 2.0’s failure classes are cognitive (losing constraints, missing mid-task info, guessing, skipping verification), and Anthropic’s launch eval (14.9% OSWorld vs 70–75% human) remains the canonical “this is hard” datapoint (https://www.anthropic.com/news/developing-computer-use). Benchmark inflation is real: v1 scores overstate everyday readiness; v2 numbers (~13–21%) are the sobering end.

4. Desktop OS coverage — Windows, macOS, Linux/Ubuntu

The substrate on each OS: a way to read UI (accessibility tree and/or pixels), a way to inject input, and a consent model between them.

OSRead UISend inputPermission/sandboxState of computer use
WindowsUI Automation (UIA) tree (COM; supersedes MSAA); FlaUI, pywinautoSendInput — “may inject input only into applications at an equal or lesser integrity level” (UIPI)No consent for desktop automation, but UAC prompts live on the secure desktop — an agent can’t click its own elevation prompt; Windows Sandbox (Pro/Enterprise+) is the sanctioned boxDeepest vendor tooling: OmniTool (Windows 11 VM), Windows Agent Arena, Windows 365 for Agents, OpenAI ChatGPT Computer Use (foreground-only), windows-harness. WinAppDriver effectively dead (per Appium’s README)
macOSAccessibility (AX) tree; AppleScript System EventsCGEvent/CGEventPost or AX actions — TCC-gated (Accessibility, Screen Recording, Input Monitoring)ScreenCaptureKit needs Screen Recording consent; notarization; Electron/CDP escape hatch via --remote-debugging-portProduct leader today: Anthropic Claude Code/Cowork + Dispatch; OpenAI Computer Use (macOS “locked use”); Electron/CDP control of Slack/Teams/VS Code-class apps
Linux / UbuntuAT-SPI2 accessibility bus (pyatspi, dogtail); opt-inX11: xdotool. Wayland: no global input injection — ydotool (kernel uinput) or wtype; screen capture consent-gated via xdg-desktop-portal ScreenCast dialogs; Xvfb for headlessNo first-party Ubuntu/Canonical driverNo vendor product runs on Linux desktops; Linux is the sandbox OS agents run inside (Anthropic container, OpenAI sessions, Google Antigravity, OSWorld’s Ubuntu tasks) and the open-source commons

Sources: https://learn.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32 https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendinput https://learn.microsoft.com/en-us/windows/security/application-security/application-control/user-account-control/how-it-works https://github.com/Genymobile/scrcpy (mechanism context), https://gitlab.gnome.org/GNOME/at-spi2-core https://flatpak.github.io/xdg-desktop-portal/docs/doc-org.freedesktop.portal.ScreenCast.html https://wayland.freedesktop.org/

5. Phone use, pad use: Android, iOS, iPadOS, tablets

Vendors on mobile

Android vs iOS vs tablets/iPads — the split

  • Android is agent-friendly: AccessibilityService yields a live UI tree; UiAutomator / adb shell input inject synthetic taps without OEM cooperation; screenshots via scrcpy/screencap (https://developer.android.com/guide/topics/ui/accessibility/service https://developer.android.com/tools/adb).
  • iOS/iPadOS is closed: no public third-party input injection on real devices; XCUITest runners are dev-time only; App Intents is the only sanctioned consumer path. Result: no genuine third-party phone-use agent for iPhone/iPad — even Google’s Artemis lists iOS as a roadmap item (https://github.com/google/artemis).
  • Tablets: Android tablets = same Android tooling; iPadOS has no sanctioned third-party agent control; no verified commercial “iPad agent” product. The universal loophole: browser agents work on any device with a browser — much “phone use” today is web use from a phone.

What it takes to run an Android pad/phone agent (quick start, Sep 2026)

Short version: no root, yes developer options, no special distro; the tooling is free, the model tokens aren’t.

QuestionAnswerSource
Root?No. scrcpy “does not require root access or an app installed on the device”; Artemis likewisehttps://github.com/Genymobile/scrcpy https://github.com/google/artemis
Developer mode?Yes — enable Developer options → “USB debugging” (tap build number 7×); Wireless debugging (Android 11+) for Wi-Fi ADB. One-time on-screen ADB authorizationhttps://developer.android.com/studio/debug/dev-options
Specific distro?No. Any Android with USB debugging. Documented OEM quirk: Xiaomi/MIUI needs the extra “USB debugging (Security Settings)” toggle or input injection fails with INJECT_EVENTS errors. scrcpy floor: Android 5.0 (API 21)scrcpy README
Where does it run?On your computer — the tablet is the display+input target (ADB/scrcpy on the host)Artemis README
Emulator alternativeAndroid Studio AVD (or cloud phone), no dev-options needed; the AndroidWorld benchmark runs this wayhttps://github.com/google-research/android_world
Accessibility-service routeNo root; a human must enable the service in Settings; Android 13+ “Restricted Settings” gates sideloaded apps; Play policy confines a11y botshttps://developer.android.com/about/versions/13/features#restricted-settings
Free?Tooling yes (Artemis Apache-2.0, scrcpy Apache-2.0, ADB free). The model brain is metered — bring your own Gemini/Claude/GPT/Qwen API keyhttps://ai.google.dev/gemini-api/docs/pricing

Deep dive: Google Artemis — how it actually controls Android (and what it can’t)

Repo status (GitHub API, 7 Sep 2026): created 13 Aug 2026, latest push 5 Sep, no releases/tags, ~172★ / 25 forks — days-old-class project, pre-release, treat claims accordingly (https://api.github.com/repos/google/artemis). Apache-2.0, host-side Python; official README: “Ensure an Android device (with USB Debugging enabled) or emulator is connected” (https://github.com/google/artemis).

Mechanism (from the code): no root, no a11y service, nothing installed on the device — the loop runs entirely over ADB via an AndroidAdbDriver (https://github.com/google/artemis/blob/main/artemis/drivers/android/adb_driver.py):

  1. Perceive: each step waits ~300–400 ms, then grabs a screenshot via ADB and the UI hierarchy via a UIAutomator2 client (uiautomator dump XML parsed into elements with bounds/clickable/focusable/text/class). On-device OCR is fused with the XML into an indexed minimal list of interactable elements (https://github.com/google/artemis/blob/main/artemis/mcp/observation.py).
  2. Decide: a multimodal model of your choice (Gemini / Claude / GPT / Qwen-VL) sees the screenshot + element index and emits the next action from a tool catalog (click/long-press/swipe/scroll/type/keys/launch-stop-app/open-url/wait-for-text).
  3. Act: input is injected as real Android input events via adb shell: input tap x y, input swipe x1 y1 x2 y2, input keyevent, text via a 3-tier scheme (clipboard+KEYCODE_PASTE → ADBKeyboard IME broadcast → input text), app launch via monkey -p <pkg> -c android.intent.category.LAUNCHER 1 (https://github.com/google/artemis/blob/main/artemis/mcp/actuators/adb.py).
  4. Guard: a local 0-token “Safety Net” re-checks targets and clears stray system popups before each action; screen video via scrcpy powers replays. Latency: Flash ~3–5 s/step; Pro ~15–40 s/step (multi-agent planner/operator/checker) (README).

Can it play games? Mechanically it taps/swipes/keys, so coarse, deliberate actions work — but there is no official gaming claim and the design says no for real gameplay: games render to a canvas with few/no a11y nodes (perception degrades to pixel/vision), latency is seconds per step, and there are no multi-touch gestures or gesture streams (inference from the action catalog; not vendor-verified).

Can it control arbitrary UI? Broadly yes for standard Android view hierarchies across apps (Compose/Flutter/WebView expose a11y semantics; cross-app flows are the point). Boundaries: FLAG_SECURE screens (banking/DRM — screenshot returns nothing), auth walls (passkeys/biometrics/2FA), and password fields it can type into but never read. Popups are self-healed; sensitive prompts still route to a human.

Most common use case (README’s own framing): mobile QA/test automation — zero-maintenance regression (no XPath/ID selectors), exploratory + 10-hour soak/“monkey-plus” stability testing with Logcat diagnostics, and MCP-native bug repro from AI IDEs (Antigravity, Claude Code, Codex, Windsurf). Secondary: autonomous cross-app daily tasks. Its “99%+ on AndroidWorld” is a self-reported README claim on the community-submitted leaderboard — treat as directional.

Mobile reliability

Official AndroidWorld leaderboard (Google-maintained, community-submitted/self-reported): humans ~80%; M3A (GPT-4 Turbo, 2024) 30.6% → Gemini 2.5 CU 69.7% (10/2025) → AutoGLM-Mobile 80.2% → Artemis 99.1% → FluizAI 100% (08/2026, closed) (leaderboard sheet in sources). Research failure modes: dynamic popups (AnTrap, arXiv 2608.24099); environmental prompt injection via on-screen text and even the a11y tree (MobileWorldSafety arXiv 2608.17659); text entry + auth screens remain blockers; benchmark reward-hacking risk (BenchJack, arXiv 2605.12673). Bottom line: emulator Android tasks approach saturation; real-device robustness, safety, and iOS/iPadOS lag far behind.

6. Can it control everything? — the honest answer

No, and the limits are structural, not just model quality. Four layers:

  1. OS consent gates. Windows: UAC elevation prompts render on the secure desktop only Windows processes can access — an agent cannot click its own elevation prompt; SendInput is integrity-limited (MS docs above). macOS: Accessibility / Screen Recording / Input Monitoring TCC grants; ScreenCaptureKit needs Screen Recording consent. Linux/Wayland: no global input injection by design; per-capture ScreenCast portal dialogs are user prompts. Android: Restricted Settings + Play policy gate a11y bots; iOS: no third-party injection at all.
  2. Auth & secret walls. WebAuthn/passkeys bind to a platform authenticator requiring local user verification — an agent driving the UI cannot satisfy it (https://www.w3.org/TR/webauthn-3/ https://developer.apple.com/passkeys/); password fields never expose values (Windows UIA marks password properties; macOS NSSecureTextField likewise); 2FA push goes to the human’s other device.
  3. Content that never reaches the agent. CAPTCHA/anti-bot is adversarial by design; DRM flows through secure paths (Android MediaDrm, https://developer.android.com/reference/android/media/MediaDrm) and never appears in the pixel feed.
  4. Vision & trust limits. Anthropic’s launch eval: 14.9% OSWorld vs 70–75% human; “flipbook” screenshot perception misses short-lived states; no drag/zoom initially (https://www.anthropic.com/news/developing-computer-use). Screenshots also expose agents to prompt injection via pixels; every vendor now ships classifiers/scanning or confirmation-steering.

The corollary — inside the sandbox, “yes”. Every vendor’s default deployment is an agent inside a disposable, pre-authenticated VM/container/browser (Anthropic Linux container, OpenAI cloud browser + container sandboxes, Microsoft Windows-11-VM tooling, Google Antigravity Linux sandbox, Artemis over an authorized ADB device). Inside that environment the agent can control everything it can see — which is the point — but only the sandbox’s contents. No vendor and no OS ships a path for an unprivileged agent to silently see and drive 100% of a user’s personal machine.

7. Where the community is building towards

  1. Protocol stack consolidation — with grounding still unsolved. MCP spec 2026-07-28 with Tasks/Apps/auth extensions (https://modelcontextprotocol.io/specification/2026-07-28); A2A v1.0 (12 Mar 2026) joined the Linux Foundation-directed Agentic AI Foundation on 27 Aug 2026 alongside MCP/goose/AGENTS.md (https://a2a-protocol.org/latest/blog/2026/08/27/a-new-chapter-for-a2a-joining-the-agentic-ai-foundation/ https://aaif.io/projects); AG-UI is the Agent-User Interaction protocol (agent↔frontend events; integrated by Microsoft Agent Framework, Google ADK, Claude Agent SDK, LangChain…) — “MCP gives tools, A2A connects agents, AG-UI brings agents into user-facing apps” (https://github.com/ag-ui-protocol/ag-ui). Correction worth repeating: there is no cross-vendor “accessibility-graph UI” wire spec — reading a screen stays per-OS accessibility trees + vision models, and researchers still propose MCP-driven a11y standardization (arXiv 2608.24898).
  2. Computer use is a first-class tool category in every major SDK. OpenAI Agents SDK v2 (15 Apr 2026: “native sandboxed execution and a model-native harness”, https://openai.com/index/the-next-evolution-of-the-agents-sdk/) with AgentKit partially rolled back (Agent Builder/Evals wind down 30 Nov 2026 → Agents SDK + “Workspace Agents in ChatGPT”, https://openai.com/index/introducing-agentkit/); Google ADK 2.0 (https://adk.dev/integrations/computer-use/); Anthropic Claude Agent SDK (https://code.claude.com/docs/en/agent-sdk/overview); Microsoft Agent Framework (~13.4k★, AG-UI integration, https://github.com/microsoft/agent-framework); LangChain langgraph-cua-py. Coding agents treat computer use as a fallback tool behind code/CLI/API tools — “when Claude doesn’t have access to the tools it needs, it will point, click, and navigate” (https://claude.com/blog/dispatch-and-computer-use).
  3. The harness movement expands past the browser. browser-harness moved under the browser-use org (17.4k★) and the org is diversifying into windows-harness, browsercode, a desktop app, cloud SDK (https://github.com/orgs/browser-use/repos). No “Steel-for-desktop” (hosted desktop sandbox) equivalent found (unverified-absence); enterprise vectors are Windows 365 for Agents and VNC-based open stacks; Browserbase raised a $40M Series B as browser infrastructure for agents (https://www.browserbase.com/blog/series-b-and-beyond).
  4. The OS-agent race. Microsoft positions Windows as the “first agentic OS” (https://www.howtogeek.com/what-is-an-agentic-os-and-why-microsoft-thinks-windows-will-soon-do-your-work-for-you/); Google killed standalone Project Mariner and folded it into Gemini Agent/Chrome Auto-Browse + an enterprise Agent Platform, while Samsung/Google preview agentic Android on Gemini 3; OpenAI pushes workspace agents; Anthropic ships Claude Cowork + Dispatch; Apple stays intent-based. Reading: native intents are the platform; GUI-driving matters exactly where intents/APIs don’t exist — legacy apps, cross-app workflows, third-party sites.
  5. Near-term shape (synthesis): accessibility-graph-first hybrids (semantic trees where they exist + screenshot/VLM fallback), computer use as one universal escape-hatch tool, per-OS a11y graphs proprietary while MCP/A2A/AG-UI become neutral under Linux Foundation governance. Per-OS position: Microsoft strongest enterprise (OS + Cloud PC sandbox + framework + benchmark); Google strongest consumer-mobile agent layer (but retreating from standalone browser agents); macOS’s strongest desktop runtime is third-party (Anthropic over a11y APIs); Apple/iOS locked down to intents; Linux/open-source is where most building happens.

8. Caveats / unverified

  • Claude Cowork specifics unverified (product page exists, https://claude.com/product/cowork body not loadable this session). OpenAI “AgentKit” identity — only launch/rollback posts found. Gemini computer-use token pricing not independently verified (https://ai.google.dev/gemini-api/docs/pricing).
  • OSWorld-2.0 score conflict flagged, not adjudicated: paper-verified ~20.6% (Claude Opus 4.8) vs vendor claims 62–73% (arXiv 2606.29537 vs https://www.simular.ai/articles/sai-tops-osworld-2-0). AndroidWorld leaderboard numbers are community-submitted/self-reported.
  • “AWC / Android World Challenge” as a separate Google product unverifiable — the live artifact is the AndroidWorld leaderboard.
  • Project Mariner shutdown sourced to Wikipedia + tech outlets, not a Google blog found this session. Artemis ~99%/100% claims are self-reported README/leaderboard entries; repo is pre-release (created 13 Aug 2026, no releases, ~172★ on 7 Sep 2026).
  • Apple: several developer/support URLs are canonical slugs not machine-verifiable this session; “iOS 27 Siri Extensions” and any “iPad agent” product are single-source/unverified; “no Apple computer-use strategy” is an absence-of-evidence statement.
  • Aggregator-blog comparisons (andrew.ooo, agentrank.tech, agentmarketcap.ai, whysogeek, etc.) are community-reported marketing/secondary content. AskUI claims come from its own blog. OpenAI Codex “Remote” from iPhone is community-reported. “Accessibility Graph UI” does not exist as a spec — the real AG-UI is the Agent-User Interaction protocol (§7).

9. Key official sources