Computer Use — Claude Controls Your Screen

Computer use lets Claude open apps, click, type, and see your screen — for tasks no CLI or API can reach: native apps, simulators, GUI-only tools.

Research preview: macOS only in the CLI (Desktop app also supports Windows), Pro/Max plans, v2.1.85+, interactive sessions only, claude.ai auth required.

When Claude Reaches for It

Claude tries the most precise tool first:

1. MCP server for the service → use that
2. Shell-doable → Bash
3. Browser work + Chrome connected → Chrome integration
4. Nothing else works → computer use (native apps, iOS Simulator, hardware panels)

Enable

1. /mcp → find computer-use → Enable (persists per project)
2. First use prompts for macOS Accessibility (click/type) + Screen Recording (see screen) permissions; may need a Claude Code restart after granting

How It Works (and stays safe)

• Per-app approval per session — Claude lists which apps it wants; terminals/IDEs are flagged “equivalent to shell access”, Finder “can read/write any file”
• Control tiers: browsers & trading platforms = view-only; terminals/IDEs = click-only; other apps = full control
• Other apps are hidden while Claude works; your terminal stays visible and is excluded from screenshots (so on-screen text can’t prompt-inject the session)
• One session at a time — machine-wide lock until the session exits
• Esc anywhere aborts instantly; a macOS notification appears when Claude starts and finishes
• Screenshots are auto-downscaled (don’t change your resolution; enlarge in-app text if Claude can’t read it)

⚠️ Unlike the Bash sandbox, this runs on your real desktop. Approve only the apps the task needs.

Real Cases

Validate a native build:

Build the MenuBarStats target, launch it, open the preferences window,
and verify the interval slider updates the label. Screenshot it when done.

Reproduce a layout bug:

The settings modal clips its footer on narrow windows. Resize the window
until you reproduce it, screenshot the clipped state, then fix the CSS.

Drive the iOS Simulator (no XCTest needed):

Open the iOS Simulator, launch the app, tap through onboarding, and tell
me if any screen takes more than a second to load.

Troubleshooting

• “in use by another Claude session” → exit the session holding the lock
• Permission prompt loops → fully quit Claude Code after granting Screen Recording
• computer-use missing from /mcp → check macOS, v2.1.85+, Pro/Max via /status, interactive mode (not -p)

🧠 Quick Recall

• Trick: escalation order — “MCP → Bash → Chrome → screen.” Computer use is the last resort.

👨‍🏫 Teach It

• Tell: the story of building + click-testing a native app in one conversation.
• They’ll trip on: expecting it in the Windows CLI — macOS only (Desktop app adds Windows).

---

Learning path: ⬅ 47-gitlab-cicd.md · Index · ➡ 49-goal-and-advisor.md