Vercel Agent-Desktop: Free Computer Use for AI Agents, Without Screenshots

Your coding agent can already read files, run tests and rewrite whole modules from a terminal. The moment it needs to touch a real desktop app, though, it hits a wall. Most computer-use tools get around that wall by taking a screenshot, sending the image to a vision model and asking it to guess where the button is. Vercel Labs took a different route. Its agent-desktop, a free and open-source CLI, reads the operating system's accessibility tree instead, the same structured map screen readers use, and hands the agent a list of real controls it can click by reference. Version 1.2.0 landed on November 5, 2026 with MCP support, and the project passed 18,800 GitHub stars eleven days after its 1.0 launch. This review covers what it does, how the setup works, what it costs, and where it still falls short.

What Agent-Desktop Is

Agent-desktop is a command-line tool and a C library from Vercel Labs that gives an AI agent computer use on macOS, Windows and Linux. It does not contain a language model and it has no graphical interface of its own. Any agent that can run shell commands can call it, and the reasoning stays in the agent. The README puts it plainly: agent-desktop gives an AI agent computer use without screenshots or pixel matching.

The tool started life in February 2026 as a personal project by developer Lahfir, then moved into the vercel-labs organization on GitHub. Vercel's Chris Tate announced it as agent-browser, for your whole computer, a reference to Vercel Labs' earlier agent-browser tool for the web. Version 1.0.0 shipped on October 28, 2026, version 1.0.2 followed on October 30, and 1.1.0 landed on November 3. The project topped GitHub's daily trending list on November 3, adding 2,275 stars in 24 hours to reach about 10,800, according to coverage that day. By November 8 it stood at 18,884 stars with 1,384 forks, and npm recorded 53,251 downloads of the package in the month to November 6.

What makes it different from the screenshot approach is the input it gives the model. A screenshot is a grid of pixels the model has to interpret. An accessibility tree is a labeled structure: every button, text field, menu and list item with its role, its name and the actions it supports. The agent reads text, not images, which is why the tool can be cheaper on tokens and more repeatable than guessing coordinates from a picture.

How the Snapshot-and-Ref Loop Works

Every interaction follows the same three-step loop, and the loop lives in your agent, not in the tool. First the agent takes a snapshot of an app, which returns the accessibility tree as JSON. Every element the agent can act on gets a ref, a short identifier such as @s8f3k2p9:e8 that is tied to that snapshot. Second, the agent acts on a ref: click it, type into it, check it, scroll it. Third, the agent takes a fresh snapshot to check whether the UI actually changed. The docs warn that a successful action does not prove the UI changed, so the verify step is part of the design, not an optional extra.

Refs are strict about their scope. A bare @e8 with no snapshot id is rejected with an INVALID_ARGS error. If the UI changed between the snapshot and the action, the tool answers STALE_REF or AMBIGUOUS_TARGET, and the agent is expected to take a new snapshot and use its refs. Dense apps get a two-step treatment: snapshot --skeleton maps the app shallowly, then snapshot --root expands only the region the agent cares about. That matters because a full tree for an Electron app such as Slack can be large enough to eat a serious chunk of a context window.

The tool ships with 60 command names, 56 of them operational. They cover observing (snapshot, find, get, screenshot, list-surfaces), acting on refs (click, double-click, type, set-value, check, scroll and more), keyboard and mouse input, window management, notifications, clipboard, waiting and sessions. Four low-level commands, key-down, key-up, mouse-down and mouse-up, refuse to run until a daemon exists that can release a held key or button. That is a deliberate safety gap rather than an oversight: a tool that can hold a key down forever is a tool that can lock up a machine.

Ref actions are headless by default. A click on a ref goes through the accessibility API without moving your pointer or taking focus, so you can keep typing in another window while the agent works. The global --headed flag switches to physical input for the cases that need it. Text commands accept --stdin for piped text, keeping secrets out of process arguments.

What Is New in Version 1.2.0

Version 1.2.0, published November 5, 2026, is the release that turns agent-desktop from a CLI your agent shells out to into a native MCP server. Running agent-desktop mcp serves every command as an MCP tool over stdio, with tool names taken from the commands themselves, such as desktop_snapshot and desktop_click. Global flags given to the mcp command, such as --session, --headed or --trace, apply to every call. That means any MCP-compatible agent can drive a desktop without writing a line of shell glue.

The same release adds macOS list-row selection for rows only their container can select, draws the cursor overlay above menus and system panels, simplifies the README and repo docs, and documents the Homebrew install. Version 1.1.0, the release before it, was mostly Windows fixes and work on the on-screen overlay that shows where the agent is clicking and typing. The pace is worth noting: four releases in eight days from 1.0.0 to 1.2.0, which tells you the project is moving fast and that you should read the changelog before upgrading.

How to Set It Up

Installation is one command on most machines. The npm route is npm install -g agent-desktop, and Bun, Yarn and pnpm can install the same package. On macOS and Linux, Homebrew also works with brew install agent-desktop, now from homebrew-core. There is no official Scoop package for Windows yet, so Windows users take the npm route or download a native archive from GitHub Releases and keep its helper beside the executable. Each release also ships libagent_desktop_ffi, a C library, so a program can call agent-desktop in process instead of shelling out.

Permissions are the step people trip over. On macOS the terminal app that launches the CLI needs an Accessibility grant, and agent-desktop permissions --guide walks you through it. Screenshots need a separate Screen Recording grant. On Windows there is no grant at the terminal's own integrity level, but an admin app can be read and not driven until the terminal matches it, and type needs --headed. On Linux the requirement is Ubuntu 24.04 or later with AT-SPI, on x86_64 or aarch64. The supported floor is macOS 13 or later, Windows 10 1809 or later, and Ubuntu 24.04 or later, on x64 or ARM64.

To teach an agent the tool, run agent-desktop skills get desktop for the loop, refs, verification and retry rules, and agent-desktop skills get platform for what differs on your OS. You can install the skill into an agent harness with npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop, or paste the two lines into your AGENTS.md. For MCP setups, add agent-desktop to your MCP servers with the command agent-desktop and the args mcp, and the full command set appears as tools.

A first run looks like this: agent-desktop status reports the platform, permissions and supported surfaces. Then agent-desktop snapshot --app Finder -i lists the interactive elements with refs, agent-desktop click @s8f3k2p9:e8 acts on one, and a second snapshot confirms the result. Retry a failed command only when error.disposition.retry says it is safe, which is the tool's own guidance in its skill files.

What It Costs

Agent-desktop costs nothing. The repository is licensed under Apache 2.0, there is no paid edition, no subscription tier and no per-seat price. The npm package is free to install and the C library ships with every release. Your bill is whatever your agent costs: the tokens the model burns reading snapshots and the API or subscription charges of the agent itself. A skeleton snapshot is the cheapest way to look at a dense app, and the full-tree dump is the most expensive, so the token cost of a task depends on how the agent is told to look, not only on what it does.

That puts it in the same free tier as the open-source tools it sits next to, with one difference worth stating plainly. Some free tools charge later through a hosted edition. Agent-desktop has no hosted edition and no company price list behind it, so there is no upgrade path to pay for and no free tier that can be taken away. The trade-off is that there is also no support desk: issues go to the GitHub repository.

The Token Numbers, Read Carefully

Vercel's own reporting says skeleton snapshots use 78 to 96 percent fewer tokens than full-tree dumps on dense Electron apps. The same reporting puts Apple Maps at 273 tokens per look with a skeleton against 3,811 for a full tree, and a simulated 52-step task at 31,000 tokens against 198,000. One write-up measured a Slack skeleton at about 3.6KB against about 17.3KB for the full tree. Treat all of these as what they are: self-reported numbers from the builder, with no independent computer-use benchmark behind them. Independent coverage of the tool makes the same point, and no outside evaluation of the token claims has been published.

What is verifiable from the repository is the mechanism, and the mechanism is the honest story. The agent reads text structure instead of images, so it pays for nodes, not pixels. A skeleton snapshot reads three levels deep, reports a children_count for cut containers and hands back a drill ref for the region you want. That is a real design choice that makes token cost controllable at all. It does not prove a 78 percent saving on your app, and dense apps with deep trees will still cost real tokens if the agent keeps dumping full trees. Prefer find for a known name, and set-value or check where you can, since those stay headless and cheap.

Agent-Desktop vs Screenshot Computer Use

The comparison is not a verdict. Screenshot tools can see anything on screen, including custom-drawn interfaces with no accessibility data, which is exactly where agent-desktop goes blind. Agent-desktop is steadier and cheaper where the app cooperates. If your workflow lives in standard desktop apps with proper accessibility support, the structured route wins on cost and repeatability. If it lives in games, canvas apps or custom-rendered interfaces, screenshots are the only route that works at all.

Our earlier coverage of phone-side computer use, Ghosthand vs Glasshand: 3 ways to give your AI agent an iPhone, covers the same problem on a phone. Agent-desktop is the desktop answer to the same question, and it is free where the phone tools carry their own pricing.

The Catches

The first catch is scope. Agent-desktop is a tool that agents call, not an agent. It embeds no language model, has no GUI and makes no decisions. If you do not already run a coding agent such as Claude Code, Codex or Cursor, this gives you nothing on its own. It also cannot see a UI that publishes no accessibility tree, and it does not match pixels, so custom-drawn apps are invisible to it.

The second catch is privacy. The README is blunt about it: snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content, and you should keep them out of public logs and uploads. An agent that can read your screen can read your email, your messages and your banking tab. Run it on a machine you control, scope its access, and do not paste its raw output into a public issue or a shared chat.

The third catch is platform unevenness. macOS needs the Accessibility grant and a separate Screen Recording grant for screenshots, and open-system-surface is unsupported there. On Windows, type needs --headed and admin apps resist driving until the terminal matches their integrity level. Linux support has gaps, and there is no Scoop package for Windows yet. Multi-agent sessions exist, with a cursor per agent, but clicks take turns: the overlay cursors move together while the actions run one at a time.

The fourth catch is the token bill on dense apps. A full Electron tree can cost enough tokens to run out of budget, and no percentage is published for the general case. The skeleton and drill-down workflow is the answer, but only if the agent is told to use it. An agent left to dump full trees on every step will burn tokens fast, and the tool will not warn you first.

Who Should Use It

Agent-desktop fits developers who already run a coding agent and want it to reach beyond the terminal into native apps: filing a bug through a desktop client, driving a settings panel, or checking a result in an app with no API. It fits agent builders who need a stable, scriptable way to operate a desktop without paying per screenshot, and it fits anyone building multi-agent setups, since the MCP server in 1.2.0 drops the whole command set into any MCP-compatible harness. Pairing it with persistent memory, such as the setup in our Beacon review, covers the other half of the problem: an agent that remembers what it did yesterday and can click through it today. The install is a single command and the license is Apache 2.0, so the cost of trying it is an evening, not a budget line.

It does not fit anyone who wants a finished agent with a chat window and a task list. There is no agent here, no model and no interface. It does not fit workflows built on custom-rendered apps with no accessibility tree, and it does not fit a shared or public machine, where the privacy warning in the README becomes a real risk. If you need a computer-use agent that simply works out of the box, a managed product is the honest choice. If you already have an agent and want to give it hands on your desktop for free, agent-desktop is the one to try first.

Is agent-desktop really free?

Yes. The repository is Apache 2.0, the npm package is free to install, and there is no paid edition, subscription or per-seat price. You pay only for the AI agent that calls it, through that agent's own model or subscription charges.

Does agent-desktop work with Claude Code?

Yes. Any shell-capable coding agent can call it, including Claude Code, Codex and Cursor. Install the bundled skill with npx skills add, or add the two skill commands to your AGENTS.md, and the agent learns the snapshot-and-ref loop. Version 1.2.0 also exposes every command as an MCP tool, so MCP-compatible agents can use it without shell commands.

Which operating systems does agent-desktop support?

macOS 13 or later, Windows 10 1809 or later or Windows Server 2019 and later, and Ubuntu 24.04 or later on Linux, on x64 and ARM64 machines. It uses the Accessibility API on macOS, UI Automation on Windows and AT-SPI on Linux.

Is it safe to let an agent control my desktop?

It is as safe as the agent driving it. The tool is headless by default, so ref actions do not move your pointer or steal focus, and four low-level commands refuse to run until a daemon exists to release held keys. The real risk is data exposure: snapshots and clipboard reads can contain private content, so keep them out of public logs and do not run it on a shared machine.

How is agent-desktop different from screenshot-based computer use?

Screenshot tools send an image to a vision model and ask it to guess where to click. Agent-desktop reads the operating system's accessibility tree and returns structured JSON with a stable ref for every element. That makes clicks repeatable and usually cheaper on tokens, but it cannot see apps that publish no accessibility tree, where screenshots still work.

Can I use agent-desktop on Linux?

Yes, on Ubuntu 24.04 or later with AT-SPI, on x86_64 or aarch64. Linux support has gaps compared with macOS and Windows, and some surfaces behave differently, so check the platform page in the docs before relying on it for a production workflow.

Sources

• agent-desktop on GitHub (vercel-labs/agent-desktop): README, releases, license and platform support, read November 8, 2026
• agent-desktop releases: v1.2.0 published November 5, 2026; v1.1.0 and v1.0.2 changelogs
• agent-desktop documentation (agent-browser.dev/desktop): install, commands, platform pages and MCP setup
• Code Sprint Consulting, "agent-desktop: Vercel's Open-Source Computer Use CLI": token figures and platform limits, November 2, 2026
• OP or Mid, "agent-desktop 1.0.2: OP on the tree, Mid as your whole computer": release dates and platform caveats, November 2, 2026
• Medium, "Vercel agent-desktop: Rust Computer Use for AI Agents": GitHub trending data and release timeline, November 2026
• TechPilot, "Agent Desktop by Vercel Labs: Free AI CLI": feature summary and token figures