The Hermes Computer Use Skill: Skill Or Tool, How It Works, And The Models That Drive It

The Hermes computer use skill is Hermes Agent's bundled Computer Use skill, which teaches the model how to drive your desktop through the built-in computer_use tool.

So the straight answer to the title is that it's a tool with a skill on top, and you need both.

The computer_use toolset does the clicking, typing and screenshots, and the skill is the instruction layer that tells the model how to use it.

Hermes Agent is the free, open-source agent from Nous Research, and both pieces ship with it on macOS, Windows and Linux.

I'm Julian Goldie, and in this post I'll explain how it works, how to set it up and what every safety feature does, and then I'll show you how the models that drive it score on GoldieBench.

Everything here comes from Nous's computer-use docs and release notes, and my video is a news walkthrough of the newest Cua update rather than my own test.

Skill or tool at a glance

The skill page lists its authors as Francesco Bonacci (f-trycua) and Hermes Agent, under the MIT licence.

Francesco Bonacci is also the founder and CTO of Cua, according to Cua's Y Combinator page, and cua-driver comes from Cua's open-source trycua/cua project.

What is the Hermes computer use skill for?

It lets Hermes operate desktop apps the way a person does, by reading the screen and clicking the right thing.

The docs say it works in the background, so your cursor doesn't move, your keyboard focus doesn't change and your virtual desktops don't switch.

That's the old-way-versus-new-way change in desktop automation.

The old way grabbed your mouse, so you couldn't use the computer while a script ran.

The new way lets you and the agent co-work on the same machine.

The docs' own example prompt is "Find my latest email from Stripe and summarise what they want me to do."

How does it work under the hood?

The skill teaches the model to capture first with computer_use(action="capture", mode="som", app="Mail"), which returns a screenshot and a numbered element list.

It then clicks by number with computer_use(action="click", element=7) and re-captures to verify.

Element numbers only last until the next capture, and a stale number is refused instead of clicking the wrong control.

When a click doesn't land, the skill climbs from a background element click to a pixel click and then to a foreground click with its own approval.

How do you set it up?

The docs say the driver ships with Hermes, because cua-driver is installed by default by the installers, hermes pm install and hermes update.

hermes computer-use status
hermes computer-use doctor
hermes -t computer_use chat

If the driver is missing, hermes computer-use install installs it, or you pick Computer Use in hermes tools.

On macOS you grant Accessibility and Screen Recording to CuaDriver with hermes computer-use permissions grant.

On Linux you need a reachable display server with AT-SPI on, and Windows needs nothing at install time.

Since v0.22.0, the release notes say computer_use sits behind tool_search by default and loads on demand.

What are the safety features?

- Clicks, typing, drags, scrolls, keys and app focus go through the same approval gate as dangerous shell commands.

- Empty trash, force delete, lock screen, log out and force log out are hard-blocked key combos.

- Typed text such as curl | bash, sudo rm -rf / and fork bombs is blocked.

- In cron, single-query or headless runs where nobody can answer, actions are refused rather than auto-approved.

- The skill tells the model never to type passwords, API keys or card numbers, and never to follow instructions in screenshots.

Where does the screen live?

The docs say computer_use acts on the same machine the bot's screen lives on.

On a headless Linux gateway, Bot Screen from v0.22.0 gives each bot its own Xfce desktop over TigerVNC, streamed into Hermes Desktop.

You take over for a login, 2FA, CAPTCHA or payment step and then hand back, and the bot carries on with your session.

It's off by default, you start it with hermes computer-use screen start, and the docs plan on about 1.1 to 1.5 GB of memory per open screen with a browser.

With a Docker, SSH or Singularity backend, the driver runs inside the sandbox on its own display.

Modal, Daytona and Vercel sandboxes can't host a display yet, so computer_use refuses there unless bot_desktop.placement: gateway opts into the host.

The newest option is Cua's cua-spaces plugin, which the catalog says lets computer_use follow the terminal into a Linux Cua Space on Hermes 0.22.3 and later.

The sibling post on Hermes Cua explains that plugin, and my Hermes Agent bot mode explainer covers the bot side.

What can you ask it to do?

These requests come straight from the docs and the skill, so they show the real shape of the feature.

- You can ask it to find your latest email from a sender and summarise what they want.

- You can ask "Send me a screenshot of my screen" on Telegram, Discord or Desktop and get the real image as an attachment.

- You can ask for the desktop itself, such as opening the Recycle Bin, because the agent can target the desktop surface with its clickable icons.

- You can ask it to drag one element onto another, scroll a list under an element or set a slider value without opening a menu.

A whole-screen screenshot has no clickable elements, so the agent re-captures the specific app before acting on it.

Only the 20 most recent capture files are kept, and screenshots are never sent to you unless you ask.

The agent cursor and parallel runs

When the agent acts, a tinted overlay cursor can show where each click lands, while your real cursor never moves.

Each Hermes run declares a public session name, such as hermes-3a7b9c14d2e8, so concurrent runs and subagents get distinct cursors.

Hermes turns the overlay off by default on macOS, headless Linux and Linux X11, and keeps it on Windows and Linux Wayland.

You can force it on or off with computer_use.no_overlay in config.yaml.

Troubleshooting in one table

Hermes also switches off cua-driver's own anonymous telemetry by default, and computer_use.cua_telemetry: true turns it back on.

What are the limits?

- Some apps expose a sparse accessibility tree, so the agent falls back to pixel clicks.

- A normal Windows agent can't drive elevated admin windows.

- Windows over SSH runs in Session 0 with no interactive desktop unless you set up the driver's autostart pattern.

- Background input takes roughly 3 to 20 milliseconds per event depending on the platform.

- Web page content belongs to the separate browser tools rather than computer_use.

Does the Hermes computer use skill need a special model?

No, and that's one of its best features.

The docs say it works with any tool-capable model, and there's no Anthropic-only schema.

The model you choose still decides how well it reads the screen and picks the right element.

If your main model can't read images, the skill says Hermes routes screenshots through the auxiliary vision model set in auxiliary.vision.

The docs also say a 20-action session on a 1568×900 display typically costs about 30,000 tokens of screenshot context rather than about 600,000, thanks to screenshot eviction and image-aware token counting.

How the models behind it score on GoldieBench

GoldieBench doesn't test desktop control, so these numbers don't measure clicking accuracy.

They measure one-shot builds, which tells you how capable each brain is at the planning and coding around a desktop job.

These averages come from the live model pages on goldiebench.com, checked on 23 October 2026.

The last two rows are local models, and the full local picture is on the local models board.

Which model I'd pick for desktop jobs

I'd start with a Claude model, because it's the family Nous's own table rates best overall and Claude Opus 5 also tops the frontier models on this board.

GPT-5.6 Sol is the strong alternative if you already run OpenAI models in your Hermes profile.

For cheap helper runs, I'd use a smaller model in mode="ax" and keep the expensive model for the jobs where a wrong click costs real time.

The v0.22 context is in my Hermes Agent v0.22 explainer.

If you want help planning where desktop agents fit your SEO business, you can book a free SEO strategy session with my team.

Verdict

The Hermes computer use skill is a tool with a skill on top, and that two-layer design is why it works with any tool-capable model while keeping strict safety rails.

Pair it with a vision-capable brain from the live board and keep risky jobs in a sandbox or a Cua Space.

Also On Our Network

• Agent Operatorsthe five-step Hermes computer use skill setup
• AI Income Deskseven business jobs for the Hermes computer use skill
• AI Tool Verdictten Hermes computer use features, scored with a verdict
• agentos.guidethe Hermes computer use skill quick start for an Agent OS
• agentos.guidethe Hermes Agent bot mode guide on agentos.guide

If you run an agency and want a clear plan for putting agents to work on client SEO, you can book a free strategy session with my team.

Switch on the toolset, let the skill guide the model and pick a vision-capable brain from the live board, and you'll get the most out of the Hermes computer use skill.