⭐ Get the Agent OS + join 3,400+ founders inside the AI Profit Boardroom → Join AIPB

GoldieBench Blog · 6 min read

Hermes vs Devin: Which AI Models To Run In Hermes, Which Devin Tier To Pick, And What The GoldieBench Scores Say

Hermes takes any model, per job. Devin gives you tiers. Here is what the bench numbers mean for each side.

A chat-and-clock lane and a cloud code-server lane feeding a glowing benchmark leaderboard of gold bars on a dark navy background

Hermes vs Devin splits the model question in two: Hermes Agent lets you wire any provider into each chat, Bot and cron job, while Devin hands you a menu of agent tiers that Cognition tunes for software work.

So you pair models with Hermes, and you pick tiers with Devin.

This page shows what the GoldieBench scores mean for each side, using only numbers I read on live goldiebench.com pages on 7 November 2026.

GoldieBench doesn't score Hermes or Devin themselves.

It scores models, and only some of them can sit inside these tools.

If you'd like my team to build a model plan with you, get your Backlink Gameplan, or try Polar Rank free for 3 days.

Why the two tools treat models so differently

Hermes has no fixed model list.

You run hermes model and pick from cloud APIs such as OpenRouter and Anthropic or self-hosted endpoints such as Ollama and vLLM, per the Hermes providers page.

If you never pick one, v0.23.0 falls back to GLM-5.3, according to the release notes.

Devin's session picker lists agents, not models, per its first-session guide.

Lite covers simple, well-scoped tasks, Normal is the everyday default, and Ultra is for the hardest tasks.

Fusion pairs a frontier model with a lighter one, and SWE-2 is Cognition's own software engineering model, offered as a research preview.

Admins decide which of those your team can see.

So the two tools ask different questions: Hermes asks which model, and Devin asks how hard the ticket is.

What a GoldieBench score measures

The methodology page says every model gets the same fixed prompt, one shot, with the result saved as a single HTML file.

Julian scores each build from 0 to 10, and the leaderboard ranks models by their average across the tasks scored.

That's a test of one-shot building.

It maps loosely to a Devin feature ticket and hardly at all to a Hermes job that summarizes a week of notes.

The scores, as read on goldiebench.com

ModelAverageScored tasksPage read
Fusion (OpenRouter panel)8.59/1047/models/fusion
Claude Opus 58.27/1050/models/opus5
Hermes MoA8.17/1047/models/moa
GPT-5.6 Sol8.16/1050/models/gpt56
Claude Fable 58.10/1047/models/fable-5
Kimi K37.89/1050/models/kimik3
GLM-5.27.77/1047/models/glm

The averages and task counts come from the all-models page.

Don't mix up the two Fusions

The top row is OpenRouter's Fusion, a panel that the GoldieBench page lists as Fable 5 plus GPT-5.5, ensembled.

Devin's Fusion is a separate agent tier inside Devin.

The 8.59 belongs to the OpenRouter panel, so it says nothing about the Devin tier with the same name.

Hermes can reach OpenRouter as a provider, which is the only place that score applies here.

Picking a Hermes model per job

Hermes lets every job carry its own model.

A cron job resolves its model in order: a per-job pin first, then cron.model in config.yaml, then the main model, per the cron docs.

In Bot Mode, each Bot can pin its own provider and model, so different Bots run different models side by side.

That's where the table helps.

A Bot that builds pages or tools in one go is a fair match for a top scorer such as Claude Opus 5 at 8.27.

A nightly digest Bot doesn't need a one-shot builder, so a long-context model such as GLM-5.2 or Kimi K3 is a sensible test, with the caveat that GoldieBench doesn't score digests.

Hermes MoA, at 8.17, is a panel run from the Agent OS, with a default of Claude Opus 4.8 and GPT-5.5 aggregated by Opus 4.8, per its page.

Picking a Devin agent per ticket

With Devin, you match the tier to the ticket rather than the model to the job.

In Slack, keywords such as !lite, !ultra, !fusion and !swe2 choose the tier when you tag @Devin, per the Slack integration page.

A copy change on a client site fits Lite.

A tangled checkout bug fits Ultra.

GoldieBench has no page for SWE-2 or for Devin's tiers, so there's no score to quote for them.

Local models: a Hermes-only option

Hermes can point at a model on your own hardware, which Devin's cloud sessions don't offer.

The GoldieBench local models page ranks strictly on-device coders, with Qwable 5 27B Coder on top at 7.14/10 over 41 tasks.

That's below the cloud leaders in the table, so I'd keep local models for private, low-stakes Hermes jobs.

Two small-business pairings

A web agency.

  1. Client tickets go to Devin on Normal, and the tricky ones get !ultra.
  2. A Hermes Bot pinned to Claude Opus 5 drafts landing-page mock-ups as single HTML files, the kind of one-shot build GoldieBench measures.
  3. A Friday cron job on a long-context model such as GLM-5.2 drafts each client update, marked draft only, and the account lead sends it.

A bookkeeping firm.

  1. No Devin, because there's no codebase.
  2. Hermes runs one Bot on a self-hosted model for anything that should stay in the office.
  3. A second Bot on a cloud model drafts client reminders into Slack for a staff member to send.

How I'd trial a model in Hermes before trusting it

Scores are a shortlist, not a decision.

  1. Create a new Bot in Bot Mode and pin the model you want to try.
  2. Give it the same three real jobs your current Bot handled last week.
  3. Compare the outputs side by side in the Bot Chat, and keep notes on misses.
  4. Only then move a cron job over, by pinning that model on the job itself.

Because a per-job pin wins over cron.model, one trial job can't change the rest of your schedule.

Members of the AI Profit Boardroom share these trial notes on the weekly coaching calls.

What the bench can't tell you here

GoldieBench scores one-shot HTML builds, not pull requests, test runs or code review.

It has no page for SWE-2, Devin's tiers, or GLM-5.3, the Hermes fallback model.

It also can't tell you how a model behaves over a long unattended run, which is most of what a Hermes cron job does.

So use the scores to choose Hermes models for build-style jobs, and judge Devin by the pull requests it sends you.

When a vendor ships a new model, check the all-models page again before you switch, because the table above is a snapshot from 7 November 2026.

Pairing verdict

Pick Devin if you want Cognition to manage the models and you just pick a tier per ticket.

Pick Hermes if you want to choose the model per Bot and per cron job, including local ones.

Run both if you have code and customers: Devin tiers for tickets, GoldieBench-ranked models for the Hermes Bots.

For the editor version of this question, read Hermes vs Cursor model pairings.

To run the same stack, join the AI Profit Boardroom.

Pin one model per Hermes Bot, pick one Devin tier per ticket, and get your Backlink Gameplan if you want help pairing models for hermes vs devin, or try Polar Rank free for 3 days.

FAQ

Can I choose the exact model Devin uses?

Devin's session picker lists agent tiers: Lite, Normal, Ultra, Fusion and the SWE-2 research preview. Admins control which ones appear.

Is the 8.59 Fusion score about Devin Fusion?

No. The GoldieBench Fusion entry is an OpenRouter panel of Fable 5 and GPT-5.5. Devin Fusion is a separate agent tier with no GoldieBench page.

Which GoldieBench model should a Hermes build Bot use?

Claude Opus 5 has the highest single-model average in the table at 8.27/10 over 50 tasks, so it is a fair first pick for one-shot build jobs.

Has GoldieBench scored SWE-2?

No. There is no SWE-2 page on goldiebench.com, so there is no score to quote.

Can Hermes run a local model from the GoldieBench local board?

Hermes supports self-hosted endpoints such as Ollama and vLLM. The local board is led by Qwable 5 27B Coder at 7.14/10 over 41 tasks.

The same stack Julian uses

Run this stack yourself.

Every demo on this bench was built inside the Agent Operating System — one prompt, one shot, single HTML file out. The Agent OS, the prompts, the templates, the weekly walkthroughs and 3,400+ founders shipping with it every day all live inside the AI Profit Boardroom.

3,400+founders
258documented wins
38countries
4coaching calls a week