GoldieBench Blog · 6 min read
Hermes Vault + Which AI Models To Pair With It: Real GoldieBench Scores For Logged-In Agent Work And Unattended Runs
The vault keeps passwords out of every model, so the choice comes down to the work. Here are six live GoldieBench scores and where each fits.
The Hermes vault keeps every password out of the model's context, so the AI model you pair with it changes how well the agent works behind a login, not how safe the login is.
That splits the model choice into two questions: which model navigates sign-in flows and logged-in pages well, and which one suits the jobs that run while you're away.
Below are the GoldieBench scores I read on the live pages on 3 November 2026, and how each fits a vault-backed Hermes setup after the v0.22.6 release of 2 November.
Want my team to match models to your logged-in workflows? Book a free SEO strategy session.
Why Model Choice Doesn't Touch Password Safety
The Hermes Passwords & Logins docs say the password never enters the model's context through Hermes.
Not in tool results, logs, the session database or any process's command-line arguments.
After a fill, the model reads {filled_fields: 1, origin: "https://github.com"}, and the password is registered with the redactor so a page read can't echo it back.
So a top-scoring model and a mid-table one see exactly the same thing: labels, origins and usernames.
The origin check, the card approval prompt and the headless refusals all sit in Hermes, outside the model.
Pick the model for the work, and let the vault handle the secrets.
The GoldieBench Scores For A Vault-Backed Shortlist
| Model | Avg score | Tasks | Gold medals | Page read |
|---|---|---|---|---|
| Claude Opus 5 | 8.27/10 | 50 | 13 | Opus 5 page |
| Hermes MoA | 8.17/10 | 47 | 3 | Hermes MoA page |
| GPT-5.6 Sol | 8.16/10 | 50 | 2 | GPT-5.6 Sol page |
| Grok | 8.09/10 | 47 | 5 | Grok page |
| Kimi K3 | 7.89/10 | 50 | 8 | Kimi K3 page |
| Gemini 3.6 Flash | 7.08/10 | 50 | 1 | Gemini 3.6 Flash page |
Fusion tops the GoldieBench models page at 8.59 across 47 tasks, but it's an OpenRouter multi-model panel, so I've left it off a list of single session models.
What These Scores Measure, And What They Don't
Every model on GoldieBench gets the same fixed prompts, one shot each, with a single self-contained HTML file out, scored 0 to 10 by Julian.
That's a read on first-try build quality.
No GoldieBench task tests signing into a site, picking the right saved login or reading a logged-in dashboard.
So treat the scores as a guide to general capability, then judge login work on your own sites.
A higher score suggests fewer wrong turns on a complex page, and nothing more specific than that.
The models in the AI Profit Boardroom Agent OS run as separate Hermes profiles, which is how I'd test them against your own logged-in pages.
Give each candidate the same chore on the same site, then compare how cleanly each one finished.
Claude Opus 5 And GPT-5.6 Sol: For Interactive Login Work
Opus 5 averages 8.27 across 50 tasks; GPT-5.6 Sol sits at 8.16 across 50.
These suit the sessions where you're at the keyboard, because that's where the vault asks for things.
A first-time login brings up the masked save prompt.
A text-message code brings up the verification prompt.
A card fill brings up the approval prompt, every time.
A strong model moves through the page while you answer those, and you approve anything that spends.
Hermes MoA: The Panel Still Sees No Password
Hermes MoA scores 8.17 across 47 tasks, and its GoldieBench page describes it as a panel of frontier models merged by a chair.
More models reading the same session doesn't widen what they can see.
The fill result only carries the field count and origin, so there's no password in the context for any panel member to read.
Use MoA where the hard part is the reasoning on a logged-in page, such as comparing figures across several dashboards.
Its page lists the default panel as Opus 4.8 and GPT-5.5, run from the Mixture tab in the Hermes Agent OS.
Kimi K3: For Unattended Runs, With Vault Caveats
Kimi K3 scores 7.89 across 50 tasks with 8 gold medals.
Its GoldieBench page says it's wired into the Agent OS as the kimi-k3 Hermes profile for long unattended agent runs.
Unattended runs are exactly where the vault's headless rules apply.
- Saved local logins keep working.
- A locked 1Password or Bitwarden returns
unavailable_in_this_session. - A site with no saved login returns
prompt_unavailable. - Card fills are refused, because nobody can approve them.
Save every login the job needs from an interactive session first, or give 1Password a token through OP_SERVICE_ACCOUNT_TOKEN.
Grok And Gemini 3.6 Flash: For Quick Checks
Grok scores 8.09 across 47 tasks, and Gemini 3.6 Flash scores 7.08 across 50.
Gemini 3.6 Flash's page tags it faster, cheaper and lighter on tokens.
That makes it a fit for short logged-in chores, like opening a portal and reporting one status line.
Because the vault fills the login, a lighter model doesn't have to handle anything sensitive to do that job.
The v0.22.6 Keychain Option And Your Model Profiles
The v0.22.6 release adds opt-in OS-keychain storage for the vault key.
Check the current state with hermes vault key, then move it with hermes vault key migrate --to keychain.
It works with the keychain your OS already has: macOS Keychain, Windows Credential Manager, or a Linux Secret Service or KWallet store.
Each profile gets its own keychain entry, so a kimi-k3 profile and an Opus profile each keep their own key.
One catch for unattended profiles: a cron job firing while you're logged out has no access to the keychain entry, and Hermes reports the failed login rather than guessing.
Keep the file key on any profile that runs overnight, and use the keychain for the interactive ones.
Plugins Matter More Than The Model
Per the docs, an unreviewed plugin loaded inside Hermes holds every power Hermes holds, and that includes filling logins from the vault.
No model score changes that.
If you've installed plugins you haven't reviewed, set plugins.isolation: host from the plugins docs before you save important logins.
Camofox users get the same fills through its REST endpoint when CAMOFOX_URL is loopback or HTTPS; our Hermes Camofox model guide covers that browser.
My Pairing, In One Table
| Job | Model from the shortlist | Vault setting |
|---|---|---|
| Interactive logins and checkouts | Claude Opus 5 or GPT-5.6 Sol | Keychain key on your own laptop |
| Reasoning across logged-in dashboards | Hermes MoA | Either |
| Long unattended runs | Kimi K3 | File key, logins saved in advance |
| Quick status checks | Gemini 3.6 Flash or Grok | Either |
Want to run that pairing inside a ready-made Agent OS? Join the AI Profit Boardroom.
Also On Our Network
- Agent Operatorsthe six-step setup order from first saved login to keychain key
- AI Income Deskhow a shop, a service firm and an agency hand logins to an agent
- AI Tool Verdicta ten-criteria scorecard rating password isolation, 2FA and keychain storage
- agentos.guidefive copy-paste prompts that run the vault inside your Agent OS
- aiprofitboardroom.comthe complete Hermes vault guide on the AI Profit Boardroom blog
Pick one model per profile today, and book a free strategy session if you want help pairing models with your Hermes vault.
FAQ
Does a stronger model see more of my vault?
No. Every model sees the same labels, origins and usernames, and fill results carry only a field count and origin.
Which GoldieBench model should run overnight login jobs?
Kimi K3, at 7.89 across 50 tasks, is the one its GoldieBench page ties to long unattended runs. Save its logins in advance and keep the file key on that profile.
Does GoldieBench test sign-in tasks?
No. It scores one-shot builds of a single HTML file, so use it as a general capability guide and judge login work on your own sites.
Do separate model profiles share one vault?
No. The vault is scoped to the profile, and in keychain mode each profile has its own entry.
Keep reading
Hermes Camofox + Which AI Models To Run It With: Real GoldieBench Scores For Every Browser Job
Hermes Webapp + Which AI Models To Run In Your Browser Sessions, With Real GoldieBench Scores For Opus 5, GPT-5.6 Sol, Kimi K3 And Grok
Hermes Agent Hooks + Which AI Models To Pair With Them, With Real GoldieBench Scores For Opus 5, Hermes MoA, Kimi K3 And Gemini 3.6 Flash🌐 Sister-site take: read this on agentos.guide