⭐ Get the Agent OS + join 3,400+ founders inside the AI Profit Boardroom → Join AIPB

GoldieBench Blog · 6 min read

Hermes Agent Bitwarden And Which AI Models To Keep Keys For, With Real GoldieBench Scores

The vault holds one secret per provider, so pick models first. Here are the live GoldieBench averages and how to turn them into a small, easy-to-rotate project.

Illustrated hero: a gold vault on a dark navy background linked by thin gold lines to a glowing leaderboard of horizontal bars

Hermes agent bitwarden setup decides where your model keys live, and GoldieBench scores decide which of those keys are worth storing in the first place.

Bitwarden Secrets Manager hands Hermes each provider key at startup, named exactly as the variable Hermes reads, such as OPENROUTER_API_KEY or ANTHROPIC_API_KEY.

Below are the scores I read on the live GoldieBench models page on 14 November 2026, and how to turn them into a tidy Bitwarden project.

If you would rather my team matched models to your agent work, book a free SEO strategy session.

Why Model Choice Shapes Your Bitwarden Project

Every model you run needs a provider key, and each key becomes one secret in the project.

The Hermes Bitwarden guide says the secret Name becomes the environment variable, so the project is really a list of the providers you trust.

Fewer providers means fewer secrets to rotate. Picking models first keeps the vault small.

The GoldieBench Scores I Used

ModelAvg scoreTasksGold medalsListed providerPage
Fusion8.59/104721OpenRouterFusion
Claude Opus 58.27/105013AnthropicClaude Opus 5
Hermes MoA8.17/10473Hermes, Mixture of AgentsHermes MoA
GPT-5.6 Sol8.16/10502OpenAIGPT-5.6 Sol
Grok8.09/10475xAIGrok
MiniMax M37.97/10472MiniMaxMiniMax M3
Kimi K37.89/10508Moonshot AIKimi K3
GLM-5.27.77/10475Zhipu / Z.aiGLM-5.2

Averages, task counts and gold medals are copied from the models page. Each name links to that model's own page.

The provider column is the label the page shows under each model, which tells you which key family it belongs to.

What A GoldieBench Score Means Here

The models page describes the method: the same fixed prompts, one shot each, scored 0 to 10.

That makes the average a fair read on how well a model handles a build in one go, which is what most Hermes jobs ask for.

It does not measure secrets handling. Bitwarden behaves the same whichever model sits behind it, because Hermes loads the keys before any model is called.

One Key, Several Top Models: The Aggregator Route

Fusion leads the page at 8.59 across 47 tasks with 21 gold medals, and its listed provider is OpenRouter.

The page describes it as a multi-model panel of Fable 5 and GPT-5.5, ensembled.

If you route through an aggregator like that, one secret, OPENROUTER_API_KEY, can cover a lot of ground in your project.

That is the smallest possible vault for a strong score.

Direct Keys For The Top Single Models

Claude Opus 5 scores 8.27 over 50 tasks with 13 gold medals. GPT-5.6 Sol scores 8.16 over 50 tasks with 2 golds. Grok scores 8.09 over 47 tasks with 5 golds.

Each sits under a different provider label, so running all three direct means three secrets.

Pick the one or two that match your work and leave the rest out of the project until you need them.

Mid-Table Models For High-Volume Profiles

MiniMax M3 averages 7.97, Kimi K3 7.89 and GLM-5.2 7.77 on the same page.

Kimi K3 collected 8 gold medals across 50 tasks, more than several higher-averaged models, so it wins individual builds more often than its average suggests.

These suit profiles that run many routine jobs, where a slightly lower average is an easy trade.

MiniMax M3 is listed by its page with a 1M context window, which helps profiles that read long documents.

Give Each Profile Its Own Model Key

When different profiles use different providers, profile aliasing in the Hermes secrets overview keeps one project serving all of them.

A secret named with a profile suffix, for example OPENROUTER_API_KEY_RESEARCH, fills OPENROUTER_API_KEY when Hermes runs as the research profile.

A secret stored under the plain name always wins over an alias, so pick one pattern per key.

Confirm Which Key Each Model Is Using

After setup, open hermes model. Hermes labels injected keys with their origin, and Bitwarden keys show "(from Bitwarden)".

If a key you expected shows no label, check the secret name in the project for a typo before blaming the model.

Run hermes secrets bitwarden status yourself to confirm the token is valid; the agent's own terminal never receives it.

Panel Models And The Keys Behind Them

Two of the top three entries are not single models.

Fusion is described on the page as an ensemble of two models, and Hermes MoA, at 8.17 over 47 tasks, as a panel of frontier models merged by a chair.

Panels can need more than one provider behind them, so open each model page and note every provider it lists before you write your secret names.

Getting that list right first saves a second round of edits to the project.

Keep The Token Away From Benchmark Scripts

If you run your own side-by-side tests through Hermes, the scripts the agent writes run in a sanitised environment.

Hermes strips BWS_ACCESS_TOKEN from the terminal, execute_code and cron scripts, and provider credentials cannot be declared for passthrough.

That is good for safety, and it means a test script should call models through Hermes rather than expect raw keys in its own environment.

A Starter Project Layout

For most readers, a first Bitwarden project needs only two or three secrets.

One aggregator key, such as OPENROUTER_API_KEY, reaches the top-scoring ensemble on the page.

One direct key for your main single model, such as ANTHROPIC_API_KEY, covers the profile you lean on most.

Add a third only when a profile genuinely needs a different provider, and name it with a profile suffix if it should stay with that profile alone.

Rotate Model Keys In One Place

Swapping a provider key is a change in the Bitwarden web app. With the default override_existing: true, every machine uses the new key at its next start.

If the machine-account token itself expires, hermes secrets bitwarden token tests the new one before saving it.

Bitwarden's Secrets Manager CLI docs confirm the same token model on their side.

Keeping logins in 1Password rather than Bitwarden? Read the Hermes 1Password guide.

My Verdict On Models For A Bitwarden Setup

Start small: one aggregator key for the top of the table, plus one direct key for the single model your heaviest profile runs.

Add mid-table keys only for high-volume profiles, and re-check the models page before each change, because the averages move as new tasks are scored.

Read the Hermes CLI reference for the setup flags if you are adding the vault to a server.

Also On Our Network

Choose your models from the scores, store one secret per provider, then talk to my team for free or get the templates inside the AI Profit Boardroom and finish your hermes agent bitwarden setup.

FAQ

Which AI model scores highest on GoldieBench right now?

Fusion, at 8.59/10 across 47 tasks on the models page read on 14 November 2026. Its listed provider is OpenRouter.

Does the model I pick change how Hermes uses Bitwarden?

No. Hermes loads keys from Bitwarden at startup before any model is called; the model only decides which key you need.

How many Bitwarden secrets do I need for several models?

One per provider key. An aggregator key can cover many models, while direct keys for Anthropic, OpenAI and xAI need one secret each.

What do the GoldieBench numbers measure?

Fixed prompts, one shot each, scored 0 to 10, per the models page. They do not measure secrets handling.

How can I check a model is using the Bitwarden key?

Open hermes model; keys loaded from the vault are labelled (from Bitwarden).

Can different profiles use different model keys from one project?

Yes. Name the secret with a profile suffix and profile aliasing fills the standard variable for that profile.

The same stack Julian uses

Run this stack yourself.

Every demo on this bench was built inside the Agent Operating System — one prompt, one shot, single HTML file out. The Agent OS, the prompts, the templates, the weekly walkthroughs and 3,400+ founders shipping with it every day all live inside the AI Profit Boardroom.

3,400+founders
258documented wins
38countries
4coaching calls a week