How To Learn Creating AI Agents: Design The Agent First, Then Choose Its Brain With Real Scores

How to learn creating AI agents is simple to describe, because you design one agent on paper and only then pick a tool and a model.

I am Julian Goldie, and I create agents every day and run GoldieBench, where I score AI models on one-shot builds.

Here is the path I would follow as a beginner.

1. You learn the nine parts every agent has.
2. You fill in a one-page agent brief for one small job.
3. You create the agent in a free tool with safe settings.
4. You test it on five real tasks.
5. You break it on purpose and fix it.
6. You compare brains, which is where benchmark scores help.

Most of this page is about design, because design is what beginners skip.

The model scores come near the end, because the brain is one choice out of nine.

What creating an AI agent actually means

An AI agent is a model that has been given a job, instructions, tools and something that starts it.

A chatbot answers a question and stops.

An agent takes a job, decides on the next step, uses a tool, reads the result and carries on until the job is done.

Creating an agent means making nine decisions.

Every agent tool asks for these parts, even when it uses different names for them.

This guide covers designing and creating one agent.

My guide on how to learn building AI agents is the project ladder that follows it.

Part 1: the job

The job is the one outcome the agent delivers.

A good job is small enough to check with your own eyes in a minute.

"Help me with marketing" is too broad, because nobody could say when it was finished.

"Read the search queries I paste in and name the three I should write about" is a job you can check.

I teach one agent doing one job.

A second job means a second agent.

Part 2: the instructions and persona

The instructions say who the agent is, what the job is, what finished looks like and what it must never do.

They also say what to do when the agent is unsure, which should be to stop and ask.

The persona is the tone of voice.

StarNet turns the persona into one click, and in my video I talked through warm, dry, composed and unhinged.

You can change the personality at any time, so it is the least important choice on the list.

In Hermes Agent the persona and instructions sit in a file called SOUL.md inside the agent's profile.

Part 3: tools and permissions

Tools are what the agent can use, such as web search, files, email or a calendar.

Permissions are what it is allowed to touch.

StarNet's app catalog is labelled "add an ability", and it includes Gmail, Notion and GitHub.

Its agent panel shows access permissions for things such as the web and local files, and you can allow or remove each one.

My rule is that an agent gets only the access its job needs.

A first agent should have one or two tools and read-only access.

Part 4: memory

Memory is what the agent carries from one session to the next.

StarNet gives each agent a Memories tab that shows what it has remembered.

Hermes ships with two small files, which are MEMORY.md for the agent's own notes and USER.md for your preferences.

The built-in memory is enough for a first agent.

Part 5: the trigger

The trigger is what starts the agent, and there are three you will use.

You should start every agent on chat, because you can watch each reply.

You add a schedule only after the agent has passed its tests.

Part 6: approvals and guardrails

Approvals decide when the agent must check with you first.

StarNet offers two modes when you create an agent, which are "ask for approval" and "full power".

I would start every new agent on ask for approval.

You watch what it tries to do for a few sessions, and you loosen the leash once you trust it.

Hermes still asks before anything risky, and the hermes pause command is an emergency stop for scheduled jobs.

The one-page agent brief

You fill this in before you open any tool.

If the job line will not fit in one sentence, the agent is not ready to be created.

A worked example: creating one agent end to end

Here is the brief for a small SEO helper, which is what I told my own StarNet agent I wanted help with.

Creating it in StarNet

StarNet is a free, open-source agent harness that shows your agents in a pixel-art space station.

Its creation screen follows the brief closely.

1. You give the agent a name.
2. You look at its appearance and change it if you want to.
3. You pick the personality.
4. You choose the approval level.
5. You connect a brain by signing in or by pasting an API key.
6. You pick the model and the reasoning level, which runs from low through medium and high up to max.
7. You create the agent, and a "waking up" animation plays.

I signed in with my Grok account in one quick step.

My agent's first line was "I'm awake. Let's give the station a direction."

It then asked me what made me want to set up an AI agent, and I told it SEO and gave it aiprofitboardroom.com.

After that you open the agent's detail panel, paste your instructions into the instructions field and set the access permissions.

I will repeat the caveat from my video.

I think StarNet is fun to play with, and I do not think there is a massive amount of practical use cases yet.

There is also a slight delay when you send a message.

It is still a very good place to learn, because every design choice is a visible button.

My StarNet AI agent post looks at the models behind the station.

Creating the same agent as a Hermes profile

Hermes Agent is an open-source agent from Nous Research that runs on your own machine.

You create the same agent with three commands.

hermes profile create scout

hermes -p scout setup

hermes -p scout

The first command makes an isolated profile, the second gives it a model and the third opens a chat.

A profile is a folder with its own settings, memory and plugins.

I keep one profile per model, and that habit matters for the model section below.

Design choices and what to pick as a beginner

Five agents worth creating first

The daily recap is a real Hermes example from my own guides.

hermes cron create "0 6 * * *" "Read my task list and send me a short to-do recap for today" --name "Daily 6AM To-Do Recap" --deliver telegram

The schedule is the trigger, the prompt is the job and the delivery target is where the result lands.

How to practise: create, break, fix

You learn creating agents by repeating a three-move loop.

You create the agent from the brief in one sitting.

You then break it on purpose.

• You give it a vague request and see whether it asks or guesses.
• You ask for something outside its job and see whether it refuses.
• You remove a permission and watch what it does when a tool is missing.
• You lower the reasoning level and compare the answers.
• You swap the brain for a different model and run the same five tasks.

You then fix one thing at a time and run the same five tasks again.

If you change three things at once, you will never know which change helped.

The last move in the break list is a small benchmark of your own, and that brings us to the models.

Choosing the brain: what GoldieBench scores can and cannot tell you

Most agent tools do not ship their own model.

StarNet and Hermes Agent both ask you to connect a brain, so the model is your decision.

GoldieBench scores one-shot builds, which means a model gets one prompt and one attempt, and the result is scored from 0 to 10.

That is a fair measure of raw building ability.

It does not measure long tool-calling loops, memory or how pleasant a model is to chat with.

My own StarNet agent is a good example of that gap.

I chose Grok 4.6 over Grok 4.7 for it, because I actually prefer Grok 4.6 for that kind of chat.

On the board, Grok 4.7 averages 7.15 across 20 tasks and Grok 4.6 averages 5.95 across 20 tasks.

So the model I preferred for a chatty station agent is the lower scorer on one-shot builds.

Both things are true, because they measure different jobs.

The lesson for a beginner is to match the brain to the job in your brief.

You can see the gap between build scores and agent use in the last row as well.

Agents-A1 scores 4.83 on one-shot builds, but it is tuned for tool calling and runs at about 95 tokens a second on my 36GB Mac.

That speed keeps a simple agent loop snappy, which a build score will not show you.

The local models page lists the models you can run on your own machine.

My best Hermes Agent model post ranks the brains I use inside Hermes.

How to test a brain yourself

You do not need my leaderboard to choose a model for your agent.

You need your five test tasks and two profiles.

You create one Hermes profile per model, so each brain gets a clean agent with the same instructions.

You run the same five tasks on both.

You score each result as good, fixable or wrong.

You keep the cheaper brain if both pass.

That is a tiny benchmark built around your own job, and it will beat any general score for your use.

My rule is that you change the brain only when the agent fails the same test twice with good instructions.

A weak brief will fail on every model.

Common mistakes when you create your first agent

The first mistake is choosing the model before you have written the job.

The second mistake is starting on full power because approvals feel slow.

The third mistake is switching on every tool in the catalog.

The fourth mistake is scheduling an agent that nobody has watched in chat.

The fifth mistake is blaming the model when the instructions were vague.

If the word agent is still new to you, how to learn AI agents from scratch is the better starting point.

If you want a structured course, the best AI agent course for beginners compares your options.

My free community, the AI Money Lab, is a good place to share your first brief.

Also On Our Network

• Agent Operatorsthe nine-step operator walkthrough for creating a first agent
• AI Income Deskhow a business owner creates an AI agent without code
• AI Tool Verdictevery agent design choice rated, with a verdict
• agentos.guidethe one-page agent brief inside an Agent OS
• aiprofitboardroom.comJulian's free path for learning AI agents

Write the brief, create one agent, test two brains on your own tasks, and you will have a real answer to how to learn creating AI agents.