⭐ Get the Agent OS + join 3,400+ founders inside the AI Profit Boardroom → Join AIPB

GoldieBench Blog · 6 min read

Hermes Agent Video Analysis: How To Set Up The Tool, The YouTube Skill And video-xray, With A Note On Models

Three routes, each with its own steps: enable the toolset and backend, summarize YouTube links with the bundled skill, or install video-xray, then pick the model behind it.

Illustrated hero: a dark navy panel of gold bars beside a filmstrip whose frames are lit by a glowing gold scan line

You set up hermes agent video analysis by enabling the opt-in video toolset with a video-capable auxiliary backend, by calling the bundled youtube-content skill for YouTube transcripts, or by installing the video-xray plugin for full reports on long files.

Below are the exact steps for each, what each one returns, the limits that trip people up, and a short note on which model to put behind the visual routes.

Hermes details come from the Hermes tools reference, the Hermes source code and the plugin's catalog entry.

If you'd rather my team set this up around your footage, book a free SEO strategy session.

What Hermes Can Do With Video

Hermes reads video; it does not watch it by itself.

Each route hands the work to something else: an auxiliary model for clips, a transcript for YouTube links, or your chat model for sampled frames.

  • Answer questions about a clip with video_analyze, which covers captions, scene breakdowns, key timestamps and visual descriptions.
  • Turn a YouTube video into text with youtube-content: summaries, chapters, quotes, threads and blog posts.
  • Break down a long recording with video-xray: windowed frame observation, a transcript, a Markdown report and subtitle files.

Step 1: Set Up The Built-In video_analyze Tool

  1. Add video to your toolsets: list, or launch with --toolsets video; the toolsets reference marks it opt-in.
  2. Configure the auxiliary vision provider under auxiliary.vision in the configuration docs.
  3. Put a model that accepts video in auxiliary.video.model, which the vision tools source checks before auxiliary.vision.model.
  4. Start a new session and ask a question about a short local file.

The tool only registers when an auxiliary vision client resolves, and the source states it has no native path to the main model.

An open request, issue 27313, asks for a fully separate video backend; until then the provider is shared with image vision.

Step 2: Summarize YouTube Videos With The youtube-content Skill

For public YouTube videos, the youtube-content skill is usually the fastest answer, because it works from the transcript rather than the pixels.

  1. Prepare the skill's youtube extra once, as its skill file explains.
  2. Run /youtube-content summarize with the link, or name another format such as chapters or quotes.
  3. Check the result against the video, as the skill's own workflow ends with a verify step.

Long transcripts, over about 50K characters, are split into overlapping chunks and summarized piece by piece.

Step 3: Install video-xray For Long Recordings

The plugin joined the catalog in the 15 November 2026 canary; its catalog entry shows version 0.2.2, community tier and Windows as the only verified platform.

  1. Use Hermes Desktop 0.22.1 or newer on Windows, with ffmpeg on the PATH.
  2. Follow the README install: copy the backend files, run hermes plugins enable video-xray and restart Desktop, then copy the panel file and reload.
  3. Open the panel with /video and switch off "Analyze attached videos on send".
  4. Pick a time window, 8 to 64 frames per segment and a speech engine, then run the job.

Cost scales linearly with frames per segment, per the README, so start low and raise it only for windows that need more detail.

What You Get Back From Each Route

RouteOutputSees on-screen detail?
video_analyzeWritten description plus an answer to your questionYes
youtube-contentSummary, chapters, quotes, thread or blog post from the transcriptNo
video-xrayMarkdown report, SRT and JSON cues, frame gallery, shot promptYes

Making clips is a different job; the Hermes Agent AI video generation guide covers it.

Videos Sent Through Messaging Apps

Clips do not have to start on your desktop.

The Telegram docs describe observed group chats, where a video posted without mentioning the bot is cached along with its saved path.

A later question about the clip makes the agent open the cached copy with video_analyze if the toolset is on, or with its file tools if not.

That means image and video tokens are only spent when a person actually asks a question.

The same canary build also changed Telegram so a comment and the video posted right after it arrive as a single turn.

For teams that swap clips in chat all day, this is often the most used route of all, and it needs nothing beyond Step 1 above.

A YouTube link dropped into the same chat can go to the transcript skill instead, which keeps frame analysis for the files that really need it.

Limits Worth Knowing

  • video_analyze accepts mp4, webm, mov, avi, mkv and mpeg, slows above 20 MB and stops near 50 MB.
  • Video processing is not available to a hosted agent, according to the source.
  • The YouTube skill needs captions on the video, and misses anything shown but not said.
  • video-xray starts a job for each attached video on send until you change that setting, sends frames to your chat model, and keeps a saved cloud speech key as readable text.
  • The plugin is maintained by LuuMoss in the community tier, not by Nous Research.

Common Setup Problems

  • Tool missing: the auxiliary vision provider is not set, so video_analyze never registered.
  • Video rejected: the auxiliary model takes images but not video; the Hermes error hint suggests a video-capable Gemini model.
  • Empty YouTube result: the video has transcripts disabled; retry without a language setting, as the skill suggests, then check the video page.
  • Workbench says the engine is not ready: send one normal chat message first, per the README.

Which Model To Set As The Auxiliary Backend

Only the visual routes depend on a model; the YouTube skill does not.

For video_analyze, the backend must accept video input, and the Gemini family is the documented starting point.

On the live GoldieBench leaderboard, read on 15 November 2026, Gemini 3.6 Flash averages 7.08 and Claude Opus 5 averages 8.27.

The Gemini 3.6 Flash page notes that game tasks are judged "on a real mid-play frame by the same Opus vision judge", and the Opus 5 page lists Fireworks Visual at 9.0.

The methodology is one fixed prompt, one shot, one HTML file, so these are building scores, not video scores.

For video-xray, use a vision-capable chat model; the vision docs name DeepSeek V4 Pro as text-only, so keep it out of that seat.

My Verdict On Getting Started

Enable the toolset with a Gemini-family auxiliary model first, and use the YouTube skill for every public link.

Add video-xray on Windows when long recordings need reports and subtitles.

Test any model on two clips you know well before trusting it with the rest.

Score the answers yourself out of 10, keep the model that gets your footage right, and recheck the leaderboard when a new model lands.

Also On Our Network

Set up the route your next video needs, test it on two clips, then ask my team on a free strategy call or compare notes in the AI Profit Boardroom as you tune your hermes agent video analysis.

FAQ

How do I make Hermes Agent analyze a video?

Enable the opt-in video toolset, configure an auxiliary vision provider with a video-capable model, then ask video_analyze a question about a file or URL.

Can Hermes summarize a YouTube video?

Yes. Run the bundled youtube-content skill with the link. It fetches the transcript and returns a summary, chapters, quotes, a thread or a blog post.

Which auxiliary model should I set for video?

One that accepts video input. The Hermes error hint points to a Gemini model, and Gemini 3.6 Flash averages 7.08 on GoldieBench.

Does GoldieBench test video understanding?

No. Every task is one fixed prompt, one shot and one self-contained HTML file, so use the scores only to shortlist.

Do I need video-xray?

Only for long local recordings on a Windows Hermes Desktop machine, when you want subtitles, timestamped reports and a frame gallery.

Can a text-only model run Hermes video analysis?

Partly. The YouTube skill and video_analyze still work, since the second uses its own auxiliary model, but video-xray needs a chat model that can see.

The same stack Julian uses

Run this stack yourself.

Every demo on this bench was built inside the Agent Operating System — one prompt, one shot, single HTML file out. The Agent OS, the prompts, the templates, the weekly walkthroughs and 3,400+ founders shipping with it every day all live inside the AI Profit Boardroom.

3,400+founders
258documented wins
38countries
4coaching calls a week