Testing and evidence¶
Every number below comes from a run, not from a claim. The live runs used Rocket.Chat 8.9.1, the official Hermes 0.23.0 image and Chrome driving the web client.
| What was checked | Result |
|---|---|
| Unit and contract tests in CI, on Hermes 0.22.3 (oldest supported) and 0.23.0 | 780 pass on both |
hermes plugins validate --install-deps |
Passes, no private core imports |
| End-to-end scenarios: mentions, threads, DMs, files both ways, concurrent askers, the command guard | 39 / 39 |
| Approval buttons clicked in a room and a thread, including another person's click doing nothing | 16 / 16 |
/new confirmation buttons |
10 / 10 |
| Approval button in a one-to-one DM | 4 / 4 |
| Gateway restart and network outage, with catch-up | 14 / 14 |
| Cron delivery through the running gateway, and with the gateway stopped | 2 / 2 |
Refused commands such as /yolo, /tools, /cron and unknown /words |
0 model calls |
@file: and @folder: from a room member, directly and hidden in an earlier message of an unverified sender |
0 files attached (before the fix: config.yaml attached, memories/ listed) |
| Real model (Nous Portal): room context, identity, threads, long answers, approval | Pass |
Personal assistant, a second Hermes with the personal preset and a real model: DM questions, memory across sessions, preset commands, /sethome and a delivered reminder, /yolo refused, other people and rooms ignored |
19 / 19 |
| Fully local: the same real-model checks against a self-hosted open-weight model on vLLM, no cloud model service (team bot: room context, identity, threads, long answers, approval; personal assistant: all of the above) | 11 / 11 and 19 / 19 |
Context references¶
Hermes expands @file: and the other @ references in every incoming chat message before the model runs (why it matters). The check, on the official Hermes 0.23.0 image with Rocket.Chat 8.9.1, read the turn text Hermes stored in its session database:
| Message | Before the fix | With the fix (default) | With ROCKETCHAT_TEAM_ALLOW_CONTEXT_REFERENCES=true |
|---|---|---|---|
bob, an allowed member: @hermes ... @file:/opt/data/config.yaml @folder:/opt/data/memories |
Both attached to the turn | Plain text, nothing attached | Both attached, as Hermes does by default |
carol, an unverified sender, writes @file:... without a mention; then alice mentions the bot |
Expansion attempted from the room context | Plain text, nothing attached | Plain text, nothing attached |
| Gateway start with the switch on | A warning in the log |
Hermes itself refused /opt/data/.env and auth.json before the fix too; everything else under the workspace was readable. The unit tests in tests/test_context_refs.py check the same three paths (message, room context, inserted document) and that Hermes's own parser finds no reference in the neutralised text.
How these were measured¶
- Unit and contract tests run in CI on every push. The contract tests import the Hermes release under test and check each member the adapter relies on, so a renamed or removed interface fails the build.
- Live runs use a real Rocket.Chat 8.9.1 in Docker (with MongoDB and NATS), the official Hermes Agent image and this plugin exactly as committed. Test users post through the Rocket.Chat REST API, and every button is clicked by Chrome in the Rocket.Chat web client, signed in as the person who clicks.
- Each check asserts an outcome you could see yourself: the answer arrives in the right room or thread and mentions its asker; a dangerous command's file is still on disk after another person's click and gone only after the asker's; a refused command produces no request at the model endpoint (a recording stub model counts them).
- Restarts and outages are real: the gateway container is stopped, or its connection to Rocket.Chat is cut, while people keep writing, then started again.
- Real-model runs use Nous Portal, and again a self-hosted model served by vLLM on an on-premises GPU. The personal assistant run uses a second Hermes instance with its own bot account.
- The demo video is recorded from such a run; nothing in it is mocked. The ChatOps clip is a real run too, against a simulated service in a sandbox.
You can repeat the core of it yourself: python3 try/check.py runs seven of these checks against the try-it stack with its scripted demo model, no account or key needed. The full live-run scripts depend on a local test server and are not in this repository.
Building it also turned up Hermes bugs, which the author reported upstream:
- #148433: a shutdown during a redirected turn overwrites an already streamed reply.
- #149169:
/loopand/heartbeatstay silent after a gateway restart until the chat gets a plain message. - #151542: a relative-path
rmnever asks for approval, while the same deletion with an absolute path does. - #148988, with a fix proposed in #149038: session previews in
/resumeand/sessionsshowed an adapter's context line instead of the person's message; found because this adapter gives every turn room context.
Continuous integration¶
Every push runs the unit and contract tests and the catalog validator against two Hermes releases: the oldest the plugin supports and the newest tested live. The contract tests fail when a Hermes release removes or renames a member the adapter relies on, so a breaking change shows up in CI before it reaches a server.
Not yet verified¶
- The Rocket.Chat mobile apps: the buttons follow the mobile app's source, but have not been clicked on a phone.
- Rocket.Chat 9.x: supported from the 9.0 API, not run live.
How to run the tests yourself is in Development.