audit(2026-11-12): the estate's last incident closed itself; a past-0.9.42 hq-push install was refused by the plugin scan (fix parked in PR #2)

- 429 row cbfe43_ed01845310d8 closed resolved 2026-11-12T00:25:36 on the next clean slot; 0 of 137 unclosed
- 11-12: 176 rows / 175 completed / 0 failed / 1 running; 11-11: 172/172/0; deliveries 9/9 and 7/7 delivered
- NEW incidents/2026-11-12-hq-push-upgrade-blocked-by-plugin-scan.md (dangerous verdict, one critical in the
  plugin's own tile skill; --force cannot override; no per-verdict override exists)
- systems/hermes-hq-push.md corrected live (0.9.27 -> 0.9.42, tree mtime 2026-11-08 02:19, config line 13226)
- current-state / scheduled-tasks / index / log updated; skills-index regenerated; freshness 2026-11-13 (0 stale)
This commit is contained in:
Tony0410
2026-11-13 00:41:15 +08:00
parent ab3652aa20
commit e033a08f17
9 changed files with 432 additions and 32 deletions
@@ -1,10 +1,10 @@
---
title: Wiki Audit — Nightly Died on Its Own 429 — the 2026-11-10 Window Was Never Audited
type: incident
status: open
status: resolved
created: 2026-11-11
updated: 2026-11-11
verified_on: 2026-11-11
updated: 2026-11-12
verified_on: 2026-11-12
review_after: 2026-12-11
confidence: high
tags: [cron, incident, omniroute, rate-limit, audit]
@@ -43,18 +43,36 @@ The three log lines from the run's own session `local-1000c1a3…` show the shap
- The **same tenant** (`tn-5ud77abtl1`) appears in the `quota exceeded` line, so this is an
account-level quota fault on the router, not a single-model lockout.
## Incident row — raised, never alerted, still open
## Incident row — raised, never alerted, closed `resolved` 2026-11-12T00:25:36
`cbfe43_ed01845310d8` — `job_id cbfe434a3b5e`, `failure_type: rate_limit`, state **`detected`**,
`alerted_at: NULL`, `acked_at: NULL`, `closed_at: NULL`, opened **2026-11-11T00:12:28.578435+08:00**
(`first_seen_at` = `last_seen_at` = that instant — seen exactly once).
- It is the **only open row** in `cron_incidents` — the table held **135** rows at audit time and **137**
- It **was** the only open row in `cron_incidents` — the table held **135** rows at audit time and **137**
by the end of this run. `acked_at` is **NULL** as well as `alerted_at`, so nothing was told and
nothing was acknowledged. The previous open item,
`5fc292_773822455856`, closed `resolved` at **2026-11-11T08:03:47.644226**.
- So the estate's single outstanding incident is **the audit's own failure**, and nothing was told
about it.
- So the estate's single outstanding incident **was** the audit's own failure, and nothing was told
about it. ⚠ **Superseded 2026-11-12** — the row closed `resolved` at **2026-11-12T00:25:36.133387+08:00**;
*cron_incidents* now holds **0 unclosed of 137**. See *Resolution* below.
## Resolution — 2026-11-12T00:25:36, self-cleared
The row closed **`resolved`** at **2026-11-12T00:25:36.133387+08:00**, at the end of the **next** slot of
the same job: `cbfe434a3b5e` ran **2026-11-11T23:59:54.486572 → 2026-11-12T00:25:36.133387 = 25 m 42 s**
and completed. Its delivery carried **both windows**, so 2026-11-10's day was reconciled a day late
rather than never (the recovery record is the `audit(2026-11-11)` commit series at 2026-11-12 00:1x-00:24).
Nothing was repaired to make this happen: the pool's own reset at ≈ **2026-11-11 06:36:57** restored the
dispatch path, exactly as the `reset after 6h 24m 31s` field predicted, and the 07:0x cohort ran clean.
`cron_incidents` now reads **0 unclosed of 137** rows — the estate is incident-clean as of 2026-11-12.
**The recurrence risk is unchanged.** The same *"all targets were skipped by pre-dispatch filters"*
signature was found again on a **second job and a second pin** by the 2026-11-12 03:00 stocktake
([[incidents/2026-10-21-omniroute-pre-dispatch-filter-skip]]), and every model-calling job still rides
`custom:omni-101` with no failover for the free-tier cohort. Closing this row proves the guard
self-heals; it is not evidence the guard is gone.
## Impact
@@ -79,7 +97,8 @@ The three log lines from the run's own session `local-1000c1a3…` show the shap
## Fix Applied
**None — this is a detection-and-verification record.** No configuration changed this window. The run's
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11.
own recovery was the pool reset at ≈ 06:36:57 on 2026-11-11, and the incident row closed on the next
slot's clean completion (2026-11-12T00:25:36).
## Prevention
@@ -0,0 +1,98 @@
---
title: hermes-hq-push 0.9.47+ — Upgrade Refused by the Plugin Security Scan (dangerous verdict)
type: incident
status: open
created: 2026-11-13
updated: 2026-11-13
verified_on: 2026-11-13
review_after: 2026-12-13
confidence: high
tags: [plugins, security-scan, hq-push, plugin-guard, install, incident]
sources: [live-hermes-plugins-list, live-plugin-manifest, live-config, live-plugin-guard-source, live-gh-pr, live-state-db]
---
# hermes-hq-push 0.9.47+ — Upgrade Refused by the Plugin Security Scan (2026-11-12)
**The scan did its job and there is no way to overrule it.** On 2026-11-12 (~17:38–18:29 AWST) the
attempt to take [[systems/hermes-hq-push]] past its installed **0.9.42** was refused before any file
was written, by the pre-install plugin security scan. The installed copy is untouched, still `enabled`,
and still delivering. The blocking finding is in the plugin's **own** `dashboard-tile` skill, and the
fix is sitting in an open pull request.
## What happened
| Field | Value |
|---|---|
| Attempted | `hermes plugins install mrcharlesiv/hermes-hq-plugins/hermes-hq-push --enable --force` |
| Refusal | `Blocked: Security scan blocked plugin install: Blocked (dangerous verdict, 1 critical of 39 findings (echo_pipe_exec)). --force does not override a dangerous verdict.` |
| Findings | **39 total — 1 critical**, the rest low/medium |
| Sole critical | `hermes-hq-push/skills/dashboard-tile/SKILL.md:41` — category *obfuscation*: the `echo '<json>' \| python3 …` one-liner |
| Verdict logic | **any** critical ⇒ `dangerous` = hard block; medium/low alone ⇒ `caution` (installable, warned) |
| Installed after | **unchanged** — `v0.9.42`, tree mtime **2026-11-08 02:19**, `enabled`, state `user` |
## Why no override was available (verified in source, not assumed)
- **The only config key is `plugins.scan_on_install`** (default `true`; currently **not set** in
`~/.hermes/config.yaml`, so scanning is on). Setting it `false` disables the scan for **every** future
install — it does not overrule a verdict, it blinds the check. That is the wrong lever.
- **`--force` lifts `caution` only.** Its own refusal text says so, and the code agrees. `--allow-removed`
is unrelated — it is the blocklist for catalogue removals.
- **No env var, no trusted-source list, no allowlist, no `INSTALL_POLICY`/`trust_level` for plugins.**
Those mechanisms exist only in `tools/skills_guard.py`, reachable from the *skills* path
(`skills.guard_agent_created`). `tools/plugin_guard.py:398` hardcodes `trust_level="community"` and
`should_allow_plugin_install` never reads it.
- **The block is a single unconditional branch** at `tools/plugin_guard.py:406-418`: any verdict outside
`safe`/`caution` returns `False, "Blocked (dangerous verdict, …)"`.
## The fix, and why it is not merged
The critical is a *pattern* match, not a defect: the tile skill embedded its JSON payload in a shell
one-liner, which the scanner reads as obfuscation however benign the intent.
| Field | Value |
|---|---|
| Repo / PR | `mrcharlesiv/hermes-hq-plugins` **#2** — the repo's **only** open PR |
| Author | `Tony0410` (Anthony) |
| Head | fork `Tony0410/hermes-hq-plugins`, branch `fix/tile-skill-heredoc` → base `main` |
| State | **OPEN**, `mergedAt: null`, `mergeable: MERGEABLE`, created 2026-11-12T10:00:01Z (updated 10:01:57Z) |
| Diff | **+7 / −2 in one file** — `hermes-hq-push/skills/dashboard-tile/SKILL.md` (the one-liner becomes a heredoc) |
| Local clone | `/home/hermes/workspace/hq-plugins-fix` (commit `90c9d28`, branch checked out, clean) |
The `gh` CLI on this host is authenticated as **`Tony0410` with read-only access** to
`mrcharlesiv/hermes-hq-plugins`, so the change had to travel fork → PR rather than as a branch push.
## Impact
- **The upgrade path to 0.9.47+ is blocked** until PR #2 merges upstream and the plugin is re-installed.
Nothing else in the estate is affected.
- **No app-side impact:** the installed 0.9.42 keeps delivering to the Hermes HQ iPhone app; the refusal
happened before any file was replaced, so there is no half-installed state to clean up.
- The block is **fail-safe by design** — the failure mode is "cannot upgrade", not "installed something
unchecked".
## Related finding — the "stored hold-and-ask instruction" was never stored
The same evening, the alleged standing instruction to *hold and ask before acting on* plugin
security-scan warnings was searched for exhaustively: a recursive sweep of `~/.hermes`
(**184,716 files**, generated/venv/browser-profile/SQLite trees excluded) for the phrases
`wait for my answer`, `show me what it found`, `security scan flags` and `security-scan` found **zero
matches in any editable text file**. The sentence exists only in `state.db` **conversation history**
(58 rows). **Nothing was removed and no policy was reverted**, because no such policy was ever
persisted. Procedure recorded as the skill `stored-instruction-search-and-removal`.
## Prevention / how this closes
- **Merge PR #2**, then re-run the install. That is the whole fix.
- If a standing exception is ever genuinely wanted, **add an explicit trusted-source or allowlist path to
`tools/plugin_guard.py`** — do **not** set `plugins.scan_on_install: false`. Turning the scan off is
not "allowing this plugin"; it is removing the check from every future install, including the ones
nobody is looking at.
- Expect the same class of false positive again: any plugin shipping a shell one-liner that pipes JSON
into an interpreter will read as *obfuscation*. Author the pattern as a heredoc or a script file.
## Related
- [[systems/hermes-hq-push]]
- [[systems/hermes-agent]]
- [[systems/scheduled-tasks]]
- [[log]]