wiki: nightly audit 2026-11-03 window — new Job Radar stream-timeout incident, hermes-hq-push system page; update 11-02 recurrence, scheduled-tasks, index, log
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: hermes-hq-push (Hermes HQ iPhone app plugin)
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-11-04
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-12-04
|
||||
confidence: medium
|
||||
tags: [push, notifications, plugins, ios, desktop, device-access]
|
||||
sources: [live-hermes-plugins-list, live-plugin-manifest, live-config]
|
||||
---
|
||||
|
||||
# hermes-hq-push (Hermes HQ iPhone app plugin)
|
||||
|
||||
## What it is
|
||||
|
||||
`hermes-hq-push` is a **standalone Hermes plugin** (`kind: standalone`, **v0.9.27**, author *Hermes HQ*)
|
||||
living at `~/.hermes/plugins/hermes-hq-push/` (tree mtime **2026-11-03 11:06**). It is the host-side half
|
||||
of the **Hermes HQ iPhone app**: Apple push, phone device access, the desktop-app theme, the app's Terminal
|
||||
page, and Grok Live voice token minting.
|
||||
|
||||
Its own manifest states it **replaces `dispatch-push`**, and that *"its registered phones carry over."*
|
||||
|
||||
## Verified state (2026-11-04)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| `hermes plugins list` | `hermes-hq-push` — **enabled**, `0.9.27`, state `user` |
|
||||
| `config.yaml` → `plugins.enabled` | line **7556**: `- hermes-hq-push` |
|
||||
| `~/.hermes/plugins/dispatch-push/` | **absent** — and no `dispatch-push` reference survives in `config.yaml` |
|
||||
| live tree | `~/.hermes/plugins/hermes-hq-push/` (module files + `dashboard/` subdir + its own test suite) |
|
||||
| README | none — the manifest `description` is the only prose |
|
||||
|
||||
The replacement is therefore **clean**: nothing references the retired plugin, and the new one is explicitly
|
||||
enabled. Enablement and version are config-verified; individual hook firing was not separately captured.
|
||||
|
||||
## Surfaces (grouped from the plugin manifest)
|
||||
|
||||
**Push / approvals**
|
||||
- Finished turns — who replied and what they said.
|
||||
- Approvals with the command shown, **Approve / Deny from the lock screen**.
|
||||
- A bot waiting for a password, login, code or answer — the site or the question, **never the value**.
|
||||
- **Live Lock Screen cards:** every step the bot takes (e.g. *Running npm test…*, the two before it, how
|
||||
many) is pushed to the card, *sealed so only the phone can read it*; the card says **Done** when the turn
|
||||
ends.
|
||||
- Tells the app which scheduled jobs a bot is still working on.
|
||||
|
||||
**Device access** (app *Settings › Device Access*) — contacts lookup, location, reminders and events, alarms
|
||||
and timers, photos, Health, motion, music, and leaving an email or text for Anthony to send. This is what
|
||||
backs the Conduit-style phone sharing on the Hermes HQ app.
|
||||
|
||||
**Desktop app** — adds the **Hermes HQ theme**; opens a shell on the host for the app's Terminal page (the
|
||||
login shell, as an SSH login would; `plugin-data`'s `terminal.off` disables it); lets a Nous sign-in delete,
|
||||
archive and rename the chats it created (Hermes refuses them otherwise) and lets the desktop app open chats
|
||||
the app made (Hermes answers `not_found` otherwise); keeps a bot's **Sessions list fast** on a host that has
|
||||
run many scheduled jobs; lets the app send into a chat the desktop app has open; page previews; and keeps
|
||||
the app's performance log on this computer at **`~/.hermes/hq-logs`**.
|
||||
|
||||
**Voice** — mints the app's **Grok Live** voice session a short-lived xAI token (subscription sign-in or
|
||||
`XAI_API_KEY`; the bearer itself never reaches the phone).
|
||||
|
||||
**Restart guard** — stops a bot's one-off macOS launchd restart job from looping (those jobs are KeepAlive,
|
||||
so a one-time restart re-runs forever and locks the app out), and tells a restart script which in-app chats
|
||||
are mid-reply so a restart can wait.
|
||||
|
||||
**Cross-posting** — on request, posts the bot's answer back to the Telegram or Discord chat a conversation
|
||||
came from.
|
||||
|
||||
## Interface
|
||||
|
||||
- **Tools:** `device_contacts_search`, `device_request_result`.
|
||||
- **Hooks:** `pre_approval_request`, `post_approval_response`, `pre_tool_call`, `on_human_input_request`,
|
||||
`on_human_input_resolved`, `pre_llm_call`, `on_session_end`.
|
||||
|
||||
## Configuration
|
||||
|
||||
All environment variables are optional. `DISPATCH_*` spellings of the APNs/relay names still work.
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `HERMES_HQ_APNS_KEY_PATH`, `HERMES_HQ_APNS_KEY_ID`, `HERMES_HQ_APNS_TEAM_ID` | **Direct APNs mode** — path to the app's `.p8` auth key, its Key ID, and the Apple Developer Team ID that owns the app. *Only the app's developer has these.* |
|
||||
| `HERMES_HQ_PUSH_RELAY_URL` | Another push relay's HTTPS URL, or `off` to send nothing. **Unset = alerts go through the public Hermes HQ relay, sealed so it cannot read them.** |
|
||||
| `HERMES_HQ_PUSH_RELAY_KEY` | Only for a private relay that asks for a key. |
|
||||
|
||||
No key material is recorded on this page and none appears in the manifest — the relay key is declared
|
||||
`password: true` and is supplied by environment only.
|
||||
|
||||
## Relationship to the other push stacks
|
||||
|
||||
- [[systems/push-notifications]] documents the **Cadu/Vory** bridge (`hermes-push`, `vory-push`,
|
||||
`~/.hermes/push/hermes_push.py`) — a **separate** stack, untouched by this plugin and still `degraded`.
|
||||
- `dispatch-push`, the previous relay plugin for the same app, is **retired and fully removed** (see the
|
||||
verification table above).
|
||||
|
||||
## Open
|
||||
|
||||
- `~/.hermes/hq-logs` retention is not documented anywhere yet — check its growth before it becomes a disk
|
||||
item.
|
||||
- The plugin carries a very large surface under one name; if it misbehaves the manifest `description` is the
|
||||
only capability index.
|
||||
- **Operational note:** the plugin's `pre_tool_call` guard rejects any tool call whose arguments contain the
|
||||
literal text of a launchd one-off-restart command — writing *about* that command into a file is blocked
|
||||
too. Describe it, don't quote it.
|
||||
@@ -3,9 +3,9 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
review_after: 2026-11-10
|
||||
updated: 2026-11-04
|
||||
verified_on: 2026-11-04
|
||||
review_after: 2026-11-11
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
@@ -14,7 +14,19 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Current as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
- **Current as of 2026-11-04 (window 2026-11-03; live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db`) — the ledger produced its own red row.** 2026-11-03 settled **170 `completed` / 1 `failed` /
|
||||
1 `running` / 0 `unknown`** (172 rows) against 11-02's 177 / 2 — the **first `failed` row since 10-31**, and
|
||||
the failure is the **Job Radar Stage 1** (`dca8482e4f76`) dying on **`RuntimeError: stream active timeout`**
|
||||
after **6 h 28 m 18 s** → [[incidents/2026-11-03-job-radar-stream-active-timeout]]. Fleet **29 jobs — 26
|
||||
enabled / 3 disabled** (unchanged). **Non-clean `last_status` 1 → 2:** `dca8482e4f76` (`error`, streak 1)
|
||||
joins `066bb5191280` (`delivery_failed`, unchanged). **Unclosed incident rows 0 → 1** (`dca848_dfd4bea84f2a`).
|
||||
**Deliveries: 7 rows, 7/7 `delivered`** (07:13:49, 07:30:35, 08:39:25, 12:37:09, **13:28:51 `for_failure`**,
|
||||
13:49:11, 18:45:39). ⚠ **The 03:00 stocktake ran long for the second day:** `1a8e90ea2271`
|
||||
**03:00:30.495724 → 10:41:13.851030 = 7 h 40 m 43 s**, `completed` / `suppressed`, **delivered nothing** and
|
||||
again left its artefacts uncommitted (`systems/skills-index.md` modified, `raw/stocktake/freshness-20261103.txt`
|
||||
untracked — committed by this audit). **Sessions in the window: 16** (14 `cron`, 1 `ios`, 1 `desktop`).
|
||||
- **Prior window — as of 2026-11-03 00:1x AWST (live `jobs.json` + `executions.db` + `cron_incidents` +
|
||||
`deliveries.db`) — the first fully clean ledger day in five, but two long runs ended in a silent cap and
|
||||
one of them raised no incident at all.** Fleet **29 jobs — 26 enabled / 3 disabled** (unchanged).
|
||||
**Window 2026-11-02 settled 177 `completed` / 2 `running` / 0 `unknown` / 0 `failed`** (179 rows;
|
||||
@@ -279,6 +291,12 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
|
||||
|
||||
### Open
|
||||
- **The Job Radar's Stage 1 lost a second consecutive day, and the *fault changed* (NEW 2026-11-04):**
|
||||
`dca8482e4f76` ran **6 h 28 m 18 s** on 2026-11-03 and died on **`RuntimeError: stream active timeout`**
|
||||
(not the iteration cap), going **silent for 65 m 21 s** first. The ledger went red this time
|
||||
(`failed` + incident `dca848_dfd4bea84f2a` + a delivered notice) →
|
||||
[[incidents/2026-11-03-job-radar-stream-active-timeout]]. **2026-11-02's and 2026-11-03's candidate sets are
|
||||
both unrecovered.**
|
||||
- **Two ~9-hour runs ended in a silent iteration cap on 2026-11-02, and the more damaging one raised no incident (NEW 2026-11-03):**
|
||||
- **`dca8482e4f76`** (Job Radar Stage 1) ran **07:00:33 → 16:01:52 = 9 h 01 m 19 s** and closed
|
||||
`status: completed` / `error: NULL` / `delivery_outcome: suppressed` **after hitting the agent
|
||||
@@ -883,3 +901,64 @@ unidentified `-15` signal source; and the **30 s worker-ack window vs this host'
|
||||
**Plus new:** whether 2026-11-02's **164 deduped Job Radar candidates are recoverable**, whether a
|
||||
cap-hit should keep being recorded as `completed`, and why the 03:00 stocktake now takes ~9 h 45 m.
|
||||
|
||||
|
||||
|
||||
## Re-verified 2026-11-04 (window 2026-11-03, AWST)
|
||||
|
||||
**Summary.** One genuine red row, raised by the harness itself. **2026-11-03 settled 170 `completed` / 1
|
||||
`failed` / 1 `running` / 0 `unknown`** (172 rows). The single `failed` row is the **Job Radar Stage 1**
|
||||
(`dca8482e4f76`) — **07:00:33.819190 → 13:28:52.305995**, `RuntimeError: stream active timeout`, a **new
|
||||
terminal signature** for a job whose last three failures were all the agent iteration cap. Its run session
|
||||
compacted **five times** (08:04:35, 08:47:19, 09:31:50, 10:49:26, 11:39:21) and went **silent for 65 m 21 s**
|
||||
after its last tool result at **12:23:28** before the harness wrote *"This turn did not complete."* at
|
||||
13:28:49 → [[incidents/2026-11-03-job-radar-stream-active-timeout]].
|
||||
|
||||
### Count (live, 2026-11-04)
|
||||
|
||||
| Measure | 2026-11-03 window | Change vs 11-02 window |
|
||||
|---|---|---|
|
||||
| Fleet | **29 jobs — 26 enabled / 3 disabled** | unchanged |
|
||||
| Ledger day | **170 `completed` / 1 `failed` / 1 `running` / 0 `unknown`** (172 rows) | 11-02: 177 / 2 / 0 / 0 |
|
||||
| Non-clean `last_status` | **2** (`dca8482e4f76` error streak 1; `066bb5191280` `delivery_failed`) | **1 → 2** |
|
||||
| Unclosed incident rows | **1** (`dca848_dfd4bea84f2a`, open, `alerted` 13:28:55.141618) | **0 → 1** |
|
||||
| Deliveries | **7 rows, 7/7 `delivered`** | 11-02: 9 / 9 |
|
||||
| Gateway | PID **175269** (started 2026-10-31 19:23:56) — **no restart** | unchanged |
|
||||
| Dashboard | PID **176771** (started 2026-10-31 19:35:02) | unchanged |
|
||||
| Sessions in window | **16** (14 `cron`, 1 `ios` `20260908_031541_49c051`, 1 `desktop` `20261031_190813_95fbde`) | 14 started, all `cron` |
|
||||
|
||||
### What changed in the window
|
||||
|
||||
- **Job Radar Stage 1 died on a provider stream timeout** — see the new incident page; it also **answered the
|
||||
11-02 page's open question 3**: output is suppressed when the terminal state is `completed`, and delivered
|
||||
when it is `failed`. The cap-produced green row is the anomaly, not the red one.
|
||||
- **Stage 2 declined twice, correctly** (`5fc292ff0f3b` 11:30:36 → 11:32:16; 14:30:32 → 14:33:11) — no
|
||||
handoff existed. `~/.hermes/radar-ref/handoff/` still holds **only** 10-27, 10-29, 10-30.
|
||||
- **New system page:** [[systems/hermes-hq-push]] — plugin **v0.9.27**, tree mtime **2026-11-03 11:06**,
|
||||
`hermes plugins list` → **enabled** (`user`), `config.yaml` `plugins.enabled` line **7556**. Its manifest
|
||||
states it **replaces `dispatch-push`**, and the replacement is clean: **no `dispatch-push` reference
|
||||
survives in `config.yaml`** and its directory is gone. Distinct from the Cadu/Vory bridge on
|
||||
[[systems/push-notifications]], which is untouched and still `degraded`.
|
||||
- **The 18:00 Weekly Review's 7 h 26 m recovery held** — no new `unknown` rows, and `ecc1fc13655a`'s ghost
|
||||
`error` did not reappear in this window.
|
||||
|
||||
### Files touched by this audit
|
||||
|
||||
- **New:** `incidents/2026-11-03-job-radar-stream-active-timeout.md`, `systems/hermes-hq-push.md`.
|
||||
- **Updated:** this page (+frontmatter, +Count bullet, +this section), `index.md`,
|
||||
`incidents/2026-11-02-job-radar-stage1-iteration-cap-recurrence.md` (11-03 row + 2026-11-04 update),
|
||||
`log.md`.
|
||||
- **Committed on the stocktake's behalf:** `systems/skills-index.md` (modified) and
|
||||
`raw/stocktake/freshness-20261103.txt` (untracked) — the 03:00 job left both in the working tree again.
|
||||
- **Nothing removed or archived:** no job added, retired, disabled or repinned; no system obsoleted; no
|
||||
decision reversed; `systems/push-notifications.md` left as-is (not re-verified this window).
|
||||
|
||||
### Still open (2026-11-04 — carries its predecessor forward)
|
||||
|
||||
The single-node no-failover topology behind every model-calling job; the GitHub Trending Radar delivery
|
||||
token; Camofox containment under multi-tab load; the Conduit health store with no retention; the
|
||||
unreconciled admission files `~/.hermes/cron/admissions/141448f9…json` and `69b5588f…json`; the unidentified
|
||||
`-15` signal source; and the **30 s worker-ack window vs this host's cold-start time**. **New:** whether the
|
||||
**65 m 21 s stream silence** is the stream-active watchdog threshold or a silently dead provider socket;
|
||||
whether the five-compaction churn contributes; whether a stream timeout on a file-deliverable job should
|
||||
**retry automatically**; `radar-ref/handoff/` retention (newest file 10-30); why Stage 2 still has no
|
||||
handoff to wait on; and why the 03:00 stocktake takes **7 h 40 m** and commits nothing.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-11-02
|
||||
verified_on: 2026-11-02
|
||||
updated: 2026-11-03
|
||||
verified_on: 2026-11-03
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
|
||||
Reference in New Issue
Block a user