wiki audit 2026-10-03: cliprox second job confirmed (2/3), work-mailbox incident page, streak recount, skills-index 341->344
This commit is contained in:
+13
-4
@@ -3,16 +3,25 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-02
|
||||
verified_on: 2026-10-02
|
||||
review_after: 2026-10-30
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
review_after: 2026-10-31
|
||||
confidence: high
|
||||
tags: [system]
|
||||
sources: [live-config, live-cron-list, live-hermes-version]
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-02
|
||||
Updated: 2026-10-03
|
||||
|
||||
## Recent Changes (2026-10-03)
|
||||
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set stays at **6**, but the composition changed: `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed on 2026-10-02 with the *identical* HTTP 401 the cliprox incident predicted — `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard`. That makes **2 of the 3** `cliprox/terra` jobs confirmed broken, so the provider-connection auth expiry is a real credential fault, not a one-job fluke. `79f9bb41ecf5` (Weekly Review, Sun 18:00) is the last untested one — its only run (2026-09-27) predates the expiry and succeeded; next due **2026-10-04 18:00**. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **Streaks recounted from `executions.db` — one number corrected:** `dca8482e4f76` 6 → **7** (one more daily run), but `1a8e90ea2271` **8 → 7** — the ×8 carried since 2026-10-01 was an overcount; that job has exactly 7 recorded runs and all 7 failed. Recount before quoting a streak.
|
||||
- **A delivered notice is not a successful job.** Both `cliprox/terra` failures are recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable — the execution `status` is `failed` and no digest/plan was produced. Judge cron health on the `status` column, not the delivery outcome.
|
||||
- **Resolved — `1b23c042e17d` stale `delivery_failed` flag cleared:** the 2026-09-30 retarget to `telegram:-1004321904721:102` worked exactly as predicted; its 2026-10-02 run delivered (then failed on the model pin). The prediction held.
|
||||
- **[[systems/skills-index]] regenerated by hand: 341 → 344 skills.** The daily stocktake job (`1a8e90ea2271`) is still blocked on a retired model (HTTP 410, `qwen3.5:397b`), but its **script is healthy** — `wiki-maintain.sh` was run manually on 2026-10-03 and completed cleanly, refreshing the index and writing `raw/stocktake/freshness-20261003.txt`. One new skill appeared: `fleet-wide-failure-triage` (devops). Only the scheduled wrapper is broken; every non-model step of the stocktake now needs a manual run each day until it is repinned.
|
||||
- **Work mailbox blocker documented — and it is older than it looks.** A new incident page records that the `work` Outlook account has been unreachable since **2026-08-26**, not this week: the account uses password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` dies with `login not supported`. The TCP/TLS path is fine — it is a tenant policy wall, not a bad password, and re-running the credential helper can never fix it. It needs an OAuth2 path. No cron job polls that mailbox, so this is an on-demand capability gap. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||||
- **GitHub Trending Radar token claim downgraded to "unverified":** the revoked-bot-token (`getMe` → 401) diagnosis could not be reproduced — `c0a9fa314542`'s last recorded run (2026-09-30 19:22, off-schedule) **completed and delivered**. Re-confirm at the next scheduled run (Mon 2026-10-05 09:00) before acting on it.
|
||||
|
||||
## Recent Changes (2026-10-02)
|
||||
- **Cron fleet re-verified:** 30 jobs, 27 enabled / 3 disabled (live `~/.hermes/cron/jobs.json`). Failing set **3 → 6**: two *new* failures landed on 2026-10-01 —
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
title: Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-03
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
confidence: high
|
||||
tags: [email, himalaya, office365, oauth2, auth, incident]
|
||||
sources: [live-himalaya-debug, live-tcp-probe, cron-session-20260826_180700_a94098]
|
||||
---
|
||||
|
||||
# Work Outlook Mailbox — Basic Auth Disabled (Himalaya)
|
||||
|
||||
**Window:** first detected 2026-08-26; re-verified 2026-10-03 (still open)
|
||||
**Status:** Open — blocked pending an OAuth2 path
|
||||
**Impact:** The agent cannot read or send from the `work` mailbox. Every work-email
|
||||
request is unanswerable until this is fixed.
|
||||
|
||||
## Symptoms
|
||||
|
||||
```
|
||||
$ himalaya envelope list -a work -s 1 --debug
|
||||
email::imap: connecting to IMAP server outlook.office365.com:993
|
||||
email::imap: supported auth mechanisms mechanisms=[XOAuth2]
|
||||
Error: cannot build IMAP client
|
||||
Caused by:
|
||||
login not supported
|
||||
```
|
||||
|
||||
Network is *not* the problem: the TCP/TLS connection to `outlook.office365.com:993`
|
||||
succeeds and the server completes its capability exchange. The failure is purely at
|
||||
the authentication step.
|
||||
|
||||
## Root Cause
|
||||
|
||||
The account is configured for **password** auth, and the tenant no longer accepts it.
|
||||
|
||||
- Account config (`~/.config/himalaya/config.toml`, account `work`) sets
|
||||
`backend.type = "imap"`, `backend.host = "outlook.office365.com"`,
|
||||
`backend.login = "<work address>"`, `backend.auth.type = "password"` with a helper
|
||||
script at `~/.config/himalaya/get_work_outlook_password.sh`. The SMTP half
|
||||
(`smtp.office365.com:587`, start-tls) is configured the same way.
|
||||
- The server advertises **only `XOAuth2`**. Basic authentication for IMAP/SMTP is
|
||||
disabled at the Microsoft 365 tenant level — a WA Government tenant policy, not a
|
||||
wrong password. Retrying the helper script can never succeed.
|
||||
|
||||
This is the same wall hit on 2026-08-26: the account was set up that day and every
|
||||
attempt to list folders returned a login rejection. The mechanism list has since
|
||||
*narrowed* — on 2026-08-26 the tenant rejected plain password login despite still
|
||||
advertising `AUTH=PLAIN`; by 2026-10-03 only `XOAuth2` is offered.
|
||||
|
||||
The personal Gmail account is unaffected (Google still allows app passwords), so
|
||||
this is specific to the work tenant.
|
||||
|
||||
## Fix Options
|
||||
|
||||
1. **OAuth2 for the `work` account — preferred, needs Anthony.** Requires an app
|
||||
registration in the tenant with the IMAP/SMTP delegated scopes, a device-code or
|
||||
interactive sign-in to mint a refresh token, and `backend.auth.type = "oauth2"`
|
||||
plus token storage in the himalaya config. Anthony's normal login is
|
||||
*number-matching approval in Microsoft Authenticator* — that is exactly the
|
||||
device-code / interactive OAuth flow, so the human step is one sign-in. Tenant
|
||||
admin consent may still be required.
|
||||
2. **Tenant exception** — an admin re-enables IMAP/SMTP basic auth or issues an app
|
||||
password. Unlikely on a government tenant; not worth relying on.
|
||||
3. **Alternative access path** — browser/OWA session or Microsoft Graph. Slower and
|
||||
more brittle than native IMAP, but needs no admin action.
|
||||
4. **Do not** keep re-running the password helper. The credential is not the fault.
|
||||
|
||||
## Consequences for tooling
|
||||
|
||||
- The `email/mail-lookup` skill resolves the account by address before searching; when
|
||||
the requested mail lives in the `work` mailbox it will reach this failure and must
|
||||
report the account as unreachable rather than as "no results found".
|
||||
- No cron job currently polls the work mailbox, so this is an on-demand capability
|
||||
gap, not a background failure.
|
||||
|
||||
## Related
|
||||
|
||||
- [[systems/messaging-integrations]]
|
||||
- [[systems/scheduled-tasks]]
|
||||
@@ -3,8 +3,8 @@ title: OmniRoute cliprox/terra Connection Auth Expiry
|
||||
type: incident
|
||||
status: open
|
||||
created: 2026-10-02
|
||||
updated: 2026-10-02
|
||||
verified_on: 2026-10-02
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
confidence: high
|
||||
tags: [cron, omniroute, provider, auth, incident]
|
||||
sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_20261001_100040]
|
||||
@@ -14,7 +14,7 @@ sources: [live-jobs-json, cron-executions-db, cron-session-cron_06a0a1c8704f_202
|
||||
|
||||
**Window:** first failure 2026-10-01 10:00 AWST (job schedule `0 10 * * 4`); still open
|
||||
**Status:** Open — not self-recovering
|
||||
**Impact:** 1 job failed outright; 2 more enabled jobs share the same model pin and are expected to fail on their next runs
|
||||
**Impact:** 2 of the 3 exposed jobs have now failed on this pin; 1 remains untested
|
||||
|
||||
## Symptoms
|
||||
|
||||
@@ -49,9 +49,32 @@ Three enabled jobs run `cliprox/terra`:
|
||||
| `1b23c042e17d` | Enhanced Weekend Adventure Planner | `0 8 * * 5` | `telegram:-1004321904721:102` |
|
||||
| `79f9bb41ecf5` | Weekly Review | `0 18 * * 0` | `origin` |
|
||||
|
||||
Only `06a0a1c8704f` had run by 2026-10-02; the other two are exposed on their next
|
||||
Only `06a0a1c8704f` had run by 2026-10-02; the other two were exposed on their next
|
||||
scheduled runs.
|
||||
|
||||
### Second job confirmed — 2026-10-02 08:00
|
||||
|
||||
`1b23c042e17d` (**Enhanced Weekend Adventure Planner**, Fri 08:00) failed on schedule
|
||||
with the *identical* error:
|
||||
|
||||
```
|
||||
HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired
|
||||
— please reconnect in the dashboard
|
||||
```
|
||||
|
||||
That is now **2 of 3** exposed jobs confirmed broken on this pin, which retires any
|
||||
doubt about a one-job fluke. Its failure notice was delivered to topic `102`, so the
|
||||
run is recorded `delivery_outcome: delivered` even though no plan was produced.
|
||||
|
||||
`79f9bb41ecf5` (**Weekly Review**, Sun 18:00) is the last untested one — its only
|
||||
recorded run was 2026-09-27 18:00, four days *before* the connection lapsed, and it
|
||||
succeeded. Its next scheduled run is **2026-10-04 18:00**, which will confirm whether
|
||||
the expiry is tenant-wide for this model or per-connection.
|
||||
|
||||
Corollary worth recording: a job can be recorded as **successfully delivered while
|
||||
having failed**, because the failure notice itself is a deliverable. Read the
|
||||
execution `status`, not the delivery outcome, when judging whether a job worked.
|
||||
|
||||
## Fix
|
||||
|
||||
Re-authenticate the OmniRoute connection for `cliprox/terra`. Do **not** repin these jobs
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-02
|
||||
> model routing and automation. Last updated: 2026-10-03
|
||||
|
||||
## Start here
|
||||
- [[current-state]] — Current systems status
|
||||
@@ -28,11 +28,12 @@
|
||||
|
||||
### Integrations
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-02)
|
||||
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily)
|
||||
- [[systems/scheduled-tasks]] — 30 cron jobs (27 enabled, 3 disabled; 6 failing as of 2026-10-03)
|
||||
- [[systems/skills-index]] — Installed skills inventory (auto-regenerated daily; being refreshed by hand while the stocktake job is blocked)
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; 1 job failed, 2 more share the pin (open 2026-10-01)
|
||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]] — OmniRoute `cliprox/terra` connection credential expired; **2 of 3 jobs sharing the pin now failed**, the third untested (open 2026-10-01)
|
||||
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]] — Work Outlook mailbox unreachable: tenant allows only `XOAuth2`, himalaya configured for password (open since 2026-08-26)
|
||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]] — Managed-runtime venvs missing `dotenv`/`mnemosyne` broke every cron worker (resolved 2026-09-30); dependency guard now runs every 15 min
|
||||
- [[incidents/2026-08-18-mnemosyne-capture-pipeline]] — Mnemosyne capture pipeline broken (active); ~48h of turns not captured
|
||||
- [[incidents/2026-08-15-skills-index-drift]] — Skills index 4x undercount (66 vs 330 installed); fixed via auto-regeneration
|
||||
|
||||
@@ -288,3 +288,13 @@
|
||||
- **Updated `index.md`** — last-updated to 2026-10-02; cron line now notes the failing count.
|
||||
- **Structural finding — the wiki exists as two trees and they have drifted both ways.** `/home/hermes/workspace/agent-estate-wiki` is canonical: it holds the Gitea remote (`Anthony/agent-estate-wiki`), the Quartz build and is what `wiki-maintain.sh` commits. `/home/hermes/wiki/agent-estate-wiki` is the Obsidian/Ignis vault copy the iPad reads (pages under `wiki/`, no git remote). The audit prompt names only the vault path. The trees have also lost pages in *opposite* directions — `systems/ignis.md`, `systems/nextcloud.md` and three `incidents/` pages exist only in the vault, while `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` and `reports/communities-wa-social-media-audit-2026.md` exist only in canonical — so canonical `index.md` currently links two pages that do not exist there. A one-way sync in either direction silently deletes files. This run wrote to **both** trees.
|
||||
- No secrets written; all credential-shaped values in config remain redacted.
|
||||
|
||||
## [2026-10-03] update | Nightly wiki audit — cliprox second job confirmed, work-mailbox incident, streak recount
|
||||
- **Updated `systems/scheduled-tasks.md`** — fleet re-verified against live `~/.hermes/cron/jobs.json`: still 30 jobs (27 enabled, 3 disabled), failing set still **6** but the composition moved. `1b23c042e17d` (Enhanced Weekend Adventure Planner, Fri 08:00) failed at its 2026-10-02 run with the *identical* `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` that the cliprox incident predicted — making it **2 of the 3** `cliprox/terra` jobs confirmed broken, which retires any "one-job fluke" reading. `79f9bb41ecf5` is the last untested job on that pin (next due 2026-10-04 18:00). **Streak recount** against `executions.db`: `dca8482e4f76` ×6→**×7**, but `1a8e90ea2271` ×8→**×7** — the ×8 carried since 2026-10-01 was an overcount (7 recorded runs, all failed). **Resolved:** `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as the 2026-09-30 fix predicted (delivered to topic 102). **Downgraded:** the GitHub Trending Radar revoked-token claim is now "unverified at last run" — `c0a9fa314542`'s 2026-09-30 19:22 run completed *and delivered*. New Known Issue recording that **a delivered notice is not a successful job** (both cliprox failures read `delivery_outcome: delivered` on a `failed` status).
|
||||
- **New page `incidents/2026-08-26-work-outlook-basic-auth-disabled.md`** — the `work` Outlook mailbox has been unreachable since **2026-08-26**: himalaya is configured for password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` dies with `login not supported`. TCP/TLS to `outlook.office365.com:993` is fine, so this is a policy wall, not a bad password — re-running the credential helper can never fix it; it needs an OAuth2 path (device-code matches Anthony's Authenticator number-matching sign-in). No cron job polls it, so this is an on-demand capability gap.
|
||||
- **Updated `systems/messaging-integrations.md`** — Email section rewritten from "connected" (wrong) to the per-account truth: personal Gmail usable, `work` unreachable; frontmatter `updated`/`verified_on` bumped. Added the tooling consequence: `email/mail-lookup` resolves the account before searching, so work-mail requests hit this wall and must be reported as *account unreachable*, not "no results found".
|
||||
- **Updated `systems/hermes-agent.md`** — skills count re-confirmed at 344 across 72 dirs by the manual stocktake; Scheduled Tasks section verified 2026-10-03 with the 6-job failing list enumerated; Known Issues broadened from three unrepinable model pins to **six jobs across three model faults plus one credential fault**, and a second entry added for the stock-take job being down; the `Email: connected` bullet corrected.
|
||||
- **Updated `current-state.md`** and **`index.md`** — new Recent Changes (2026-10-03) block; cron line and incident list refreshed; last-updated bumped.
|
||||
- **`systems/skills-index.md` regenerated 341 → 344** — including a new skill, `fleet-wide-failure-triage` (devops). This was done **by hand**: job `1a8e90ea2271` is still blocked on HTTP 410 (retired `qwen3.5:397b`) and cannot self-recover. `wiki-maintain.sh` itself is healthy and was run manually, also writing `raw/stocktake/freshness-20261003.txt` — recorded so the next reader knows the stocktake is on manual refresh, not on schedule.
|
||||
- **Vault mirror:** the same page edits were mirrored into `/home/hermes/wiki/agent-estate-wiki/wiki/` (Obsidian/Ignis vault) so the iPad view matches the Gitea source. Vault pages live under `wiki/`; `index.md` sits at the vault root.
|
||||
- No secrets written; all credential-shaped values in config remain redacted.
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
# Wiki freshness audit — 2026-10-03
|
||||
Threshold: >21 days since `updated`
|
||||
|
||||
## STALE (68)
|
||||
|
||||
- [ 73d] comparisons/camoufox-vs-chrome.md (updated 2026-07-22)
|
||||
- [ 73d] comparisons/hermes-vs-nanobot.md (updated 2026-07-22)
|
||||
- [ 73d] concepts/agent-memory.md (updated 2026-07-22)
|
||||
- [ 73d] concepts/model-routing.md (updated 2026-07-22)
|
||||
- [ 73d] concepts/persistent-shells.md (updated 2026-07-22)
|
||||
- [ 73d] concepts/tool-calling.md (updated 2026-07-22)
|
||||
- [ 73d] decisions/index.md (updated 2026-07-22)
|
||||
- [ 73d] decisions/keep-openrouter-as-fallback.md (updated 2026-07-22)
|
||||
- [ 73d] decisions/persistent-shell-enabled.md (updated 2026-07-22)
|
||||
- [ 73d] experiments/index.md (updated 2026-07-22)
|
||||
- [ 73d] infrastructure/backups.md (updated 2026-07-22)
|
||||
- [ 73d] infrastructure/credentials-map.md (updated 2026-07-22)
|
||||
- [ 73d] infrastructure/docker-services.md (updated 2026-07-22)
|
||||
- [ 73d] infrastructure/domains-and-tunnels.md (updated 2026-07-22)
|
||||
- [ 73d] infrastructure/network-map.md (updated 2026-07-22)
|
||||
- [ 73d] queries/retained-research-answers.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/backup-wiki.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/diagnose-docker.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/provider-health.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/recover-docker-service.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/restart-browser.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/restart-hermes.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/restore-openrouter.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/test-openrouter.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/troubleshoot-tool-loop.md (updated 2026-07-22)
|
||||
- [ 73d] runbooks/update-hermes-safely.md (updated 2026-07-22)
|
||||
- [ 73d] systems/cloudflare-tunnel.md (updated 2026-07-22)
|
||||
- [ 73d] systems/headless-server.md (updated 2026-07-22)
|
||||
- [ 73d] systems/home-assistant.md (updated 2026-07-22)
|
||||
- [ 73d] systems/openrouter.md (updated 2026-07-22)
|
||||
- [ 73d] systems/terminal-backend.md (updated 2026-07-22)
|
||||
- [ 70d] runbooks/gateway-resource-resilience.md (updated 2026-07-25)
|
||||
- [ 70d] systems/tool-search.md (updated 2026-07-25)
|
||||
- [ 69d] systems/homepage-dashboard.md (updated 2026-07-26)
|
||||
- [ 69d] systems/obsidian.md (updated 2026-07-26)
|
||||
- [ 67d] runbooks/job-radar.md (updated 2026-07-28)
|
||||
- [ 67d] runbooks/multiplexer-setup.md (updated 2026-07-28)
|
||||
- [ 66d] systems/opencode-go.md (updated 2026-07-29)
|
||||
- [ 65d] systems/hermes-ui.md (updated 2026-07-30)
|
||||
- [ 63d] incidents/2026-08-01-mnemosyne-capture-failure.md (updated 2026-08-01)
|
||||
- [ 63d] systems/memory-backend.md (updated 2026-08-01)
|
||||
- [ 62d] comparisons/steel-vs-camoufox.md (updated 2026-08-02)
|
||||
- [ 62d] concepts/browser-automation.md (updated 2026-08-02)
|
||||
- [ 62d] decisions/rollback-steel-to-camofox.md (updated 2026-08-02)
|
||||
- [ 62d] reports/uber-eats/index.md (updated 2026-08-02)
|
||||
- [ 62d] systems/browser-backend.md (updated 2026-08-02)
|
||||
- [ 57d] systems/nanobot.md (updated 2026-08-07)
|
||||
- [ 52d] comparisons/memory-backends.md (updated 2026-08-12)
|
||||
- [ 52d] comparisons/model-routers.md (updated 2026-08-12)
|
||||
- [ 51d] infrastructure/proxmox.md (updated 2026-08-13)
|
||||
- [ 50d] infrastructure/hosts.md (updated 2026-08-14)
|
||||
- [ 50d] systems/guanaco.md (updated 2026-08-14)
|
||||
- [ 50d] systems/omniroute.md (updated 2026-08-14)
|
||||
- [ 49d] agents/hermes-production.md (updated 2026-08-15)
|
||||
- [ 49d] agents/index.md (updated 2026-08-15)
|
||||
- [ 49d] agents/open-claw.md (updated 2026-08-15)
|
||||
- [ 49d] agents/qwenpaw.md (updated 2026-08-15)
|
||||
- [ 49d] agents/zeroclaw.md (updated 2026-08-15)
|
||||
- [ 49d] incidents/2026-08-15-skills-index-drift.md (updated 2026-08-15)
|
||||
- [ 48d] CONTRIBUTING.md (updated 2026-08-16)
|
||||
- [ 47d] agents/hermes-chaos.md (updated 2026-08-17)
|
||||
- [ 47d] agents/nanobot.md (updated 2026-08-17)
|
||||
- [ 47d] systems/nextcloud.md (updated 2026-08-17)
|
||||
- [ 46d] incidents/2026-08-18-mnemosyne-capture-pipeline.md (updated 2026-08-18)
|
||||
- [ 45d] systems/ignis.md (updated 2026-08-19)
|
||||
- [ 36d] reports/communities-wa-social-media-audit-2026.md (updated 2026-08-28)
|
||||
- [ 35d] incidents/2026-08-29-gmail-read-state.md (updated 2026-08-29)
|
||||
- [ 29d] incidents/2026-09-02-sftpgo-outage.md (updated 2026-09-04)
|
||||
|
||||
## FRESH (11)
|
||||
|
||||
- [ 1d] current-state.md
|
||||
- [ 3d] incidents/2026-09-28-hermes-managed-runtime-deps.md
|
||||
- [ 1d] incidents/2026-10-01-cliproxy-connection-auth-expired.md
|
||||
- [ 12d] infrastructure/tailscale.md
|
||||
- [ 1d] systems/current-profile.md
|
||||
- [ 2d] systems/hermes-agent.md
|
||||
- [ 1d] systems/messaging-integrations.md
|
||||
- [ 2d] systems/mnemosyne.md
|
||||
- [ 2d] systems/model-providers.md
|
||||
- [ 1d] systems/scheduled-tasks.md
|
||||
- [ 0d] systems/skills-index.md
|
||||
|
||||
## NO updated DATE (7)
|
||||
|
||||
- SCHEMA.md
|
||||
- experiments/active/example.md
|
||||
- incidents/2026-07-29-mnemosyne-relay.md
|
||||
- incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
|
||||
- index.md
|
||||
- log.md
|
||||
- reports/uber-eats/2026-07.md
|
||||
|
||||
TOTAL: 68 stale, 11 fresh, 7 no-date of 86 pages
|
||||
+11
-8
@@ -3,11 +3,11 @@ title: Hermes Agent
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-01
|
||||
verified_on: 2026-10-01
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
confidence: high
|
||||
tags: [system, hermes, agent, llm]
|
||||
sources: [live-hermes-version, live-config]
|
||||
sources: [live-hermes-version, live-config, live-skill-inventory]
|
||||
---
|
||||
|
||||
# Hermes Agent
|
||||
@@ -74,9 +74,10 @@ Primary personal agent for Anthony. Runs automation, research, communication, kn
|
||||
- See [[terminal-backend]] for details.
|
||||
|
||||
## Skills
|
||||
- **Total installed:** 344 `SKILL.md` files across 72 top-level skill directories (live count, 2026-10-01; the last full stocktake on 2026-09-25 recorded 341)
|
||||
- **Total installed:** 344 `SKILL.md` files across 72 top-level skill directories (live count 2026-10-03; re-confirmed by the manual stocktake the same day — the last generated [[skills-index]] also reads 344 / 70 categories)
|
||||
- Newest addition: `fleet-wide-failure-triage` (devops) — surfaced by the 2026-10-03 regeneration
|
||||
- Key skills: hermes, hermes-gateway-troubleshooting, hermes-mcp-troubleshooting, hermes-api-vault, omniroute, job-search-automation, weekly-review, fresh-rss-intelligence-filter, etc.
|
||||
- See [[skills-index]] for categorized list (auto-regenerated daily).
|
||||
- See [[skills-index]] for categorized list (auto-regenerated daily — but the generating job is currently blocked, so it is being refreshed by hand; see Known Issues).
|
||||
|
||||
## Multi-Profile Multiplexer
|
||||
|
||||
@@ -106,7 +107,7 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
||||
- **Telegram:** enabled, `allowed_chats: -1004321904721,-1003914987043,-1003932503629`, rich messages
|
||||
- **Discord:** enabled, streaming true
|
||||
- **Slack:** enabled
|
||||
- **Email:** connected
|
||||
- **Email:** ⚠ partially connected — personal Gmail works; the `work` Outlook account is unreachable (tenant allows only `XOAuth2`, the account uses password auth). See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||||
- **Mattermost:** connected; gateway warning on 2026-08-15 was caused by a disabled system-level `hermes-gateway.service` unit, not a live duplicate gateway. Only the user-level service is active.
|
||||
- **ntfy:** connected
|
||||
- **Google Chat:** available (`hermes-google_chat`)
|
||||
@@ -115,7 +116,8 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
||||
- See [[messaging-integrations]] for details.
|
||||
|
||||
## Scheduled Tasks
|
||||
- **Cron jobs:** 30 total (27 enabled, 3 disabled) — verified 2026-10-01
|
||||
- **Cron jobs:** 30 total (27 enabled, 3 disabled) — verified 2026-10-03
|
||||
- **Failing: 6** — `dca8482e4f76` (403 plan-gated), `1a8e90ea2271` (HTTP 410, retired model), `fad5a86979d4` (HTTP 400), `0e753e9720fd` (HTTP 400), `06a0a1c8704f` and `1b23c042e17d` (HTTP 401 — the `cliprox/terra` provider connection auth expired, now confirmed on 2 of the 3 jobs sharing that pin)
|
||||
- Delivery targets: Telegram (primary), Discord, origin, local
|
||||
- Notable: Birthday checker, job radar, newsletter digests, GitHub trending, mnemosyne health/watchdog/backup, AgentMail watcher, skills stock-take regeneration
|
||||
- See [[scheduled-tasks]] for full table.
|
||||
@@ -129,7 +131,8 @@ The `allowed_chats` gate runs on the **default profile's adapter** before `profi
|
||||
|
||||
## Known Issues
|
||||
- Managed-runtime venvs previously drifted on dependencies, breaking cron workers — now guarded every 15 min. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||
- **Three cron jobs are pinned to models that no longer exist upstream and cannot self-recover** (`1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25, HTTP 410; `dca8482e4f76` → `gpt-5.6-terra` plan-gated, 403 `MODEL_NOT_IN_PLAN`; `fad5a86979d4` → `deepseek-v4-flash` absent from the NVIDIA live catalog, HTTP 400). Repinning to the `smart-route` alias is the durable fix. See [[systems/scheduled-tasks]] and [[systems/model-providers]].
|
||||
- **Four cron jobs cannot self-recover — three pinned to models that no longer exist upstream, plus a credential fault:** `1a8e90ea2271` → `qwen3.5:397b` retired 2026-09-25 (HTTP 410); `dca8482e4f76` → `gpt-5.6-terra` plan-gated (403 `MODEL_NOT_IN_PLAN`); `fad5a86979d4` + `0e753e9720fd` → `deepseek-v4-flash` absent from the NVIDIA live catalog (HTTP 400); and `06a0a1c8704f` + `1b23c042e17d` → the `cliprox/terra` provider *connection* auth expired (HTTP 401), now confirmed on 2 of the 3 jobs sharing that pin. Repinning to the `smart-route` alias is the durable fix for the model faults; the cliprox pair needs the OmniRoute connection re-authenticated first. See [[systems/scheduled-tasks]], [[systems/model-providers]] and [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **The daily skills stock-take is not running.** Job `1a8e90ea2271` is blocked on the retired model above, so [[skills-index]] and the freshness audit are only refreshed when `wiki-maintain.sh` is run by hand (last done 2026-10-03). The script itself is healthy — only the scheduled wrapper is broken.
|
||||
- **Telegram thread delivery failures — root-caused and fixed 2026-09-30.** A bare `deliver: "telegram"` resolves to the home DM (`1793951355`), where group forum topic IDs cannot exist; the send hard-failed and the message was lost. Two jobs were retargeted to `telegram:-1004321904721:102` and verified. Three jobs still use bare `telegram` targets. See [[systems/messaging-integrations]].
|
||||
- **Two wiki trees drift:** `/home/hermes/workspace/agent-estate-wiki` (Gitea + Quartz source, pages at root) and `/home/hermes/wiki/agent-estate-wiki` (Obsidian/Ignis vault, pages under `wiki/`). They are separate git repos with different history. See [[current-state]].
|
||||
- **Vault repo git objects can be unwritable:** `/home/hermes/wiki/.git` still holds root-owned object fanout dirs/objects (`04`, `42`, `82`, `91`) left by root-run maintenance commits on 2026-07-22. When a new blob or tree hashes into one of those fanouts, `git commit` dies with `insufficient permission for adding an object to repository database` (even though `git add`/`write-tree` look fine). Workaround that works: build the objects in a writable store via `GIT_OBJECT_DIRECTORY=<tmp>` + `GIT_ALTERNATE_OBJECT_DIRECTORIES=/home/hermes/wiki/.git/objects`, then `git pack-objects /home/hermes/wiki/.git/objects/pack/pack-<name>` and move the branch with `git update-ref refs/heads/master <commit>`. Permanent fix is a root `chown -R hermes:hermes /home/hermes/wiki/.git`. Do **not** run overwrite/rename probes against `.git/objects`: on 2026-09-30 one such probe overwrote loose tree `0d23eba37eb65f5e5021f558a370358f1e394c58` (breaking the 2026-07-22 commit `9141a88`); it was reconstructed byte-exactly from its parent tree `d8185c99` plus the two replaced children (`comparisons` tree `822b90a7`, `log.md` blob `043dc1cd`) and reinserted, and `git fsck` is clean again.
|
||||
|
||||
@@ -3,11 +3,11 @@ title: Messaging Integrations
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-02
|
||||
verified_on: 2026-10-02
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
confidence: high
|
||||
tags: [messaging, telegram, discord, email, integrations]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||||
---
|
||||
|
||||
# Messaging Integrations
|
||||
@@ -34,9 +34,13 @@ sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json]
|
||||
- **Platform toolsets:** cli, discord
|
||||
|
||||
## Email
|
||||
- **Status:** connected
|
||||
- **Libraries:** himalaya CLI available
|
||||
- **Platform toolsets:** cli
|
||||
- **Status: ⚠ partially broken — the personal account works, the `work` account does not.**
|
||||
- **Personal (Gmail):** connected and usable. Google still permits app-password auth.
|
||||
- **`work` (Outlook / Microsoft 365):** **unreachable.** `himalaya envelope list -a work` fails with `Error: cannot build IMAP client / Caused by: login not supported`. The account is configured for password auth (`backend.auth.type = "password"` + `~/.config/himalaya/get_work_outlook_password.sh`) while the tenant advertises **only `XOAuth2`** — basic auth for IMAP/SMTP is disabled tenant-wide. The TCP/TLS connection to `outlook.office365.com:993` succeeds; only the login is refused. This is a policy wall, **not a wrong password** — re-running the credential helper cannot fix it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||||
- Fix path: OAuth2 (`backend.auth.type = "oauth2"` + token store) after an app registration with IMAP/SMTP scopes; Anthony's normal Office sign-in is number-matching approval in Microsoft Authenticator, which is the device-code flow this needs. Tenant admin consent may be required.
|
||||
- Consequence for tooling: `email/mail-lookup` resolves the account before searching, so a request for work mail will reach this wall and must report the account as unreachable — not as "no results found".
|
||||
|
||||
## Slack
|
||||
- **Status:** available (toolsets: cli, slack-platform)
|
||||
|
||||
+30
-22
@@ -3,43 +3,45 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-02
|
||||
verified_on: 2026-10-02
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
sources: [live-cron-list, live-jobs-json]
|
||||
sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
---
|
||||
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-02)
|
||||
- **Failing (last run):** 6 —
|
||||
- `dca8482e4f76` — 403 `MODEL_NOT_IN_PLAN`, streak 6
|
||||
- `1a8e90ea2271` — HTTP 410, model retired upstream, streak 8
|
||||
- **Total jobs:** 30 — 27 enabled, 3 disabled (verified from live `~/.hermes/cron/jobs.json`, 2026-10-03)
|
||||
- **Failing (last run failed outright):** 6 —
|
||||
- `dca8482e4f76` — 403 `MODEL_NOT_IN_PLAN`, streak 7
|
||||
- `1a8e90ea2271` — HTTP 410, model retired upstream, streak 7
|
||||
- `fad5a86979d4` — HTTP 400, model absent from the NVIDIA live catalog, streak 3 (last run 2026-09-30)
|
||||
- `1b23c042e17d` — **new 2026-10-02** — HTTP 401, provider-connection auth expired (same pin as `06a0a1c8704f`)
|
||||
- `06a0a1c8704f` — **new 2026-10-01** — HTTP 401, provider-connection auth expired
|
||||
- `0e753e9720fd` — **new 2026-10-01** — HTTP 400, same NVIDIA pin as `fad5a86979d4`
|
||||
- `066bb5191280` — **new 2026-10-01** — delivery failed (DM thread); the report itself generated
|
||||
- **Stale status flag:** 1 — `1b23c042e17d` still reads `delivery_failed` from its pre-fix run. Its delivery target was corrected on 2026-09-30; the next run (2026-10-02 08:00) clears the flag — but that run is also exposed to the `cliprox/terra` auth fault below.
|
||||
- **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. One of the three (`06a0a1c8704f`) has already failed on that pin.
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02)
|
||||
- **Failing (delivery only):** 1 — `066bb5191280` (2026-10-01): the report itself generated; the send failed against the DM thread.
|
||||
- **Resolved since the last entry:** 1 — `1b23c042e17d`'s stale `delivery_failed` flag cleared exactly as predicted. Its 2026-10-02 run delivered to topic `102` (`delivery_outcome: delivered`); it then failed on the model pin, above.
|
||||
- **Shared model pin at risk:** 3 enabled jobs run `cliprox/terra` — `06a0a1c8704f`, `1b23c042e17d`, `79f9bb41ecf5`. **Two of the three have now failed on that pin.** The third (`79f9bb41ecf5`) last ran 2026-09-27 — before the credential lapsed — and is next due 2026-10-04 18:00. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **Streak recount (2026-10-03):** recounted directly from `~/.hermes/cron/executions.db`. `dca8482e4f76` 6 → 7 (one more daily run since), and `1a8e90ea2271` 8 → **7** — the ×8 carried in the 2026-10-02 entry was an overcount: that job has exactly 7 recorded runs and all 7 failed.
|
||||
- Historic count progression: 26 (2026-09-19) → 30 (2026-09-25) → 30 (2026-09-30) → 30 (2026-10-01) → 30 (2026-10-02) → 30 (2026-10-03)
|
||||
|
||||
## Job Table
|
||||
|
||||
| Name | ID | Schedule | Delivery | Skills | Script | Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| AgentMail Rhino Inbox Poll | `0a7f933d06e2` | `* * * * *` | origin | terminal | `agentmail-poll.py` | **disabled** since 2026-07-27 (legacy) |
|
||||
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ⚠ HTTP 401 ×1 (2026-10-01) — `cliprox/terra` provider connection auth expired; shares the pin with `1b23c042e17d`, `79f9bb41ecf5` |
|
||||
| AI Newsletter Curation for Patrick (Weekly) | `06a0a1c8704f` | `0 10 * * 4` | telegram:-1004321904721:346 | — | — | ⚠ HTTP 401 ×1 (2026-10-01) — `cliprox/terra` provider connection auth expired; **a second job on this pin failed 2026-10-02** (`1b23c042e17d`), so this is a confirmed credential fault, not a fluke. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]] |
|
||||
| Daily Birthday Checker | `a432898b55a4` | `30 7 * * *` | telegram:1793951355 | — | `birthday_check.py` | |
|
||||
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×6 — see Known Issues |
|
||||
| Daily Marketing Job Radar | `dca8482e4f76` | `0 7 * * 1-5` | telegram:-1004321904721:1030 | job-search-automation | — | ⚠ 403 `MODEL_NOT_IN_PLAN` ×7 — see Known Issues |
|
||||
| DHL Tracking - Parcel 8052838382 | `62619aa6bcb3` | `0 8,11,14,17,20 * * *` | telegram:1793951355 | — | `dhl_tracking_monitor.py` | no-agent; added 2026-09-24 |
|
||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | delivery target corrected 2026-09-30 (was bare `telegram` → home DM); `delivery_failed` flag is stale. Runs on `cliprox/terra` — same pin that failed 2026-10-01 |
|
||||
| Enhanced Weekend Adventure Planner | `1b23c042e17d` | `0 8 * * 5` | telegram:-1004321904721:102 | — | — | 2026-09-30 delivery-target fix **confirmed working** (delivered 2026-10-02, flag cleared). ⚠ Now HTTP 401 ×1 (2026-10-02) — `cliprox/terra` connection auth expired, same pin as `06a0a1c8704f`; no plan produced |
|
||||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ⚠ HTTP 400 ×1 (2026-10-01) — `nvidia/deepseek-ai/deepseek-v4-flash` not in the active NVIDIA catalog; delivery also failed on the DM thread. Bare `telegram` → home DM; retarget to a group topic |
|
||||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ⚠ HTTP 400 ×3 — `nvidia/deepseek-ai/deepseek-v4-flash` not in the active NVIDIA catalog; bare `telegram` target should move to a group topic |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | |
|
||||
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ Telegram bot token in `github-trending-radar/config.json` revoked (`getMe` → 401); digest builds, push fails |
|
||||
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | Token-revocation claim **not reproduced at the last recorded run** (2026-09-30 19:22, off-schedule: completed and delivered). The revoked-token (`getMe` → 401) diagnosis comes from an earlier run; the next scheduled run Mon 2026-10-05 09:00 confirms whether it still holds |
|
||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | |
|
||||
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
||||
| Job Tracker Dashboard Refresh | `2fd50c87b0d0` | `0 */2 * * *` | telegram:-1004321904721:1030 | — | `dashboard_watchdog.py` | no-agent |
|
||||
@@ -54,10 +56,10 @@ sources: [live-cron-list, live-jobs-json]
|
||||
| Rhino AgentMail Inbox Watcher | `f77b7e46bd09` | `every 5m` | origin | — | `agentmail-rhino-watch.sh` | **disabled** since 2026-07-27 (legacy) |
|
||||
| Tech & AI Newsletter Digest (Analytical) | `85edf6772bca` | `0 7 * * *` | telegram:-1004321904721:346 | — | — | |
|
||||
| Uber Eats Monthly Spend Report | `066bb5191280` | `0 2 1 * *` | telegram | camofox-browser-automation, browser-backend-verification | — | ⚠ delivery failed 2026-10-01 (`configured thread_id 34795 for telegram:1793951355 was not found`) — the 2026-09 report generated fine, only the send failed. Bare `telegram` → home DM; retarget to a group topic |
|
||||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | runs on `cliprox/terra` — same pin that failed 2026-10-01 |
|
||||
| Weekly Review | `79f9bb41ecf5` | `0 18 * * 0` | origin | weekly-review | — | runs on `cliprox/terra`. Last run 2026-09-27 completed and delivered — **before** the connection lapsed. Next due 2026-10-04 18:00: the last untested job on this pin |
|
||||
| weekly-save-restricted-bot-restart | `a688ed7b7ab0` | `0 3 * * 0` | local | — | `restart-save-restricted-bot.sh` | no-agent; script restored 2026-09-30 |
|
||||
| Wiki Audit — Nightly | `cbfe434a3b5e` | `59 23 * * *` | local | task-approach | — | Reviews past 24h for wiki changes |
|
||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ⚠ HTTP 410 ×8 — `qwen3.5:397b` retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea |
|
||||
| wiki-stocktake-maintenance | `1a8e90ea2271` | `0 3 * * *` | local | — | `wiki-maintain.sh` | ⚠ HTTP 410 ×7 — `qwen3.5:397b` retired 2026-09-25; needs repinning. Regenerates skills-index, freshness audit, rebuilds Quartz, commits to Gitea. **The script itself is healthy** — run by hand on 2026-10-03 and completed cleanly |
|
||||
| 🦝 Hermes Config Backup to Gitea (Daily) | `e27da6c34bbb` | `0 11 * * *` | origin | — | `hermes-config-backup.sh` | no-agent |
|
||||
|
||||
## Removed Jobs
|
||||
@@ -81,16 +83,19 @@ sources: [live-cron-list, live-jobs-json]
|
||||
- **Managed-runtime dependency failure (2026-09-28 → 30):** Jobs failed with `Restart-safe cron worker dispatch failed … external worker exited before ownership acknowledgement (exit 1)`. Real cause was `ModuleNotFoundError: No module named 'dotenv'` / `'mnemosyne'` in uv-managed runtime venvs. Mnemosyne Watchdog failed 176 consecutive runs. Fixed by `hermes-runtime-deps-guard.sh` v2, run every 15 min by job `96eb95ce12f2`. All managed venvs verified. See [[incidents/2026-09-28-hermes-managed-runtime-deps]].
|
||||
- **`weekly-save-restricted-bot-restart` missing script (2026-08-30):** Previously failed with "Script not found: `restart-save-restricted-bot.sh`". The script existed at `/home/hermes/scripts/` but cron only resolves inside the profile folder; copied in 2026-09-30 19:17. Job now `ok`, streak 0.
|
||||
- **Residual pre-fix dispatch errors cleared:** `cbfe434a3b5e` returned to `ok` on its next run as predicted. The `1a8e90ea2271` error of the same era also cleared — its current failure is a *different*, upstream problem (below).
|
||||
- **Stale `delivery_failed` flag on `1b23c042e17d` cleared (confirmed 2026-10-02):** the 2026-09-30 retarget held — the job's next run delivered to topic `102`.
|
||||
|
||||
### Open
|
||||
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-01, streak 6):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
|
||||
- **`cliprox/terra` provider connection auth expired — NEW 2026-10-01 (streak 1):** Job `06a0a1c8704f` (AI Newsletter Curation for Patrick) failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Its failure notice reached topic 346; no digest was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. **Two more enabled jobs run the same pin** (`1b23c042e17d` Enhanced Weekend Adventure Planner, `79f9bb41ecf5` Weekly Review) and are expected to fail on their next runs. Re-authenticate the OmniRoute connection before repinning anything.
|
||||
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 8):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. Until then the daily 03:00 stocktake (skills-index regeneration, freshness audit, Quartz rebuild, Gitea commit) does not run. The page most visibly affected is [[systems/skills-index]], whose last generated run was 2026-09-27 (341 skills) against 344 skills live on 2026-10-02.
|
||||
- **Daily Marketing Job Radar — 403 `MODEL_NOT_IN_PLAN` (2026-09-26 → 10-02, streak 7):** Job `dca8482e4f76` is pinned to `custom:omni-101` / `gpt-5.6-terra`; the upstream (`command-code`) rejects it with `GPT-5.6 Terra available in Pro and above plans`. Anthony's primary job pipeline — needs a model or provider change; it cannot self-recover. See [[runbooks/job-radar]].
|
||||
- **`cliprox/terra` provider connection auth expired — 2 of 3 jobs confirmed (streak 1 each):** Jobs `06a0a1c8704f` (2026-10-01) and `1b23c042e17d` (2026-10-02) both failed with `HTTP 401: [openai-compatible-chat-2ea20f63-…] All 1 connection(s) authentication expired — please reconnect in the dashboard` (provider `custom:Omniroute:20128`, model `cliprox/terra`). Failure notices reached their topics; no output was produced. This is a *credential/connection* problem at the proxy, not a missing model — it will not self-recover until the connection is re-authenticated. `79f9bb41ecf5` is the remaining untested job on this pin (next run 2026-10-04 18:00). Re-authenticate the OmniRoute connection before repinning anything. See [[incidents/2026-10-01-cliproxy-connection-auth-expired]].
|
||||
- **A delivered notice is not a successful job:** both `cliprox/terra` failures recorded `delivery_outcome: delivered` because the *failure notice* is itself a deliverable. Read the execution `status` column, not the delivery outcome, when judging whether a job produced its output.
|
||||
- **wiki-stocktake-maintenance — HTTP 410, model retired (streak 7):** Job `1a8e90ea2271` is pinned to `ollama-cloud / qwen3.5:397b`, which Ollama retired on 2026-09-25 (`qwen3.5:397b was retired at 2026-09-25 00:00:00 -0700 PDT`). It cannot self-recover — it needs a new model pin. **The job's script is fine:** `wiki-maintain.sh` was run by hand on 2026-10-03 and completed cleanly, regenerating [[systems/skills-index]] (344 skills / 70 categories) and refreshing the freshness audit. Only the scheduled wrapper is broken — so every non-model step of the daily stocktake needs a manual run until it is repinned.
|
||||
- **Facebook Birthday scrapes — HTTP 400, model missing from catalog (streak 3 / 1):** Job `fad5a86979d4` (weekly) is pinned to `nvidia/deepseek-ai/deepseek-v4-flash`; the provider reports the model is `not available in the active live catalog for provider 'nvidia'`. **Confirmed as a shared-pin issue on 2026-10-01**, when sibling job `0e753e9720fd` (monthly) failed on the same pin at its own 09:00 run. Safest durable fix is repinning both to a routing alias (`smart-route`) instead of a raw model name.
|
||||
- **GitHub Trending Radar — revoked Telegram bot token:** Job `c0a9fa314542` builds its digest successfully, then fails the Telegram push (`getMe` → 401 Unauthorized). The token lives in `/home/hermes/workspace/github-trending-radar/config.json` and needs replacing from @BotFather. Digest content is recoverable from the script's `--dry-run` output in the interim.
|
||||
- **GitHub Trending Radar — revoked Telegram bot token (claim now unverified):** The recorded diagnosis is that job `c0a9fa314542` builds its digest successfully, then fails the Telegram push (`getMe` → 401 Unauthorized), with the token in `/home/hermes/workspace/github-trending-radar/config.json` needing replacement from @BotFather. However its **last recorded run (2026-09-30 19:22, off-schedule) completed and delivered**, so the failure was not reproduced then. Treat "token revoked" as needing re-confirmation at the next scheduled run (Mon 2026-10-05 09:00) rather than as a standing fact. Digest content remains recoverable from the script's `--dry-run` output in the interim.
|
||||
- **Stale provider key `custom:omni` (2 jobs):** the provider was removed from `custom_providers` (the live 2026-10-01 set is `litellm`, `guacano`, `Omniroute:20128`, `command-code`, `omni-101`). Jobs `e530c89f2be0` and `d40085da631c` still name it; both run `smart-route`, and the default `custom` provider resolves to the same base URL (`http://100.78.63.20:20128`), so no failure has been observed — but the references should be normalised to `custom:omni-101`.
|
||||
- **Bare `telegram` delivery targets (3 jobs) — now demonstrably breaking sends:** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with `configured thread_id 34795 for telegram:1793951355 was not found`. Work generated correctly; the delivery did not. Retarget all three to `telegram:-1004321904721:<topic>`. See [[systems/messaging-integrations]].
|
||||
- **Bare `telegram` delivery targets (3 jobs) — demonstrably breaking sends:** `0e753e9720fd`, `fad5a86979d4`, `066bb5191280` deliver to the home DM rather than a group topic. As of 2026-10-01 this is no longer theoretical: `066bb5191280` (02:11) and `0e753e9720fd` (09:01) both recorded `delivery_outcome: failed` with `configured thread_id 34795 for telegram:1793951355 was not found`. Work generated correctly; the delivery did not. Retarget all three to `telegram:-1004321904721:<topic>`. See [[systems/messaging-integrations]].
|
||||
- **Two wiki trees, drifted in both directions (NEW 2026-10-02):** `/home/hermes/workspace/agent-estate-wiki` is the canonical source (Gitea remote `Anthony/agent-estate-wiki`, Quartz build, `wiki-maintain.sh` target). `/home/hermes/wiki/agent-estate-wiki` is the **Obsidian/Ignis vault copy** the iPad reads — pages nested under `wiki/`, and **no git remote at all**. The nightly audit job's prompt names only the vault path, so edits that miss the canonical tree never reach Gitea or the published site. The two have also diverged *backwards* (`systems/ignis.md`, `systems/nextcloud.md`, three `incidents/` pages exist only in the vault; `raw/stocktake/*`, `agents/`, `incidents/2026-09-02-sftpgo-outage.md` only in canonical) — a naive one-way sync deletes pages. See [[systems/hermes-agent]].
|
||||
- **Work mailbox unreachable (non-cron; affects on-demand lookups):** the `work` Outlook account is configured with password auth while the tenant advertises only `XOAuth2`, so `himalaya -a work` fails with `login not supported`. No cron job polls it. See [[incidents/2026-08-26-work-outlook-basic-auth-disabled]].
|
||||
|
||||
### Historical
|
||||
- **Mnemosyne Watchdog incident (2026-08-19):** Job `96eb95ce12f2` cycled for ~2 hours reporting no consolidation for 36–41.5+ hours. See [[systems/mnemosyne]].
|
||||
@@ -102,4 +107,7 @@ sources: [live-cron-list, live-jobs-json]
|
||||
- [[systems/mnemosyne]]
|
||||
- [[runbooks/job-radar]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[systems/messaging-integrations]]
|
||||
- [[incidents/2026-09-28-hermes-managed-runtime-deps]]
|
||||
- [[incidents/2026-10-01-cliproxy-connection-auth-expired]]
|
||||
- [[incidents/2026-08-26-work-outlook-basic-auth-disabled]]
|
||||
|
||||
+11
-5
@@ -3,8 +3,8 @@ title: Installed Skills
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-09-27
|
||||
verified_on: 2026-09-27
|
||||
updated: 2026-10-03
|
||||
verified_on: 2026-10-03
|
||||
confidence: high
|
||||
tags: [skills, hermes]
|
||||
sources: []
|
||||
@@ -12,7 +12,7 @@ sources: []
|
||||
|
||||
# Installed Skills
|
||||
|
||||
**Total installed: 341 skills** in `~/.hermes/skills/`.
|
||||
**Total installed: 344 skills** in `~/.hermes/skills/`.
|
||||
|
||||
This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
@@ -132,7 +132,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `ddia-systems` — # Designing Data-Intensive Applications Framework
|
||||
|
||||
## Devops (111)
|
||||
## Devops (112)
|
||||
|
||||
- `abyss-ios-pairing` — # Pair iPhone with The Abyss (iOS)
|
||||
- `agent-extension-audit` — # Agent Extension Audit
|
||||
@@ -159,6 +159,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
- `docker-management` — # Docker Management
|
||||
- `external-auth-script-integration` — # External Auth Script Integration
|
||||
- `external-skill-integration` — # External Skill Integration
|
||||
- `fleet-wide-failure-triage` — # Fleet-Wide Failure Triage
|
||||
- `free-web-tools` — # Free Web Search & Extract
|
||||
- `freshrss-intelligence-filter` — # FreshRSS Intelligence Filter
|
||||
- `gcp-vertex-ai-provider` — # GCP Vertex AI Provider
|
||||
@@ -270,7 +271,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `domain-driven-design` — # Domain-Driven Design Framework
|
||||
|
||||
## Email (16)
|
||||
## Email (17)
|
||||
|
||||
- `agent-email-patterns` — # Agent Email Patterns
|
||||
- `agentmail` — # AgentMail — Agent-Owned Email Inboxes
|
||||
@@ -287,6 +288,7 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
- `email-inbox-triage` — # Email Inbox Triage
|
||||
- `gmail-readonly-fetch` — # Gmail Read-Only Fetch (Large Inbox)
|
||||
- `himalaya` — # Himalaya Email CLI
|
||||
- `mail-lookup` — # Ad-Hoc Mail Lookup
|
||||
- `realtime-email-monitor` — # Real-time Email Monitor (IMAP IDLE)
|
||||
|
||||
## Email Himalaya (1)
|
||||
@@ -340,6 +342,10 @@ This page is auto-regenerated from the live skill inventory. Do not hand-edit.
|
||||
|
||||
- `hermes-mcp-troubleshooting` — # Hermes MCP Troubleshooting
|
||||
|
||||
## Hermes Runtime Worker Diagnostics (1)
|
||||
|
||||
- `hermes-runtime-worker-diagnostics` — # Hermes Runtime / Cron Worker Diagnostics
|
||||
|
||||
## Hermes Themes (1)
|
||||
|
||||
- `hermes-themes` — # Hermes Themes Skill
|
||||
|
||||
Reference in New Issue
Block a user