audit(2026-10-19): degenerate-stream incident page + push-notifications page; refresh scheduled-tasks, current-state, index, log, messaging, mnemosyne, tailscale, proxmox
This commit is contained in:
+37
-3
@@ -3,8 +3,8 @@ title: Current State
|
||||
type: concept
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [system]
|
||||
@@ -12,7 +12,41 @@ sources: [live-cron-list, live-sqlite-cron-executions, live-sqlite-sessions, liv
|
||||
---
|
||||
|
||||
# Current State
|
||||
Updated: 2026-10-18
|
||||
Updated: 2026-10-19
|
||||
|
||||
## Recent Changes (2026-10-19, nightly audit; live probes 2026-10-19 00:04-00:2x AWST)
|
||||
|
||||
- **A model stream started repeating itself — and Hermes delivered the placeholder to Telegram before it
|
||||
aborted.** The FreshRSS Midday slot (`ecc1fc13655a`, 12:00) ended `failed` at 12:08:25 with
|
||||
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the
|
||||
degenerate response.` The run had already written and **delivered** a canned line — *"Your request was
|
||||
not processed. Send it again if you still want me to carry it out."* — at 12:08:11
|
||||
(`delivery_outcome: delivered`), so the digest topic received a non-answer that does not read as an
|
||||
error. **A delivered message is not evidence of delivered content.** Open question: should the guard
|
||||
suppress delivery or re-queue, instead of substituting text on the job's normal route?
|
||||
→ [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
- **Window tally: `10-18 = 168 completed / 1 failed`** — that row is the only failure of the day;
|
||||
`10-17 = 163 / 5`, and `10-19` so far `5 / 0` plus this audit. **Non-clean `last_status` 8 → 7**;
|
||||
**open incidents 8 → 7** (6 `alerted` / 1 `detected`), with **1 opened and 2 resolved** in-window
|
||||
(`cbfe434a3b5e` 00:42:54, `1a8e90ea2271` 03:01:22 — both auto-closed by their own next clean slot).
|
||||
**11 sessions in the window, all `cron`; no interactive sessions.** → [[systems/scheduled-tasks]]
|
||||
- **The 10-18 03:00 stocktake landed and closed the two-day gap** the last audit flagged: `1a8e90ea2271`
|
||||
exited 0, `raw/stocktake/freshness-20261018.txt` exists, [[systems/skills-index]] was regenerated
|
||||
(346 skills / 70 categories; freshness **47 stale / 41 fresh / 7 no-date**) and committed (`dd53e5f`).
|
||||
- **Mattermost's cause is finally known: CT255 is stopped.** The adapter's 5-minute reconnect loop is
|
||||
unchanged (`attempt 471` at 00:14:07; `mattermost connect timed out after 30s`), the peer is offline
|
||||
(last seen **29d**), and the host answers neither TCP `:8065` nor ICMP from CT460. The 10-11 entry's
|
||||
"cause not determined" is now superseded. → [[systems/messaging-integrations]]
|
||||
- **New page: [[systems/push-notifications]] — and the push bridge is broken, and has been for a long
|
||||
time.** `status.json` says `connected: false` with `connection refused (nothing listens there on this
|
||||
machine)`: the bridge dials `HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net`, which resolves
|
||||
on this host to **`127.0.1.1`** via `/etc/hosts` — where nothing serves 443 — while the gateway itself is
|
||||
healthy (`0.0.0.0:9119`, `/` → 302) and the tailnet Serve route is intact. The error string appears
|
||||
**23,469** times in the retained logs. Fix not applied (change the dial target, or the name resolution).
|
||||
→ [[systems/push-notifications]]
|
||||
- **One more standing alert, unchanged: the GitHub Trending Radar delivery token is still dead**
|
||||
(`getMe` → 401, re-verified 10-18), and its next exercise is **Mon 2026-10-19 09:00**. Not repaired — a
|
||||
replacement bot token is Anthony's call. → [[systems/scheduled-tasks]]
|
||||
|
||||
## Recent Changes (2026-10-18, nightly audit; live probes 2026-10-18 00:31-00:36 AWST)
|
||||
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
title: Degenerate Model Stream Delivered a Canned Non-Answer — FreshRSS Midday Briefing
|
||||
type: incident
|
||||
status: active
|
||||
created: 2026-10-18
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [cron, agent-loop, delivery, freshrss, provider]
|
||||
sources: [live-sqlite-cron-executions, live-sqlite-cron-incidents, live-sessions, live-logs]
|
||||
---
|
||||
|
||||
# Degenerate Model Stream Delivered a Canned Non-Answer — FreshRSS Midday Briefing (2026-10-18)
|
||||
|
||||
## What happened
|
||||
|
||||
The 2026-10-18 12:00 slot of **FreshRSS Midday News Briefing** (`ecc1fc13655a`, `0 12 * * *`,
|
||||
delivery `telegram:-1004321904721:346`, pin `custom:omni-101` / `smart-route`) **delivered a
|
||||
placeholder sentence to Telegram instead of the midday briefing**, then failed.
|
||||
|
||||
The delivered message, stored verbatim as the run's assistant output:
|
||||
|
||||
> Your request was not processed. Send it again if you still want me to carry it out.
|
||||
|
||||
## Timeline (AWST; `executions.db` row `c53b69ce19e24d888635b38b7552d200`)
|
||||
|
||||
| Time | Event |
|
||||
|---|---|
|
||||
| 12:00:56 | run claimed |
|
||||
| 12:01:11 | run started |
|
||||
| 12:08:11 | `progress_at` — canned reply written **and delivered** (`delivery_outcome: delivered`) |
|
||||
| 12:08:22 | incident `ecc1fc_21a72aa11cc4` opened — state `alerted`, `failure_type: agent` |
|
||||
| 12:08:25 | run ended `failed` — `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` |
|
||||
|
||||
## Impact
|
||||
|
||||
- The user-visible Sunday midday news briefing for 2026-10-18 was replaced by a non-answer. The run's
|
||||
own incident alert fired, but the Telegram message that arrived did not read as a failure — it read
|
||||
as a terse prompt. **A delivered message is not evidence of delivered content.**
|
||||
- `last_status: error`, `failure_streak: 1`; the row clears on the job's next clean slot
|
||||
(2026-10-19 12:00).
|
||||
- The 2026-10-18 18:01 `79f9bb41ecf5` weekly review run correctly listed it
|
||||
(“Oct 18 (today): FreshRSS Midday News Briefing — repetition loop, stream cut”), so the failure was
|
||||
visible in the weekly health summary even though the delivered message was not.
|
||||
|
||||
## Cause and open questions
|
||||
|
||||
- **Confirmed:** the model stream hit a repetition loop; Hermes's degenerate-stream guard cut the
|
||||
stream, discarded the response, and substituted a canned “not processed” line — which was then
|
||||
**delivered** on the job's normal route before the run aborted.
|
||||
- **Only recorded instance in the retained logs:** the string `repetition loop` appears in
|
||||
`agent.log*` only on 2026-10-18 (2 lines: the guard message and the cron error). Log retention is
|
||||
short, so this is “first recorded”, not “first occurrence”.
|
||||
- **Open:** should the guard **suppress** delivery (or re-queue the run) instead of delivering a
|
||||
non-answer under the job's delivery route? To a reader scanning a digest topic, a canned sentence is
|
||||
indistinguishable from content.
|
||||
- **Open:** whether the repetition loop is a model/provider-quality symptom. The job rides the same
|
||||
`custom:omni-101` alias family that produced the 2026-10-14 → 10-15 connection-refused storm and the
|
||||
2026-10-16 → 10-17 resource-pressure 503 burst. Single-node topology unchanged.
|
||||
|
||||
## Related
|
||||
- [[systems/scheduled-tasks]]
|
||||
- [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||
|
||||
No secrets recorded; job IDs, incident IDs, timestamps and status codes only.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Anthony's Systems Wiki
|
||||
> Operational knowledge for Anthony's agents, infrastructure,
|
||||
> model routing and automation. Last updated: 2026-10-18
|
||||
> model routing and automation. Last updated: 2026-10-19
|
||||
>
|
||||
> ⚠ **Two Proxmox hosts exist — always qualify a guest with its host.** A second unclustered
|
||||
> hypervisor, `m93p` (`100.104.51.98`, PVE 9.2.0), was discovered during the 2026-10-14 stocktake.
|
||||
@@ -32,11 +32,13 @@
|
||||
- [[systems/tool-search]] — Always-on lazy discovery for external tools
|
||||
|
||||
### Integrations
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc.
|
||||
- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **8 carry a non-clean `last_status`** as of 2026-10-18 00:3x — down from 12. The **`omniroute-1` 503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00)** cost **9 model-call failures across 7 jobs** and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See [[incidents/2026-10-16-omniroute-503-resource-pressure]]. Prior incident: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs** and ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. ⚠ **2026-10-17** stocktake slot failed (`HTTP 503`), so `freshness-20261017.txt` is missing and the index is stamped **10-16**; the **10-18 03:00** slot closes it. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
|
||||
- [[systems/messaging-integrations]] — Telegram, Discord, email, Slack, etc. **Mattermost's long-running red now has a cause: CT255 is stopped** (peer offline 29d; host answers neither `:8065` nor ICMP).
|
||||
- [[systems/push-notifications]] — Cadu/Vory push bridge (`hermes-push`, `vory-push`): gateway on `:9119`, state in `~/.hermes/push/`. **Broken — `connected: false`** (the bridge dials a name that resolves to `127.0.1.1` where nothing serves 443); chronic, not a blip.
|
||||
- [[systems/scheduled-tasks]] — **28 cron jobs (25 enabled, 3 disabled)**; **7 carry a non-clean `last_status`** as of 2026-10-19 00:2x — down from 12. The **2026-10-18 window was clean except one run**: `ecc1fc13655a` (FreshRSS Midday) delivered a canned non-answer to Telegram and then died on a model repetition loop → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. The **`omniroute-1` 503 resource-pressure window (2026-10-16 08:00 → 10-17 03:00)** cost **9 model-call failures across 7 jobs** and was followed by a clean 10-17 06:00 → 10-18 window; the Mosquito Tracker was removed (29 → 28 jobs). See [[incidents/2026-10-16-omniroute-503-resource-pressure]]. Prior incident: the **connection-refused storm (2026-10-14 → 10-15)** — **24 model-call failures across 12 jobs** and ~34 h of digests blanked. **All 18 model-calling jobs still sit on `custom:omni-101` with no failover.**
|
||||
- [[systems/skills-index]] — Installed skills inventory (**346 skills / 70 categories**; regenerated daily by the 03:00 stocktake job). Note: 347 `SKILL.md` files exist on disk — the generator de-dups one nested duplicate, see [[systems/scheduled-tasks]]. The **2026-10-17** slot failed (`HTTP 503`), but the **10-18 03:00** slot closed the hole: `freshness-20261018.txt` written, index regenerated and committed (`dd53e5f`) — freshness **47 stale / 41 fresh / 7 no-date**. ⚠ The **2026-10-13** regeneration wrote the file but the stocktake job died before committing, so the artifact sat stranded until the 2026-10-14 audit committed it (date-only change: `updated`/`verified_on` 10-12 → 10-13).
|
||||
|
||||
## Incidents
|
||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]] — **The repetition-loop failure mode, 2026-10-18:** the FreshRSS Midday run (`ecc1fc13655a`) wrote *"Your request was not processed…"*, **delivered it to topic `:346` at 12:08:11**, then died at 12:08:25 on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop`. **A delivered message is not evidence of delivered content** — open question: should the guard suppress delivery / re-queue instead of substituting text on the job's normal route?
|
||||
- [[incidents/2026-10-16-omniroute-503-resource-pressure]] — **OmniRoute's second failure mode, 2026-10-16 → 10-17:** `omniroute-1` load-shed model calls with **`HTTP 503 … resource pressure`** — **9 model calls across 7 jobs** over ~19 h, while `/api/health` stayed `200 ok` and `/v1/models` stayed `401`. **Self-cleared** (last 503 10-17 03:00:28; 10-17 closed 162 / 5, 10-18 so far 7 / 0). 9 incidents opened, 6 closed; **8 open** overall, and the ledger tags **7 of the 9 `failure_type: unknown`**. **Single-node topology still open.**
|
||||
- [[incidents/2026-10-14-omniroute-node-connection-refused-storm]] — **Fleet-wide 34-hour cron blackout, 2026-10-14 → 10-15:** `omniroute-1` (`100.78.63.20:20128`, provider `custom:omni-101`) refused TCP connections, so **every model-calling cron job failed on every slot** — 24 genuine model-call failures across 12 jobs (+12 derivative watchdog reds), and all digests for ~34 h were silently suppressed. **All 18 enabled model-calling jobs ride this one node with no failover.** Recovered 10-15 22:01 (resolved)
|
||||
- [[incidents/2026-10-09-cron-provider-timeout-storm]] — Fleet-wide provider timeout storm: 15 failed executions across 10 jobs on 2026-10-09, all `RuntimeError: Request timed out.` on the aliases behind `custom:omni-101`, 00:00 → 15:00 AWST; **11 of its 12 incidents are now closed** (five on 2026-10-10, the last hold-out `dca848_23fcd2b18aaa` on 2026-10-12 07:33:41). **1 remains `alerted`** — `1b23c042e17d`, the weekly Friday job, which cannot clear before its next slot (Fri 2026-10-16 08:00)
|
||||
@@ -71,7 +73,7 @@
|
||||
|
||||
### Networking
|
||||
- [[infrastructure/proxmox]] — PVE hosts (**two: `pve` 8.x + `m93p` 9.2.0, unclustered, IDs collide**), VMs, LXC containers, storage (`/mnt/ssd0` 88.17 % / ~63 GiB free, 2026-10-14)
|
||||
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **45-node** tailnet inventory (2026-10-14)
|
||||
- [[infrastructure/tailscale]] — Tailscale serve routes (CT460 verified exactly) and the **48-node** tailnet inventory (re-verified 2026-10-19: `guest-vps`, `openmausbot`, `openmausbot-vm` added; `mattermost` offline 29d)
|
||||
- [[systems/cloudflare-tunnel]] — ct207-hosted tunnel serving Hermes publicly
|
||||
|
||||
## Concepts
|
||||
|
||||
@@ -38,7 +38,7 @@ Hypervisor hosting all VMs and LXC containers for the homelab.
|
||||
| 207 | webserver | running | Web server |
|
||||
| 209 | cloudreve | **stopped** | Cloudreve (deprecated) |
|
||||
| 215 | n8n | running | N8n automation |
|
||||
| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer last seen 21d ago) |
|
||||
| 255 | mattermost | **stopped** | Mattermost — unreachable since ~2026-09-20 (Tailscale peer offline, **29d** as of 2026-10-19) |
|
||||
| 270 | nextcloud | running | Nextcloud (apache2 active) |
|
||||
| 299 | sftpgo | running | SFTPGo file server |
|
||||
| 333 | clawtest | running | ClawTest — Nanobot, Zeroclaw |
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Tailscale
|
||||
type: infrastructure
|
||||
status: active
|
||||
created: 2026-07-26
|
||||
updated: 2026-10-16
|
||||
verified_on: 2026-10-16
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
confidence: high
|
||||
tags: [tailscale, vpn, networking, infrastructure]
|
||||
---
|
||||
@@ -19,10 +19,11 @@ WireGuard-based mesh VPN connecting all homelab hosts and providing HTTPS access
|
||||
- **Primary node:** CT460 (`hermes.kangaroo-eel.ts.net`, `100.118.5.51`)
|
||||
- **Runtipi node:** CT201 (`runtipi.kangaroo-eel.ts.net`)
|
||||
|
||||
## Tailscale Node Inventory (verified 2026-10-16)
|
||||
Full node list from `tailscale status` — 46 non-funnel nodes. Offline peers are listed with their last-seen age.
|
||||
## Tailscale Node Inventory (verified 2026-10-19)
|
||||
Full node list from `tailscale status` — **49 non-funnel nodes** (48 live IPv4 peers + `hermes` itself). Offline peers are listed with their last-seen age.
|
||||
Re-verified 2026-10-16: the 2026-10-11 table had drifted (15 nodes missing, several states wrong).
|
||||
Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state, offline ages and the omni/omarchymac rows refreshed from `tailscale status`.
|
||||
Re-verified 2026-10-19 00:2x: three live peers had never been listed — `guest-vps` (100.117.14.84, online), `openmausbot` (100.91.214.91, online) and `openmausbot-vm` (100.77.107.30, online) — added; `mattermost` offline age 26d → **29d**. `tailscale status` reports 71 peers total, of which 23 are Tailscale's own entries (see the note below); the 48 with IPv4 addresses are the homelab set.
|
||||
|
||||
| Node | IP | State |
|
||||
|---|---|---|
|
||||
@@ -44,12 +45,13 @@ Re-verified 2026-10-16: `omarchymac` (100.84.180.69) added; `mattermost` state,
|
||||
| freshrss-new | 100.110.180.115 | idle/offline 59d |
|
||||
| freshrss | 100.83.60.93 | online |
|
||||
| gitea | 100.82.100.128 | online |
|
||||
| guest-vps | 100.117.14.84 | online |
|
||||
| hermesagent | 100.98.205.112 | online |
|
||||
| homeassistant | 100.68.139.5 | active; offers exit node |
|
||||
| litellm | 100.79.29.64 | online |
|
||||
| m93p | 100.104.51.98 | online |
|
||||
| mailflow | 100.115.60.92 | online |
|
||||
| mattermost | 100.98.116.14 | active; relay "syd"; idle/offline 26d — CT255 stopped |
|
||||
| mattermost | 100.98.116.14 | active; relay "syd"; idle/offline 29d — CT255 stopped (re-verified live 2026-10-19) |
|
||||
| n8n | 100.84.124.30 | online |
|
||||
| nextcloud | 100.90.102.19 | online |
|
||||
| ntfy | 100.67.153.116 | active |
|
||||
|
||||
@@ -848,3 +848,67 @@ second provider pin on a subset of jobs, or router-side failover. **Not applied*
|
||||
an audit one.
|
||||
|
||||
No secrets written; host names, IPs, VMIDs, job/incident IDs and probe status codes only.
|
||||
|
||||
## [2026-10-19] Nightly wiki audit — window 2026-10-18 00:00 → 23:59 AWST
|
||||
|
||||
**Window verdict: one failure all day, and it was a failure mode the wiki had never recorded.** Fleet
|
||||
**28 jobs (25 enabled / 3 disabled)**; `10-18 = 168 completed / 1 failed` against `10-17 = 163 / 5`;
|
||||
`10-19` so far `5 / 0` plus this audit. **11 sessions in the window, all `cron` — no interactive
|
||||
sessions**, so no user decisions arrived to document; every finding below is machine-observed.
|
||||
|
||||
- **A model stream repeated itself and Hermes delivered the placeholder to Telegram.** `ecc1fc13655a`
|
||||
(FreshRSS Midday, 12:00) wrote *"Your request was not processed. Send it again if you still want me to
|
||||
carry it out."*, **delivered it** (`delivery_outcome: delivered`, 12:08:11) and then died at 12:08:25 on
|
||||
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the
|
||||
degenerate response.` New page → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. The open
|
||||
question is written down, not fixed: should the degenerate-stream guard suppress delivery (or re-queue)
|
||||
instead of substituting text on the job's normal route?
|
||||
- **Mattermost: cause found — CT255 is stopped**, superseding the 10-11 entry's "cause not determined".
|
||||
Reconnect loop unchanged (`attempt 471` at 00:14:07, `connect timed out after 30s`, 5-minute cadence),
|
||||
peer offline **29d**, and the host answers neither TCP `:8065` nor ICMP. Platform config still says
|
||||
`enabled: true`, so the loop will keep logging until the container is started or the block retired.
|
||||
- **New page: [[systems/push-notifications]] — the bridge is broken and has been for as long as the logs
|
||||
go back.** `status.json`: `connected: false`, `error: …connection refused (nothing listens there on
|
||||
this machine)`; the bridge dials `HERMES_PUSH_GATEWAY_URL=https://hermes.kangaroo-eel.ts.net`, which
|
||||
`/etc/hosts` resolves to **`127.0.1.1`** where nothing serves 443, while the gateway itself is healthy
|
||||
on `0.0.0.0:9119` (`/` → 302) and Serve still maps the name to it. **23,469** occurrences of that string
|
||||
across `agent.log(.1-3)` + `errors.log(.1-2)`. Fix documented, **not applied**.
|
||||
- **Mnemosyne: the 10-18 12:0x red was a capture-staleness false positive, not a break.** `working_memory`
|
||||
had no non-cron row since 10-16 11:07 because no conversation had happened; consolidation ran 10-18
|
||||
03:55 with 0 items; coverage 96.8 % working / 99.4 % episodic (from the 10-18 12:1x read now carried in
|
||||
the page); 1 unresolved conflict. The 24 h threshold in `mnemosyne-health-check.sh:92` will keep
|
||||
flagging idle weekends.
|
||||
- **Stocktake landed and closed the two-day hole:** `1a8e90ea2271` exited 0 on 10-18 03:00,
|
||||
`freshness-20261018.txt` written, [[systems/skills-index]] regenerated (346 skills / 70 categories;
|
||||
freshness 47 stale / 41 fresh / 7 no-date) and committed (`dd53e5f`).
|
||||
- **Carried forward, still open:** the GitHub Trending Radar delivery token (`getMe` → 401, verified
|
||||
10-18) — its next exercise is **Mon 2026-10-19 09:00**; the single-node `custom:omni-101` topology with
|
||||
no model-call failover; and the vault drift below.
|
||||
|
||||
**Pages touched:** `incidents/2026-10-18-degenerate-stream-canned-substitute.md` (**new**);
|
||||
`systems/push-notifications.md` (**new**); `systems/scheduled-tasks.md` (frontmatter + Count bullet +
|
||||
FreshRSS row + totals); `systems/messaging-integrations.md` (frontmatter + new Mattermost section);
|
||||
`systems/mnemosyne.md` (frontmatter + new tier/health block); `infrastructure/tailscale.md` (frontmatter +
|
||||
inventory header + three missing peers + `mattermost` age); `infrastructure/proxmox.md` (CT255 age);
|
||||
`current-state.md`; `index.md`; `log.md`. **Nothing removed or archived** — no page became obsolete and no
|
||||
one-way sync was run.
|
||||
|
||||
**Bridge housekeeping + vault drift (verified 2026-10-19 00:2x).** Of the ten files this run touched, six
|
||||
were byte-identical between the authoring tree and the vault copy *before* the edits; two more
|
||||
(`infrastructure/tailscale.md`, `infrastructure/proxmox.md`) differed only because the vault holds
|
||||
**older** versions (2026-07-27 and 2026-10-12), i.e. the vault never received the 10-14/10-16 tailscale and
|
||||
proxmox work. Files were committed and pushed in the authoring tree, then copied into the vault (`wiki/`
|
||||
nesting) with sha256 verified per file.
|
||||
|
||||
**Vault drift measured:** against authoring `HEAD`, the vault holds **74 identical pages, 11 stale** and
|
||||
**12 missing**. Stale: `infrastructure/network-map.md`, `infrastructure/tailscale.md`,
|
||||
`infrastructure/proxmox.md`, `infrastructure/docker-services.md`, `systems/skills-index.md`,
|
||||
`systems/current-profile.md`, `systems/home-assistant.md`, `systems/ignis.md`, `systems/nanobot.md`,
|
||||
`systems/obsidian.md`, `incidents/2026-08-01-mnemosyne-capture-failure.md`. Missing:
|
||||
`incidents/2026-09-02-sftpgo-outage.md` (flagged by the 10-18 audit and still absent), the seven
|
||||
`agents/*` pages, `reports/communities-wa-social-media-audit-2026.md`, `CONTRIBUTING.md`, plus this run's
|
||||
two new pages (copied immediately after). The vault repo's own HEAD is still `9bacd8e mirror(2026-10-16)`
|
||||
with a working tree carrying uncommitted 10-17/10-18 changes, and it has **no remote**, so it cannot be
|
||||
pushed. **Not fixed** — a full mirror pass is a bigger call than an audit's mandate; flagged here.
|
||||
|
||||
No secrets written; host names, IPs, VMIDs, job/incident IDs, ports and probe status codes only.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Messaging Integrations
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-11
|
||||
verified_on: 2026-10-11
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
confidence: high
|
||||
tags: [messaging, telegram, discord, email, integrations]
|
||||
sources: [raw/configs/hermes-config-sanitized.txt, live-jobs-json, live-himalaya-debug]
|
||||
@@ -93,3 +93,23 @@ not merely an application-level refusal.
|
||||
|
||||
**Open item:** re-probe the host; if the host is back but the adapter stays down, restart the
|
||||
gateway's Mattermost platform.
|
||||
|
||||
## 2026-10-19 00:15 AWST — Mattermost adapter: cause identified (CT255 stopped), still down
|
||||
|
||||
This pass replaced the 10-11 observation's "cause not determined" with a definite cause. Live probes,
|
||||
2026-10-19 00:0x-00:15 AWST:
|
||||
|
||||
- `tailscale status`: peer `mattermost` (`100.98.116.14`) — **offline, last seen 29d ago**, `tx 7176 rx 0`.
|
||||
- `gateway.log`: the reconnect loop is unchanged — `Reconnecting mattermost (attempt 470)` at 00:08:36
|
||||
and `(attempt 471)` at 00:14:07, each followed by `Mattermost: disconnected` and
|
||||
`Reconnect mattermost error: mattermost connect timed out after 30s, next retry in 300s`
|
||||
(5-minute cadence; 3,992 such lines in the retained `gateway.log`).
|
||||
- Probed from CT460 at the configured endpoint (`MATTERMOST_URL=http://100.98.116.14:8065/`):
|
||||
**TCP 8065 refused/closed and 100 % ICMP loss** — the host is unreachable, not the application.
|
||||
- **Cause: CT255 (`mattermost`) is `stopped` on the PVE host** (see [[infrastructure/proxmox]]).
|
||||
The attempt counter restarted at the gateway restart of 2026-10-17 05:10 and now reads 471.
|
||||
- Impact unchanged: messaging path only. No cron job delivers via Mattermost, and the cron fleet
|
||||
performed normally throughout.
|
||||
|
||||
**Open item:** start CT255, or retire the platform block. The 5-minute reconnect loop continues until
|
||||
one of those happens — restarting the adapter while the host is stopped achieves nothing.
|
||||
|
||||
@@ -3,8 +3,8 @@ title: Mnemosyne
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
confidence: high
|
||||
tags: [memory, mnemosyne, backend]
|
||||
sources: [live-probe, mnemosyne-db, config, ops-profile-config]
|
||||
@@ -49,6 +49,8 @@ Primary agent memory store for Hermes.
|
||||
|
||||
## Known Issues
|
||||
|
||||
**Tier counts + pipeline status (live, 2026-10-18 12:1x — health job `ea79a2107e36`, exited 1):** DB 27 MB; working **801**, episodic **1,065**; embeddings working **775/801 (96.8 %)**, episodic **1,059/1,065 (99.4 %)** — both above the 95 % floor, no `reindex` needed. FTS5 aligned on all three indexes (`fts_working` / `fts_episodes` / `fts_facts`); consolidation ran 2026-10-18 03:55 (**0 items** — nothing new to consolidate); **1 unresolved conflict** (benign, threshold 10); Mnemosyne provider registered, 40 tools; watchdog exit 0. The job's red was a **capture-staleness false positive**: `working_memory` had no non-cron row since **2026-10-16 11:07** simply because no user conversation had occurred — the gateway restarted 2026-10-17 05:10, Telegram reconnected 10:45, and capture was idle rather than broken. **The 24 h threshold in `~/.hermes/scripts/mnemosyne-health-check.sh:92` will keep flagging idle weekends**; expect repeat reds until it gains a "no recent conversations" guard or the threshold is raised. Same trap class as the naive-UTC clock note below.
|
||||
|
||||
**Tier counts (live, 2026-10-09 22:27):** working `773`, episodic `1,047`, `canonical_facts` `97`, `facts` `871`, `memoria_facts` `407`, `memoria_preferences` `132`, `conflicts` `152` (**1 open** — id `152`, `contradiction`, created 2026-10-09 13:11:38), working embeddings `760/773`, episodic embeddings `1,043/1,047 (99.6%)`. Ranking relevance is the live issue, not coverage — see [[incidents/2026-10-09-memory-recall-relevance]].
|
||||
|
||||
**Tier counts (live, 2026-10-12 12:02 — health job `ea79a2107e36`, green):** working **760/786 embedded (96.7%)**, episodic **1050/1056 (99.4%)** — both above the 95% floor, no `reindex` needed (`b63ed5817212` continues to backfill). Orphan embeddings **0**. FTS5 exactly aligned: `fts_working` 786/786, `fts_episodes` 1056/1056, `fts_facts` 876/876. **0 unconsolidated** working rows; **1 unresolved conflict** (threshold 10); last consolidation 0.3 h earlier; 2,252 memories total (786 working / 1,056 episodic / 1 triple); disk 15 GB free (81% used). Interpretation: coverage is *not* the live issue — ranking relevance is (see [[incidents/2026-10-09-memory-recall-relevance]]).
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: Push Notifications (Cadu / Vory bridge)
|
||||
type: system
|
||||
status: degraded
|
||||
created: 2026-10-19
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
review_after: 2026-10-26
|
||||
confidence: medium
|
||||
tags: [push, notifications, plugins, gateway, integrations]
|
||||
sources: [live-probe, live-config, live-push-dir, live-logs]
|
||||
---
|
||||
|
||||
# Push Notifications (Cadu / Vory bridge)
|
||||
|
||||
## Purpose
|
||||
|
||||
Delivers Hermes activity to Anthony's phone (APNs push) and carries remote
|
||||
answer/approval traffic back to the gateway. Two plugin layers plus a standalone bridge script.
|
||||
|
||||
## Components
|
||||
|
||||
| Piece | Where | Notes |
|
||||
|---|---|---|
|
||||
| `hermes-push` | plugin (enabled) | Cadu push layer — device delivery |
|
||||
| `cadu-device`, `cadu-mail`, `cadu-rich-cards`, `cadu-secrets-vault` | plugins (enabled) | Cadu device/mail/card/vault surfaces |
|
||||
| `vory-push` | plugin (enabled) | runs the push bridge alongside the gateway |
|
||||
| `hermes_push.py` | `~/.hermes/push/hermes_push.py` (91 KB, **v1.0.37**, sha256 `59cabfcc…`) | the bridge itself; connects to the gateway's `/api/ws` as an ordinary client |
|
||||
| `install.sh` | `~/.hermes/push/install.sh` | installer (Vory) |
|
||||
| `hermes-push.conf` | `~/.hermes/push/hermes-push.conf` | written by Vory, read by the bridge via `HERMES_PUSH_CONFIG` |
|
||||
|
||||
`hermes-push.conf` holds `HERMES_PUSH_GATEWAY_URL`, `HERMES_PUSH_PUBLIC_URL`,
|
||||
`HERMES_PUSH_GATEWAY_BEARER`, `HERMES_PUSH_GATEWAY_REFRESH_TOKEN` and `HERMES_PUSH_DEVICES_DIR`.
|
||||
**The two tokens are JWT-shaped secrets and are deliberately not recorded here.** State dirs:
|
||||
`devices/`, `goals/`, `origins/`, plus `status.json` and `relay.lock`.
|
||||
|
||||
Config (live `config.yaml`): `plugins.enabled` lists `cadu-device`, `cadu-mail`, `cadu-rich-cards`, `cadu-secrets-vault`, `hermes-push`, `vory-push` (alongside `platforms/mattermost`, `homeassistant`, `hermes-browser-stream`, `a2a-platform`); under `platforms:`, `push.gateway_restart_notification: false` and `relay.enabled: true`.
|
||||
|
||||
## Status file — how to read it
|
||||
|
||||
`~/.hermes/push/status.json` is rewritten every retry cycle. Fields seen live:
|
||||
`version`, `script_sha256`, `pid`, `updated_at`, `profile`, `connected`, `gateway`, `transport`, `error`.
|
||||
|
||||
## Current state (live, 2026-10-19 00:18 AWST) — BROKEN, and it has been for a while
|
||||
|
||||
```json
|
||||
{"version": "1.0.37", "pid": 2496376, "profile": "default", "connected": false,
|
||||
"gateway": "https://hermes.kangaroo-eel.ts.net", "transport": "",
|
||||
"error": "https://hermes.kangaroo-eel.ts.net: connection refused (nothing listens there on this machine)"}
|
||||
```
|
||||
|
||||
- **Cause (verified):** `hermes.kangaroo-eel.ts.net` resolves **on this host to `127.0.1.1`**
|
||||
(`getent hosts` → `127.0.1.1 hermes.kangaroo-eel.ts.net hermes`, from the host's `/etc/hosts`
|
||||
convention), so the bridge dials loopback port 443 — where nothing listens. `/etc/hosts` wins over
|
||||
the tailnet DNS name.
|
||||
- **The gateway and the Serve route are both healthy** — this is a resolution/dial-target problem,
|
||||
not a gateway outage: the gateway listens on `0.0.0.0:9119` (`curl http://127.0.0.1:9119/` → **302**),
|
||||
`tailscale serve status` still shows `https://hermes.kangaroo-eel.ts.net (tailnet only)` with
|
||||
`/ → http://127.0.0.1:9119`, and the public name is served on Funnel `:8443`.
|
||||
- **Chronic, not a blip:** the `connection refused (nothing listens there on this machine)` string
|
||||
appears **23,469** times across the retained log set (`agent.log`, `agent.log.1-3`,
|
||||
`errors.log`, `errors.log.1-2`). `relay.lock` was last touched 2026-10-19 00:03.
|
||||
- **Impact:** phone push delivery and any remote answer/approve path through the bridge are down.
|
||||
Nothing else observably depends on it — no cron job's delivery uses it.
|
||||
|
||||
### Fix options (diagnosed, NOT applied)
|
||||
|
||||
1. Point `HERMES_PUSH_GATEWAY_URL` at an address that actually answers from this host —
|
||||
`http://127.0.0.1:9119` (the gateway's own listener) or the tailnet IP
|
||||
(`100.118.5.51:443`, where `tailscaled` serves the name) — instead of the FQDN that resolves to
|
||||
`127.0.1.1`.
|
||||
2. Or give the FQDN a resolvable answer on this host: remove/qualify the `/etc/hosts` line that maps
|
||||
`hermes`/`hermes.kangaroo-eel.ts.net` to `127.0.1.1`, letting tailnet DNS resolve it.
|
||||
3. Either way, confirm WSS/WS scheme matches what the bridge expects before changing the value.
|
||||
|
||||
**Not yet traced in this pass:** which process launches the bridge and under which unit/service, and
|
||||
whether `HERMES_PUSH_PUBLIC_URL` also embeds the broken FQDN.
|
||||
|
||||
## Relay contention (benign, recurring)
|
||||
|
||||
`relay.lock` serialises bridge instances. The log line
|
||||
|
||||
```
|
||||
vory-push relay running in another Hermes process; waiting for it to stop
|
||||
```
|
||||
|
||||
appears when a second process contends for the relay (last recorded 2026-10-15; `relay.lock`
|
||||
touched 2026-10-19 00:03). It resolves without intervention.
|
||||
|
||||
## Related
|
||||
- [[systems/messaging-integrations]]
|
||||
- [[infrastructure/tailscale]]
|
||||
- [[systems/hermes-agent]]
|
||||
- [[current-state]]
|
||||
|
||||
No secrets recorded; key names, hashes, ports, status codes and paths only.
|
||||
@@ -3,8 +3,8 @@ title: Scheduled Tasks
|
||||
type: system
|
||||
status: active
|
||||
created: 2026-07-22
|
||||
updated: 2026-10-18
|
||||
verified_on: 2026-10-18
|
||||
updated: 2026-10-19
|
||||
verified_on: 2026-10-19
|
||||
review_after: 2026-10-25
|
||||
confidence: high
|
||||
tags: [cron, scheduled, automation]
|
||||
@@ -14,10 +14,11 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
# Scheduled Tasks
|
||||
|
||||
## Count
|
||||
- **✅ Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **✅ Current as of 2026-10-19 00:20 AWST (`jobs.json.updated_at 2026-10-19T00:17:06+08:00`) — one failure in 24 h, and it was a failure mode this wiki had not recorded before.** Fleet **28 jobs — 25 enabled / 3 disabled** (unchanged). Window 2026-10-18: **168 `completed`, 1 `failed`** — the failure is `ecc1fc13655a` (FreshRSS Midday, 12:01:10 → 12:08:25), which **wrote and delivered a canned “Your request was not processed…” sentence to Telegram** (`delivery_outcome: delivered` at 12:08:11) before dying on `RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding the degenerate response.` → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. **Non-clean `last_status`: 8 → 7** — `1a8e90ea2271` (10-18 03:01) and `cbfe434a3b5e` (this audit, 10-18 00:42) cleared on schedule, leaving `1b23c042e17d` (`error`, streak **2**, next **Fri 10-23 08:00**), `0e519874d280` (**Mon 10-19 08:00** — next to fire), `c8d52904e1d6` + `06a0a1c8704f` (Thu 10-22), `ecc1fc13655a` (**10-19 12:00**), and the monthlies `0e753e9720fd` + `066bb5191280` (both 11-01). **Open incidents 8 → 7** (6 `alerted` / 1 `detected`); **1 opened, 2 resolved** in-window. Tallies: 10-17 = 163 / 5, **10-18 = 168 / 1**, 10-19 so far 5 / 0 (+ this audit `running`); **11 sessions in the window, all `cron` — no interactive sessions.** → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
- **Current as of 2026-10-18 00:31 AWST (`jobs.json.updated_at 2026-10-18T00:31:17+08:00`) — the `omni-101` 503 burst is over and the fleet is unwinding normally.** Fleet **28 jobs — 25 enabled / 3 disabled**. **8 carry a non-clean `last_status`** (down from 12 at the 10-16 storm peak): `1b23c042e17d` (`error`, streak **2**), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`). Four rows cleared as their slots came round: `b63ed5817212` (10-17 06:01), `08eb86f02d37` (10-17 08:31), `ecc1fc13655a` and `ea79a2107e36` (both 10-17 12:02). **Open incidents 12 → 8** (5 `alerted` / 3 `detected`); **9 incidents opened and 6 closed inside this window**. Daily tallies: **10-16 = 165 completed / 8 failed**, **10-17 = 162 / 5** (all five before 03:01) — against 10-15 = 152 / 24 (storm tail) and 10-12 = 175 / 0. New failure mode in the same topology: `RuntimeError: HTTP 503: Service temporarily unavailable due to resource pressure.` on `custom:omni-101`, 10-16 08:00 → 10-17 03:00. → [[incidents/2026-10-16-omniroute-503-resource-pressure]]
|
||||
- **⚠ Current as of 2026-10-16 00:1x AWST — the fleet is in storm recovery.** **12 non-clean `last_status`** (was 3 at the 10-13 audit): **9 new rows from the `omni-101` outage** — `08eb86f02d37` (streak 2), `ecc1fc13655a` (2), `dca8482e4f76`, `c8d52904e1d6`, `06a0a1c8704f`, `85edf6772bca`, `0e519874d280`, `1a8e90ea2271`, `cbfe434a3b5e` (all streak 1) — plus the three long-standing rows `1b23c042e17d` (since 10-09), `0e753e9720fd` (10-01) and `066bb5191280` (`delivery_failed`, 10-01). **Three jobs already cleared inside this window:** `b63ed5817212` (10-16 00:02), `1b31187397a5` (10-15 22:01), `ea79a2107e36` (10-16 00:09). **Structural finding: all 18 enabled model-calling jobs are pinned to `custom:omni-101`** — the cron fleet has **no model-call failover**. → [[incidents/2026-10-14-omniroute-node-connection-refused-storm]]
|
||||
- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-18 00:31 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs.
|
||||
- **Non-clean `last_status`:** **8 as of 2026-10-18 00:31** — `1b23c042e17d` (`error`, streak 2), `cbfe434a3b5e` (1), `1a8e90ea2271` (1), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`, streak 0). History: **3** at the 2026-10-12 audit (when `fad5a86979d4` and `dca8482e4f76` cleared on schedule) → **12** at the 2026-10-16 storm peak → 8 now. The six jobs that fixed themselves did so on their next scheduled slot; the remaining weekly ones (`1b23c042e17d` Fri, `0e519874d280` Mon, `06a0a1c8704f` + `c8d52904e1d6` Thu) cannot clear before their next run.
|
||||
- **Total jobs:** **28 — 25 enabled, 3 disabled** (verified from live `~/.hermes/cron/jobs.json` at 2026-10-19 00:20 AWST). The count fell 30 → 29 on **2026-10-10** (redundant DHL monitor `62619aa6bcb3` retired) and 29 → 28 when **Mosquito Tracker `1b31187397a5` was removed** — its last execution was 2026-10-16 15:00 (`completed`, delivered) and it is absent from `jobs.json` as of this audit. See Removed Jobs.
|
||||
- **Non-clean `last_status`:** **7 as of 2026-10-19 00:20** — `1b23c042e17d` (`error`, streak 2), `0e519874d280` (1), `06a0a1c8704f` (1), `c8d52904e1d6` (1), `ecc1fc13655a` (`error`, streak 1 — degenerate stream, 10-18 12:08), `0e753e9720fd` (`error`, streak 0), `066bb5191280` (`delivery_failed`, streak 0). `cbfe434a3b5e` and `1a8e90ea2271` cleared on schedule (10-18 00:42:54 and 03:01:22). History: **3** at the 2026-10-12 audit (when `fad5a86979d4` and `dca8482e4f76` cleared on schedule) → **12** at the 2026-10-16 storm peak → 8 at the 2026-10-18 audit → **7 now**. The six jobs that fixed themselves did so on their next scheduled slot; the remaining weekly ones (`1b23c042e17d` Fri, `0e519874d280` Mon, `06a0a1c8704f` + `c8d52904e1d6` Thu) cannot clear before their next run.
|
||||
- **Failing (last run failed outright):** **0** — down from 7. All seven broken jobs were repinned on 2026-10-05 23:16–23:45 AWST to provider `custom:omni-101` using routing aliases, and every `failure_streak` in `jobs.json` now reads **0**. Four of the seven jobs had still carried a *pre-repin* `last_status: error` (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`, `1a8e90ea2271`) — a stale field, not a live failure; it clears on each job's next scheduled run. **`1a8e90ea2271` cleared on 2026-10-06 03:00**, running on schedule for the first time since 2026-09-25; the remaining three (`1b23c042e17d`, `fad5a86979d4`, `0e753e9720fd`) are due Fri 08:00 / Mon 03:00 / 1st 09:00. **`fad5a86979d4` cleared on 2026-10-12 03:00**, leaving two (`1b23c042e17d`, `0e753e9720fd`). See [[incidents/2026-10-05-dead-model-pin-repin]].
|
||||
- Historical (2026-10-01 → 2026-10-05), now all remediated: `dca8482e4f76` 403 `MODEL_NOT_IN_PLAN` ×7; `1a8e90ea2271` HTTP 410 ×11 (retired `qwen3.5:397b`); `fad5a86979d4` HTTP 400 ×3; `0e753e9720fd` HTTP 400 ×1; `06a0a1c8704f`, `1b23c042e17d` and `79f9bb41ecf5` HTTP 401 (the `cliprox/terra` connection auth expiry, settled 3/3 on 2026-10-04).
|
||||
- **Failing (delivery only):** 1 — `066bb5191280`: the report itself generated; the send failed against the DM thread (bare `telegram` target, 2026-10-01). Unchanged this window.
|
||||
@@ -45,7 +46,7 @@ sources: [live-cron-list, live-jobs-json, cron-executions-db]
|
||||
| Evening Stand-Down Briefing | `e530c89f2be0` | `45 18 * * 1-5` | telegram:-1004321904721 | — | `evening_standdown.py` | no-agent |
|
||||
| Facebook Birthday Monthly Scrape | `0e753e9720fd` | `0 9 1 * *` | telegram | — | — | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×1). ⚠ Not yet re-run — its incident `0e753e_e7f8194a7d06` (opened 2026-10-01 09:00:55) is **still open** in state `detected`, and `last_status` still reads `error`; **next slot 2026-11-01 09:00**, which is when the repin finally gets its live test. ⚠ Bare `telegram` still targets the home DM — retarget to a group topic |
|
||||
| Facebook Birthday Weekly Registry Sync | `fad5a86979d4` | `0 3 * * 1` | telegram | — | `birthday_check.py` | ✅ **Repinned 2026-10-05** → `custom:omni-101` / `free-stack` (was the catalog-missing `nvidia/deepseek-ai/deepseek-v4-flash` pin, HTTP 400 ×3). ✅ **Verified live 2026-10-12 03:00** — completed and delivered (448 birthdays); the stale `last_status: error` cleared after two weeks and **both** `deepseek-v4-flash` incidents auto-closed at 03:05:39 (`fad5a8_a13bbc837e1c`, `fad5a8_c80b59a907f0`). Live `last_status: ok`, streak 0. ⚠ Bare `telegram` target should move to a group topic |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed the incident. Live `last_status: ok` |
|
||||
| FreshRSS Midday News Briefing | `ecc1fc13655a` | `0 12 * * *` | telegram:-1004321904721:346 | — | `freshrss_intelligence_v2.py` | ⚠ The **2026-10-16 12:00** slot failed on the `omni-101` 503; cleared by its own 10-17 12:02 run (delivered), which auto-closed that incident. ⚠ **Then 2026-10-18 12:00 failed differently** — a model repetition loop aborted the run at 12:08:25 **after** it had delivered a canned "Your request was not processed…" line to the topic (12:08:11) → [[incidents/2026-10-18-degenerate-stream-canned-substitute]]. Live `last_status: error`, `failure_streak: 1`, next slot **2026-10-19 12:00** |
|
||||
| GitHub Trending Radar — Weekly Roundup | `c0a9fa314542` | `0 9 * * 1` | telegram:1793951355 | — | — | ⚠ **Delivery hop broken — the script's bot token is revoked (verified live 2026-10-13).** A `getMe` call against the `telegram.bot_token` in `/home/hermes/workspace/github-trending-radar/config.json` returns **HTTP 401 Unauthorized**. The 2026-10-12 09:00 run completed and fetched trending repos fine, then died at `digest.py`'s `send()`. The agent recovered the content with `--dry-run` and its own response was delivered, so the digest still reaches the DM — but the script's own send fails until a new **@BotFather** token replaces the dead one. `last_status: ok` because the 401 is inside the script, not the cron run. **This supersedes the 2026-10-05 "token-revocation claim refuted" note — that conclusion was wrong.** |
|
||||
| Gmail Action Digest | `08eb86f02d37` | `30 8 * * *` | telegram:-1004321904721:notifications | — | `gmail_action_digest_v3.py` | ⚠ The **2026-10-16 08:30** slot failed on the `omni-101` 503 (streak 2 at the storm peak); cleared by the 10-17 08:31 run, which auto-closed the incident. Live `last_status: ok` |
|
||||
| Important-mail monitor | `d40085da631c` | `every 30m` | telegram:1793951355 | — | `_cron_silent_monitor.py` | **disabled** since 2026-08-29 (Gmail read-state incident) |
|
||||
|
||||
Reference in New Issue
Block a user