wiki: 2026-11-14 audit corrections — withdraw the 'sign not magnitude' / '12 h of lead time' / '18:03 run' claims; verified escalation mechanism (activity-aware magnitude gate) + 29-cycle zero-item consolidation streak

This commit is contained in:
Tony0410
2026-11-15 01:28:53 +08:00
parent 2d439145bf
commit 82876abdfb
5 changed files with 134 additions and 31 deletions
+47 -12
View File
@@ -46,25 +46,60 @@ has no working_memory row (last capture 2026-11-14 01:32:59)
- `cron_incidents` row `ea79a2_601037516abd`: `state: alerted`, `failure_type: auth`,
`first_seen_at = last_seen_at = 2026-11-15T00:03:35.815373+08:00`, `closed_at: null`.
Output file: `~/.hermes/cron/output/ea79a2107e36/2026-11-15_00-03-35.md`.
- ⚠ **The `auth` label looks wrong** — the failure is a *staleness* check, not a credential fault, and
the page records no auth error text. **Observation, not verified diagnosis.**
- ⚠ **The `auth` label describes the classifier, not the fault — now evidenced, not just suspected.** The
failure is a *staleness* check and the page records no auth error text. This job's **twelve earlier
`cron_incidents` rows** read `agent` ×3, `timeout` ×1, `unknown` ×8 (`ea79a2_649d8ac46e8a`,
`_f17b45759db0`, `_573d17e10fc1`, `_6fb70b032305`, `_54d6b1f0f58a`, `_747b27556fd0`, `_c3ab2d784ba3`,
`_a4661554a3c2`, `_d93c3acdc4f2`, `_9197df32320b`, `_111e67a3d83c`, `_f317fcd843ae`), and the estate-wide
table is **53 `unknown` / 47 `agent` / 15 `timeout` / 14 `script` / 5 `auth`**. `agent` is also impossible
here: the job runs `no_agent` (plain script). Treat the field as opaque routing metadata.
- ⚠ **The alert text prints the raw stored value (`01:32:59`, UTC) with no timezone**, which reads as a
~23 h gap when the true AWST gap was ~14.5 h. Same class of clock confusion as the monitor's own
log line below.
## The earlier, quieter reading — 12:00:52 AWST 2026-11-14
## What the earlier slots actually saw — correction, re-derived 2026-11-15 01:2x AWST
`~/.hermes/logs/mnemosyne-health-monitor.log` shows the **12:00:52 AWST** run already looking at the
same hole and calling it benign:
**An earlier revision of this page got this backwards.** It claimed the **12:00:52 AWST** run had
"already seen the same hole and called it benign" (`capture delay -589.5 min`), giving the estate ~12 h
of unalarmed lead time, and blamed a "sign, not magnitude" threshold. **Both claims are withdrawn.** The
`-589.5 min` line is not the 12:00 run's — it sits inside the **00:03:35 run's own output block**.
```
ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min
```
Verified by line number in `~/.hermes/logs/mnemosyne-health-monitor.log` (**1103 lines**):
That is **9 h 49.5 m** of missing capture, logged as ℹ️ informational (the delay is **negative**, i.e.
capture sits *behind* the message clock — the sign convention makes a growing gap look small). The
job exited 0 at 12:00 and 18:03; it only escalated at 00:03 the next morning.
**So the fault had ~12 hours of lead time that produced no alarm.**
| line | content |
| --- | --- |
| 884 | `=== 2026-11-14 00:09:12 AWST health monitor (no_agent) ===` |
| 900 | `ℹ️ last user msg 2026-11-13 11:21 UTC (desktop) capture delay 75.2 min` |
| 957 | `=== 2026-11-14 12:01:52 AWST health monitor (no_agent) ===` |
| 973 | `ℹ️ last user msg 2026-11-14 01:03 UTC (ios) capture delay 29.4 min` |
| 1030 | `=== 2026-11-15 00:03:35 AWST health monitor (no_agent) ===` |
| 1046 | `ℹ️ last user msg 2026-11-14 11:22 UTC (desktop) capture delay -589.5 min` |
| 1047 | `❌ STALE MEMORY PIPELINE: …` |
Two further corrections to the earlier revision:
- **There was no 18:03 run.** The job's schedule is `0 */12 * * *`; the only 2026-11-14 slots are
00:09:12 and 12:01:52 (both exit 0, delivery `suppressed`). The "exited 0 at 12:00 and 18:03"
sentence is withdrawn — 18:03 belongs to some other job.
- **The detection chain was not broken.** `mnemosyne_freshness_gate.py` fires on **magnitude**
(`delay_min < -CAPTURE_LAG_MIN`, `MNEMOSYNE_CAPTURE_LAG_MIN` default **120** min) — there is no
sign-only test to blame. The 12:00:52 measurement of **+29.4 min** is the delay of the newest message
*that had been captured* (the 09:03 AWST `ios` turn, captured 09:32:59). The gate computes against the
newest **uncaptured** message and only reports a problem when one exists; with the conversation idle
from 09:03 to 19:22, every user message still had a capture row, so there was **nothing to detect**.
The first message to postdate the stall was the **19:22 AWST desktop** turn (`11:22 UTC`, the same
message the 00:03 alert names); the next scheduled slot after it was 00:03:35, which escalated
(`-589.5 min` = 09:32:59 − 19:22).
**Corrected statement of the gap:** the pipeline died at **09:32:59 AWST**; it became *detectable* at
**19:22 AWST** (first uncaptured message) and was *detected* at **00:03:35** by the next slot — a
detection latency of 4 h 41 m after detectability, driven by cadence, not by a broken threshold.
**What was genuinely unread sits in the consolidation counters:** the cycles ran every ~31 min and
returned **0 items for 29 consecutive runs** (streak opening `2026-11-14T02:04:42+00:00` = **10:04:42
AWST**; last productive cycles 4622/4621/4620 at `09:33:34` / `09:33:16` / `09:01:40`). Every run
printed `Last consolidation: … (0 items)` — including the "healthy" 12:00:52 one — but the gate tests
consolidation **age** only (`> 36 h`, default), never the item count. **An empty-but-current cycle is
the loudest signal available and the only one nothing watches.**
## Impact