Files
agent-estate-wiki/incidents/2026-08-04-omniroute-free-deepseek-silent-stops.md
T
Hermes ab3652aa20 Stocktake 2026-11-12: tailscale live-vs-page drift corrected; sftpgo incident re-verified; pre-dispatch-filter signature found recurring on a second job/pin
- infrastructure/tailscale.md: node table rebuilt from live tailscale status --json.
  mailflow/qwenpaw/springroll read Online:false in the peer record but all three ping
  (CT1262 + CT465 running; springroll via DERP(sin)) - stale peer records, rows stay online.
  omarchymac genuinely stale in the other direction (offline 2d/11-08 -> offline 4d, ping timeout).
  9 named offline ages + 6 codex ephemeral peers refreshed. New guest CT1007 homebridge
  (running, tailscaled inactive). DEAD Serve targets 8769/9123 re-confirmed dead; CT201 table
  unchanged; container counts 17 LXC+4 VM (pve) / 20 LXC+2 VM (m93p) confirmed.
- incidents/2026-09-02-sftpgo-outage.md: 22d stale, re-probed live (pct list running, peer
  Online:true, pong, healthz 401; swap now 512 not 43Gi). Cleared.
- incidents/2026-10-21-omniroute-pre-dispatch-filter-skip.md: signature recurred 2026-11-10 on
  cbfe434a3b5e (custom:omni-101/free-stack, HTTP 429 credits 100%); incident row resolved
  2026-11-12T00:25. Original subject clean 4 slots. Retention-floor caveat recorded.
- Threshold re-verifications (19d/18d): mnemosyne-relay (tools:None gone, 0 log hits),
  work-outlook (himalaya still login not supported), omniroute-free-deepseek
  (good-free/free-stack both HTTP 200, non-deepseek resolution).
- Correction recorded: previous entry mis-described the 2026-11-12 stale page as log.md.
- wiki-maintain.sh re-run after all edits: freshness 0 stale / 130 fresh / 3 no-date of 133.
2026-11-12 03:19:12 +08:00

4.6 KiB
Raw Blame History

title, type, status, created, updated, verified_on, confidence, tags, sources
title type status created updated verified_on confidence tags sources
OmniRoute free-DeepSeek Silent Stops (Empty-After-Tool) — 2026-08-04 incident monitoring 2026-08-04 2026-11-12 2026-11-12 high
omniroute
deepseek
free-tier
silent-failure
incident
mitigated
live-probe
omniroute-log
session-20260804

[2026-08-04] OmniRoute free-DeepSeek silent stops (empty-after-tool)

Status

Mitigated — combos reworked; upstream still degraded.

Symptom

On both Hermes instances (ops/Tusk + the other instance), turns using deepseek-flash-free (and smart-route via good-free) did one tool call, then went silently — no text, no error. "continue" → one more tool call → silent stop again.

Root cause

The free-DeepSeek upstreams behind the combos started returning valid-but-empty HTTP 200s right after a tool_calls turn (no content, no further tool call). Hermes can't distinguish "empty" from "model chose to stop" → ends turn silently. Began 2026-08-01 ~09:14 UTC, still active. OmniRoute log markers:

[STREAM] Empty assistant response after tool_calls completion (<provider>:<model>)
[ProxyEgress] <provider>/<conn-hash> ... status=error

Per-model evidence (counts in storage.sqlite era log):

  • 238d5978/deepseek-v4-flash — 63× empty-after-tool (conn d4b9fee0, 189 proxy errors)
  • 238d5978/deepseek-v4-flash-0731 — 70× empty-after-tool
  • eb5af0da/deepseek-v4-flash-thinking:free (conn f8169018) — 504 proxy errors (combo #1 slot)
  • nvidia deepseek + opencode-zen deepseek-free — noisy, errors

Strategy trap: both combos used priority, which retries the same broken connection and never fails over on an empty 200.

Change applied (2026-08-04, ~03:30 AWST)

Edited combos in /var/lib/omniroute/storage.sqlite on CT600 (live DB, hot-reloads — no restart):

  • deepseek-flash-free — strategy priority → fill-first; 8 → 7 models: healthy free first (mimo-v2.5-free, laguna-s-2.1:free ×2 providers, step-3.7-flash, glm-5.2-cheap:free), one deepseek (deepseek-v4-flash-thinking:free) demoted to last resort.
  • good-free — strategy priority → fill-first; 15 → 12 models. Dropped the proven culprits: 238d5978/deepseek-v4-flash, nvidia/deepseek-ai/deepseek-v4-pro, nvidia/deepseek-ai/deepseek-v4-flash. opencode/deepseek-v4-flash-free kept as last resort.

Backup of both pre-change rows: /tmp/combo_backup_20260804.json on the Hermes gateway LXC.

Verification

  • Live API (/v1/combos) reflects both changes immediately (hot reload).
  • 2-step tool-call round-trip through deepseek-flash-free returns proper final content.
  • Direct round-trips (stream + non-stream) through the OLD combo were also healthy — the failure needs the degraded upstream + real session, hence counts not repro.

Follow-ups

  • Upstream free-DeepSeek tier still degraded; re-check counts in a few days and restore deepseek-first ordering if it recovers.
  • Consider auto+explorationRate for self-tuning if free tier stays flaky.
  • Hermes-side: empty-post-tool turns are nudge/retry/fallback handled, but still end silently once exhausted — possible future improvement: surface "(empty after tool calls)" more visibly.

Re-verified 2026-11-12 03:4x AWST (stocktake — page was 19d old, inside the staleness window)

Both reworked combos probed live on omniroute-1 (100.78.63.20:20128, custom:omni-101) with the correct key_env (HERMES_CUSTOM_100_78_63_20_20128_API_KEY) and the exact aliases used:

Probe Result
good-free (the 12-model rework) HTTP 200, resolving to glm-5.2-cheap:free with a valid non-empty completion
free-stack HTTP 200, resolving to stepfun-ai/step-5-preview with a valid completion

The mitigation is holding, and the 2026-08-04 follow-up can be partially closed: both combos answer with real content, and neither resolves to a deepseek model — which is exactly the intent of dropping the proven culprits from good-free. ⚠ What is not re-verified: I did not individually probe the free-DeepSeek upstreams (238d5978/deepseek-v4-flash, nvidia/deepseek-ai/deepseek-v4-pro, deepseek-v4-flash-thinking:free), so the original claim "upstream free-DeepSeek tier still degraded" stands neither confirmed nor refuted — the combos serve healthy non-deepseek free models, which may mean the upstream recovered or simply that it is no longer reached. Restoring deepseek-first ordering is therefore still not justified and should wait on a direct probe of those three upstreams.

Refs