Files
agent-estate-wiki/runbooks/troubleshoot-tool-loop.md
T
Hermes 341dda95dd stocktake 2026-11-11: 36 stale pages re-verified (0 stale); fallback chain rewritten 4/6 slots; CT333 'clawtest' naming claim corrected; nous 404 credits signature replaces refresh-token 401; estate shrank (VM 310 + CT601 gone)
- freshness: 36 stale / 93 fresh / 3 no-date -> 0 stale / 129 fresh / 3 no-date of 132 (same-day report regenerated after the corrections, per pattern 13)
- skills: 299 skills / 67 categories (count unchanged; generator arithmetic re-verified by hand against the live tree: 300 SKILL.md files - 1 nested duplicate = 299)
- runbooks/test-openrouter: live fallback chain rewritten (primary back to smart-route/custom:omni-101; mimo-v2.5, stepfun, qwen3.5/ollama-cloud out; guacano + kilo-reflection-beam-free in); ollama.com dropped from the URL test list; guacano/litellm noted as internal + key-gated
- comparisons/hermes-vs-nanobot: the 2026-10-20 'CT333 is zeroclaw in Proxmox naming' claim was itself wrong - pct list and hostname both return clawtest
- incidents/2026-10-22-nous: refresh-token failure gone from logs; new signature 404 insufficient_credits_for_paid_model from auxiliary.pulse_memory (ninth slot, back on nous, paid model) - open item revised
- incidents/2026-08-29-gmail: email crons now ok but root cause unfixed 74 days (himalaya message read still in fetch_newsletters.py; goimapnotify back up PID 441; monitor still paused)
- 30 further pages re-verified live and de-flagged with dated probe notes (ct207 ingress, CT205 guanaco v0.8.15, CT270 nextcloud 33.0.2.2, headless stack, hermes-ui both hosts, systemd units + cgroup limits, terminal/tool_search/mnemosyne/custom_providers config blocks, ops profile still deleted, hosts/pve guest lists)
- log.md addendum + index.md date bump; no page deleted, no secrets written
2026-11-11 03:52:48 +08:00

108 lines
4.6 KiB
Markdown

---
title: Troubleshoot Tool Loop
type: runbook
status: unresolved
created: 2026-07-22
updated: 2026-11-11
review_after: 2026-11-19
verified_on: 2026-11-11
confidence: medium
tags: [runbook, hermes]
sources: []
---
> **Filled in 2026-10-20** from live configuration, live process state and two recorded incidents.
> The "stub" note below is retained as history.
# Troubleshoot Tool Loop
## Symptoms
- The agent repeats the **same tool call** with identical arguments and makes no progress.
- The agent repeats **itself in the response stream** (no tool involved) — the model loops on its
own output and Hermes cuts the stream.
**These are two different faults with two different detectors.** Do not assume one covers the other.
## 1. Tool-call loop — detected by `tool_loop_guardrails`
### Symptoms
Agent loops on one tool or endpoint. Hermes emits a warning naming the tool.
### Configuration (verified live 2026-10-20, `~/.hermes/config.yaml`)
```yaml
tool_loop_guardrails:
warnings_enabled: true
hard_stop_enabled: false
warn_after:
exact_failure: 2
same_tool_failure: 3
idempotent_no_progress: 2
hard_stop_after:
exact_failure: 5
same_tool_failure: 8
idempotent_no_progress: 5
```
> **`hard_stop_enabled: false` is the key fact.** Hermes will *warn* about a stuck tool loop but
> will **not** force-stop itself. A genuinely stuck loop only gets a warning — a human (or the
> messaging integration) is the real backstop. See [[concepts/tool-calling]].
### Procedure
1. Inspect recent Hermes agent logs: `~/.hermes/logs/agent.log*` (grep for the tool name).
2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead
Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns
200 with an empty body.
3. **Check the dependency, not just the agent.** On this estate the most frequent tool-loop cause
was an upstream service being down while the tool still returned a response. Verify with the
relevant runbook: [[runbooks/restart-browser]], [[runbooks/restart-hermes]],
[[runbooks/provider-health]].
4. If the loop is a stuck worker, restart it: `systemctl --user restart hermes-gateway.service`.
5. Re-run a limited test scenario and confirm the guard no longer fires.
### Verification
Agent handles the same scenario without repeating, and no `tool_loop_guardrails` warning appears
in `agent.log`.
## 2. Model-output (degenerate stream) loop — detected by the stream guard
### Symptoms
The run **delivers a canned non-answer** such as *"Your request was not processed. Send it again
if you still want me to carry it out."*, and then fails with
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding
the degenerate response.`
### This is the 2026-10-18 case — see
[[incidents/2026-10-18-degenerate-stream-canned-substitute]]
### Procedure
1. Read the run's `error` in `~/.hermes/cron/executions.db` — the phrase `repetition loop`
identifies this fault.
2. **Check `delivery_outcome` on the same row.** In the 2026-10-18 incident the row was
`delivery_outcome: delivered` — the canned non-answer reached Telegram *before* the run
failed. **A delivered message is not evidence of delivered content.**
3. Confirm the provider is healthy: the 2026-10-18 case rode the same `custom:omni-101` alias
that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19
resource-pressure 503 bursts. See [[runbooks/provider-health]] and
[[incidents/2026-10-16-omniroute-503-resource-pressure]].
4. Do **not** lower `tool_loop_guardrails` thresholds for this fault — they watch tool calls, and
the degenerate-stream guard is a separate mechanism.
### Verification
The next scheduled run of the job completes with `status: completed` and `error: null`, and the
delivered message contains real content.
## Last successfully used
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
> **Guardrail config re-read 2026-11-11 — byte-for-byte unchanged.**
> `tool_loop_guardrails`: `warnings_enabled: true`, `hard_stop_enabled: false`,
> `warn_after {exact_failure: 2, same_tool_failure: 3, idempotent_no_progress: 2}`,
> `hard_stop_after {exact_failure: 5, same_tool_failure: 8, idempotent_no_progress: 5}`.
> `hard_stop_enabled: false` therefore still means a stuck tool loop only ever *warns*.
## Related
- [[concepts/tool-calling]]
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
- [[runbooks/provider-health]]
- [[runbooks/restart-hermes]]