wiki: publish pending edits (2026-10-20)
This commit is contained in:
@@ -3,24 +3,99 @@ title: Troubleshoot Tool Loop
|
||||
type: runbook
|
||||
status: unresolved
|
||||
created: 2026-07-22
|
||||
updated: 2026-07-22
|
||||
review_after: 2026-08-22
|
||||
confidence: low
|
||||
updated: 2026-10-20
|
||||
review_after: 2026-11-19
|
||||
verified_on: 2026-10-20
|
||||
confidence: medium
|
||||
tags: [runbook, hermes]
|
||||
sources: []
|
||||
---
|
||||
|
||||
> **Stub — structure only.** Headings exist but content has not been verified/written yet. Flagged during 2026-07-22 wiki audit; fill in with verified facts or mark deprecated if no longer relevant.
|
||||
> **Filled in 2026-10-20** from live configuration, live process state and two recorded incidents.
|
||||
> The "stub" note below is retained as history.
|
||||
|
||||
# Troubleshoot Tool Loop
|
||||
|
||||
## Symptoms
|
||||
Agent loops on one tool or endpoint.
|
||||
## Procedure
|
||||
1. Inspect recent Hermes agent logs.
|
||||
2. Identify the looping tool path.
|
||||
3. Restart worker or flush stuck request.
|
||||
4. Re-run limited test scenario.
|
||||
## Verification
|
||||
Agent handles the same scenario without repeating.
|
||||
- The agent repeats the **same tool call** with identical arguments and makes no progress.
|
||||
- The agent repeats **itself in the response stream** (no tool involved) — the model loops on its
|
||||
own output and Hermes cuts the stream.
|
||||
|
||||
**These are two different faults with two different detectors.** Do not assume one covers the other.
|
||||
|
||||
## 1. Tool-call loop — detected by `tool_loop_guardrails`
|
||||
|
||||
### Symptoms
|
||||
Agent loops on one tool or endpoint. Hermes emits a warning naming the tool.
|
||||
|
||||
### Configuration (verified live 2026-10-20, `~/.hermes/config.yaml`)
|
||||
```yaml
|
||||
tool_loop_guardrails:
|
||||
warnings_enabled: true
|
||||
hard_stop_enabled: false
|
||||
warn_after:
|
||||
exact_failure: 2
|
||||
same_tool_failure: 3
|
||||
idempotent_no_progress: 2
|
||||
hard_stop_after:
|
||||
exact_failure: 5
|
||||
same_tool_failure: 8
|
||||
idempotent_no_progress: 5
|
||||
```
|
||||
> **`hard_stop_enabled: false` is the key fact.** Hermes will *warn* about a stuck tool loop but
|
||||
> will **not** force-stop itself. A genuinely stuck loop only gets a warning — a human (or the
|
||||
> messaging integration) is the real backstop. See [[concepts/tool-calling]].
|
||||
|
||||
### Procedure
|
||||
1. Inspect recent Hermes agent logs: `~/.hermes/logs/agent.log*` (grep for the tool name).
|
||||
2. Identify the looping tool path. Common culprits in this estate: a browser tool against a dead
|
||||
Camofox service, a terminal tool against a stale persistent shell, or a webhook that returns
|
||||
200 with an empty body.
|
||||
3. **Check the dependency, not just the agent.** On this estate the most frequent tool-loop cause
|
||||
was an upstream service being down while the tool still returned a response. Verify with the
|
||||
relevant runbook: [[runbooks/restart-browser]], [[runbooks/restart-hermes]],
|
||||
[[runbooks/provider-health]].
|
||||
4. If the loop is a stuck worker, restart it: `systemctl --user restart hermes-gateway.service`.
|
||||
5. Re-run a limited test scenario and confirm the guard no longer fires.
|
||||
|
||||
### Verification
|
||||
Agent handles the same scenario without repeating, and no `tool_loop_guardrails` warning appears
|
||||
in `agent.log`.
|
||||
|
||||
## 2. Model-output (degenerate stream) loop — detected by the stream guard
|
||||
|
||||
### Symptoms
|
||||
The run **delivers a canned non-answer** such as *"Your request was not processed. Send it again
|
||||
if you still want me to carry it out."*, and then fails with
|
||||
`RuntimeError: Model output entered a repetition loop and the stream was cut mid-loop; discarding
|
||||
the degenerate response.`
|
||||
|
||||
### This is the 2026-10-18 case — see
|
||||
[[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
|
||||
### Procedure
|
||||
1. Read the run's `error` in `~/.hermes/cron/executions.db` — the phrase `repetition loop`
|
||||
identifies this fault.
|
||||
2. **Check `delivery_outcome` on the same row.** In the 2026-10-18 incident the row was
|
||||
`delivery_outcome: delivered` — the canned non-answer reached Telegram *before* the run
|
||||
failed. **A delivered message is not evidence of delivered content.**
|
||||
3. Confirm the provider is healthy: the 2026-10-18 case rode the same `custom:omni-101` alias
|
||||
that produced the 2026-10-14/15 connection-refused storm and the 2026-10-16/17 and 2026-10-19
|
||||
resource-pressure 503 bursts. See [[runbooks/provider-health]] and
|
||||
[[incidents/2026-10-16-omniroute-503-resource-pressure]].
|
||||
4. Do **not** lower `tool_loop_guardrails` thresholds for this fault — they watch tool calls, and
|
||||
the degenerate-stream guard is a separate mechanism.
|
||||
|
||||
### Verification
|
||||
The next scheduled run of the job completes with `status: completed` and `error: null`, and the
|
||||
delivered message contains real content.
|
||||
|
||||
## Last successfully used
|
||||
2026-07-22
|
||||
2026-07-22 — re-verified against live config and live incidents on 2026-10-20; the *procedure* has
|
||||
not itself been executed end-to-end in a live loop, so `confidence` was raised only to `medium`.
|
||||
|
||||
## Related
|
||||
- [[concepts/tool-calling]]
|
||||
- [[incidents/2026-10-18-degenerate-stream-canned-substitute]]
|
||||
- [[runbooks/provider-health]]
|
||||
- [[runbooks/restart-hermes]]
|
||||
|
||||
Reference in New Issue
Block a user