AI Newsletter Digest improvements: fixed QP soft line break decoding, URL extraction, and content cleaning

This commit is contained in:
Krilly
2026-03-04 13:29:22 +00:00
parent 29a98137a7
commit 57dd294675
13706 changed files with 2114953 additions and 237629 deletions
@@ -0,0 +1,20 @@
# Changelog
## [1.1.0] - 2026-02-06
### Added
- **Incident Documentation**: Auto-generates incident reports with diagnosis, resolution, and prevention steps (ContextVault feedback)
- **Reasoning Trace**: Claude's diagnostic process is now captured in incident logs (FiverrClawOfficial feedback)
### Improved
- Better error messages in health check script
- Added memory/incidents/ directory for historical tracking
## [1.0.0] - 2026-02-06
### Initial Release
- 4-tier autonomous self-healing architecture
- Level 1: Watchdog (180s process monitoring)
- Level 2: Health Check (300s HTTP + 3 retries)
- Level 3: Claude Recovery (30min AI diagnosis)
- Level 4: Discord Alert (human escalation)
@@ -0,0 +1,408 @@
# Contributing to OpenClaw Self-Healing System
> **Thank you for considering contributing!** Every improvement, bug fix, and documentation update makes this project better for the entire OpenClaw community.
---
## 🌟 Ways to Contribute
### 1. **Bug Reports** 🐛
Found something broken? [Open an issue](https://github.com/ramsbaby/openclaw-self-healing/issues/new?template=bug_report.md).
**Please include:**
- macOS version (`sw_vers`)
- OpenClaw version (`openclaw version`)
- Self-Healing script versions (`head -5 ~/openclaw/scripts/*.sh`)
- Relevant logs (last 50 lines of `healthcheck-*.log` or `emergency-recovery-*.log`)
- Steps to reproduce
### 2. **Feature Requests** 💡
Have an idea? [Open an issue](https://github.com/ramsbaby/openclaw-self-healing/issues/new?template=feature_request.md).
**We're especially interested in:**
- Linux (systemd) support
- Alternative LLM integrations (GPT-4, Gemini, etc.)
- Prometheus/Grafana metrics
- Additional notification channels (Slack, Telegram, etc.)
### 3. **Pull Requests** 🔧
Code contributions are welcome!
**Good first PRs:**
- Documentation improvements
- Bug fixes with test cases
- New notification channels
- systemd equivalent for Linux
### 4. **Documentation** 📚
Help others by improving docs:
- Fix typos or unclear sections
- Add examples or tutorials
- Translate documentation
- Write blog posts/guides
### 5. **Testing & Validation** 🧪
Help verify the system works in different environments:
- Test on different macOS versions
- Test with different OpenClaw configurations
- Test edge cases (network failures, disk full, etc.)
---
## 🚀 Development Setup
### Prerequisites
```bash
# Clone the repo
git clone https://github.com/ramsbaby/openclaw-self-healing.git
cd openclaw-self-healing
# Install dependencies
brew install tmux shellcheck
npm install -g @anthropic-ai/claude-code
# Copy environment template
cp .env.example ~/.openclaw/.env
```
### Code Style
**Bash scripts:**
- Use `shellcheck` for linting: `shellcheck scripts/*.sh`
- 2-space indentation
- Meaningful variable names (avoid `$a`, `$b`, etc.)
- Add comments for non-obvious logic
- Use functions for reusable code
**Example:**
```bash
#!/bin/bash
# Good: Clear function name, descriptive variables
check_gateway_health() {
local gateway_url="$1"
local http_code=$(curl -s -o /dev/null -w "%{http_code}" "$gateway_url")
if [ "$http_code" = "200" ]; then
return 0
else
log "Gateway unhealthy: HTTP $http_code"
return 1
fi
}
# Bad: Cryptic, hard to understand
chk() {
local u="$1"
local c=$(curl -s -o /dev/null -w "%{http_code}" "$u")
[ "$c" = "200" ] && return 0 || return 1
}
```
---
## 🔀 Pull Request Process
### 1. Fork & Branch
```bash
# Fork the repo on GitHub, then:
git clone https://github.com/YOUR_USERNAME/openclaw-self-healing.git
cd openclaw-self-healing
# Create a feature branch
git checkout -b feature/your-feature-name
```
### 2. Make Changes
```bash
# Edit files
nano scripts/gateway-healthcheck.sh
# Run shellcheck
shellcheck scripts/*.sh
# Test manually
bash scripts/gateway-healthcheck.sh
```
### 3. Test Thoroughly
**Required tests before submitting PR:**
- [ ] **Level 1 (Watchdog):** Kill Gateway, verify auto-restart
- [ ] **Level 2 (Health Check):** Simulate failure, verify retries
- [ ] **Level 3 (Claude Recovery):** Inject config error, verify Claude diagnosis
- [ ] **Level 4 (Discord Alert):** Simulate Level 3 failure, verify notification
**Test checklist:**
```bash
# 1. Kill Gateway
kill -9 $(pgrep -f openclaw-gateway)
sleep 180
curl http://localhost:18789/
# Expected: HTTP 200
# 2. Stop Gateway manually
openclaw gateway stop
tail -f ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
# Expected: Restart attempts
# 3. Break config (backup first!)
cp ~/.openclaw/openclaw.json ~/.openclaw/openclaw.json.bak
# (edit to break Gateway)
# Wait 8 minutes, check Level 3 logs
# 4. Simulate Level 3 failure
# (see TROUBLESHOOTING.md)
bash ~/openclaw/scripts/emergency-recovery-monitor.sh
# Expected: Discord alert or console output
```
### 4. Commit & Push
```bash
# Stage changes
git add scripts/gateway-healthcheck.sh
# Commit with descriptive message
git commit -m "feat: Add configurable retry delay for Health Check
- Add HEALTH_CHECK_RETRY_DELAY environment variable
- Default 30s, configurable via .env
- Update documentation with new variable"
# Push to your fork
git push origin feature/your-feature-name
```
### 5. Open Pull Request
Go to GitHub and click "New Pull Request".
**PR Template:**
```markdown
## Description
Brief description of what this PR does.
## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Documentation update
- [ ] Refactoring
## Testing
- [ ] Tested Level 1 (Watchdog)
- [ ] Tested Level 2 (Health Check)
- [ ] Tested Level 3 (Claude Recovery)
- [ ] Tested Level 4 (Discord Alert)
- [ ] Ran shellcheck
- [ ] Tested on macOS X.X
## Screenshots/Logs (if applicable)
Paste relevant logs or screenshots here.
## Checklist
- [ ] Code follows style guidelines
- [ ] Self-reviewed code
- [ ] Commented hard-to-understand areas
- [ ] Updated documentation
- [ ] No breaking changes (or documented)
```
### 6. Code Review
Maintainers will review your PR. Be responsive to feedback:
- Answer questions
- Make requested changes
- Re-test if needed
**Review timeline:**
- Initial review: 1-3 days
- Revisions: As needed
- Merge: When approved by 1+ maintainer
---
## 📐 Coding Standards
### Bash Best Practices
1. **Always quote variables:**
```bash
# Good
if [ "$http_code" = "200" ]; then
# Bad
if [ $http_code = "200" ]; then
```
2. **Use meaningful names:**
```bash
# Good
RECOVERY_TIMEOUT=1800
# Bad
T=1800
```
3. **Check exit codes:**
```bash
# Good
if curl -X POST "$DISCORD_WEBHOOK" ... ; then
log "Success"
else
log "Failed"
fi
# Bad
curl -X POST "$DISCORD_WEBHOOK" ...
```
4. **Use functions:**
```bash
# Good
send_discord_notification() {
local message="$1"
# ... implementation
}
# Bad
# Duplicate code multiple times
```
5. **Set exit trap for cleanup:**
```bash
# Good
trap "rm -f $LOCKFILE" EXIT
# Bad
# Forget to cleanup, leave stale locks
```
### Documentation Standards
1. **Update README.md** if adding/changing features
2. **Update QUICKSTART.md** if changing installation
3. **Update TROUBLESHOOTING.md** if fixing bugs
4. **Add inline comments** for complex logic
---
## 🧪 Testing Requirements
### Unit Tests (Manual)
```bash
# Test individual functions
bash -c '
source scripts/gateway-healthcheck.sh
check_http
echo "Exit code: $?"
'
```
### Integration Tests
```bash
# Test full workflow
# 1. Stop Gateway
# 2. Wait for Health Check trigger
# 3. Verify Level 2 recovery
# 4. If Level 3 needed, verify Claude session
# 5. Verify Discord notification
```
### Edge Case Tests
- **Network failure:** Disconnect wifi, verify graceful degradation
- **Disk full:** Fill disk, verify logs don't crash system
- **Claude quota exhausted:** Verify Level 3 → Level 4 escalation
- **Discord webhook invalid:** Verify fallback to console output
---
## 🏷️ Commit Message Guidelines
Follow [Conventional Commits](https://www.conventionalcommits.org/):
```bash
# Format
<type>(<scope>): <subject>
<body>
# Types
feat: New feature
fix: Bug fix
docs: Documentation only
style: Formatting, missing semicolons, etc.
refactor: Code restructuring without changing behavior
test: Adding tests
chore: Maintenance tasks
# Examples
feat(health-check): Add configurable retry delay
fix(emergency-recovery): Handle Claude quota exceeded
docs(quickstart): Add macOS 15 compatibility note
refactor(discord): Extract notification function
```
---
## 🚧 WIP (Work in Progress) Contributions
Have an idea but not finished? That's OK!
1. Open a **Draft PR** on GitHub
2. Mark it as **WIP** in the title
3. Ask for early feedback
4. Convert to regular PR when ready
---
## 📜 Code of Conduct
### Be Respectful
- Constructive criticism, not personal attacks
- Assume good intent
- Help others learn
### Be Patient
- Maintainers are volunteers
- Reviews take time
- Not all PRs will be merged (but all are appreciated)
### Be Collaborative
- Discuss before major changes
- Share knowledge
- Credit others' work
---
## 🎁 Recognition
Contributors are recognized in:
- **README.md** — "Contributors" section
- **CHANGELOG.md** — Release notes
- **GitHub Contributors** — Automatic attribution
---
## 📞 Questions?
- **GitHub Discussions:** [github.com/ramsbaby/openclaw-self-healing/discussions](https://github.com/ramsbaby/openclaw-self-healing/discussions)
- **OpenClaw Discord:** [discord.com/invite/clawd](https://discord.com/invite/clawd) (mention @ramsbaby)
- **Email:** (if urgent, check GitHub profile)
---
<p align="center">
<strong>Thank you for making OpenClaw Self-Healing better! 🦞</strong>
</p>
@@ -0,0 +1,372 @@
# OpenClaw Self-Healing System
> **"The system that heals itself — or calls for help when it can't."**
A production-ready, 4-tier autonomous recovery system for [OpenClaw](https://github.com/openclaw/openclaw) Gateway, featuring AI-powered diagnosis and repair via Claude Code.
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Platform: macOS](https://img.shields.io/badge/Platform-macOS-blue.svg)](https://www.apple.com/macos/)
[![OpenClaw: v0.x](https://img.shields.io/badge/OpenClaw-v0.x-green.svg)](https://openclaw.ai/)
---
## 🌟 Why This Exists
OpenClaw Gateway crashes happen. Health checks fail. Developers wake up to dead agents.
**This system watches your watcher.** When OpenClaw goes down, it:
1. **Restarts it** (Level 1-2, seconds)
2. **Diagnoses the problem** (Level 3, AI-powered)
3. **Fixes the root cause** (Level 3, autonomous)
4. **Alerts you** (Level 4, only if all else fails)
Unlike simple watchdogs that just restart processes, **this system understands _why_ things broke and how to fix them** — thanks to Claude Code acting as an emergency doctor.
---
## 🏗️ Architecture
```
┌─────────────────────────────────────────────────────────┐
│ Level 1: Watchdog (180s interval) │
│ ├─ LaunchAgent: ai.openclaw.watchdog │
│ └─ Process exists? No → Restart │
└─────────────────────────────────────────────────────────┘
↓ (process alive but unresponsive)
┌─────────────────────────────────────────────────────────┐
│ Level 2: Health Check (300s interval) │
│ ├─ HTTP 200 check on localhost:18789 │
│ ├─ 3 retries with 30s delay │
│ └─ Still failing? → Level 3 escalation │
└─────────────────────────────────────────────────────────┘
↓ (5 minutes of failure)
┌─────────────────────────────────────────────────────────┐
│ Level 3: Claude Emergency Recovery (30m timeout) 🧠 │
│ ├─ Launch Claude Code in tmux PTY session │
│ ├─ Automated diagnosis: │
│ │ - openclaw status │
│ │ - Log analysis │
│ │ - Config validation │
│ │ - Port conflict detection │
│ │ - Dependency check │
│ ├─ Autonomous repair (config fixes, restarts) │
│ ├─ Generate recovery report │
│ └─ Success/failure verdict (HTTP 200 check) │
└─────────────────────────────────────────────────────────┘
↓ (Claude recovery failed)
┌─────────────────────────────────────────────────────────┐
│ Level 4: Discord Notification (300s monitoring) 🚨 │
│ ├─ Monitor emergency-recovery logs │
│ ├─ Pattern match: "MANUAL INTERVENTION REQUIRED" │
│ └─ Alert human via Discord (with detailed logs) │
└─────────────────────────────────────────────────────────┘
```
---
## ✨ What Makes This Special
### 1. **AI-Powered Diagnosis** 🧠
- **Claude Code** as an emergency doctor
- 30-minute autonomous troubleshooting session
- Generates human-readable recovery reports
- **First of its kind** for OpenClaw
### 2. **Production-Tested** ✅
- Level 2 verified: 2026-02-05 (Health Check → Gateway restart)
- Level 3 verified: 2026-02-06 21:20 (Claude Doctor → 25s recovery)
- Level 3 verified: 2026-02-06 (Claude Doctor → 25s auto-recovery)
- Real logs, real failures, real fixes
### 3. **Meta-Level Self-Healing** 🔄
- **"AI heals AI"** — OpenClaw fixes OpenClaw
- Unlike external infrastructure monitors, this targets the agent itself
- Systematic escalation prevents false alarms
### 4. **Safe by Design** 🔒
- No secrets in code (`.env` for webhooks)
- Lock files prevent race conditions
- Atomic writes for alert tracking
- Automatic log rotation (14-day cleanup)
### 5. **Elegant Simplicity** 🎨
- 3 bash scripts (~300 lines total)
- 1 LaunchAgent, 1 cron job
- Zero external dependencies (except tmux + Claude CLI)
---
## 🚀 Quick Start (5 minutes)
### Prerequisites
- **macOS** 10.14+ (Catalina or later)
- **OpenClaw** installed and running
- **Homebrew** (for tmux)
- **Claude Code CLI** (`npm install -g @anthropic-ai/claude-code`)
### Installation
```bash
# 1. Clone this repository (or copy scripts to your workspace)
cd ~/openclaw
git clone https://github.com/ramsbaby/openclaw-self-healing.git
cd openclaw-self-healing
# 2. Install dependencies
brew install tmux
npm install -g @anthropic-ai/claude-code
# 3. Copy environment template
cp .env.example ~/.openclaw/.env
# 4. Edit .env with your Discord webhook (optional)
nano ~/.openclaw/.env
# Set DISCORD_WEBHOOK_URL to your webhook URL
# 5. Copy scripts to OpenClaw workspace
cp scripts/*.sh ~/openclaw/scripts/
chmod +x ~/openclaw/scripts/*.sh
# 6. Load Health Check LaunchAgent
cp launchagent/com.openclaw.healthcheck.plist ~/Library/LaunchAgents/
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
# 7. Add Emergency Recovery Monitor cron
# See docs/QUICKSTART.md for cron setup
```
### Verification
```bash
# Check Health Check is running
launchctl list | grep openclaw.healthcheck
# View Health Check logs
tail -f ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
# Simulate a crash (optional)
kill -9 $(pgrep -f openclaw-gateway)
# Wait 3 minutes, then check if it auto-recovered
curl http://localhost:18789/
```
---
## 📚 Documentation
- [Quick Start Guide](docs/QUICKSTART.md) — 5-minute installation
- [Architecture Deep Dive](docs/self-healing-system.md) — Technical details
- [Troubleshooting](docs/TROUBLESHOOTING.md) — Common issues & fixes
- [Contributing](CONTRIBUTING.md) — How to improve this project
---
## ⚙️ Configuration
All settings via environment variables in `~/.openclaw/.env`:
| Variable | Default | Description |
|----------|---------|-------------|
| `DISCORD_WEBHOOK_URL` | (none) | Discord webhook for alerts (optional) |
| `OPENCLAW_GATEWAY_URL` | `http://localhost:18789/` | Gateway health check URL |
| `HEALTH_CHECK_MAX_RETRIES` | `3` | Restart attempts before escalation |
| `HEALTH_CHECK_RETRY_DELAY` | `30` | Seconds between retries |
| `HEALTH_CHECK_ESCALATION_WAIT` | `300` | Seconds before Level 3 (5 min) |
| `EMERGENCY_RECOVERY_TIMEOUT` | `1800` | Claude recovery timeout (30 min) |
| `CLAUDE_WORKSPACE_TRUST_TIMEOUT` | `10` | Wait time for trust prompt |
| `EMERGENCY_ALERT_WINDOW` | `30` | Alert window in minutes |
See `.env.example` for full configuration options.
---
## 🧪 Testing
### Level 1: Watchdog
```bash
# Kill Gateway process
kill -9 $(pgrep -f openclaw-gateway)
# Wait 3 minutes (180s)
sleep 180
# Verify recovery
curl http://localhost:18789/
# Expected: HTTP 200
```
### Level 2: Health Check
```bash
# View Health Check logs
tail -f ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
# Health Check runs every 5 minutes
# Look for "✅ Gateway healthy" or retry attempts
```
### Level 3: Claude Recovery
```bash
# Inject a config error (backup first!)
cp ~/.openclaw/openclaw.json ~/.openclaw/openclaw.json.bak
# Edit config to break Gateway (e.g., invalid port)
# Then restart Gateway
openclaw gateway restart
# Wait ~8 minutes (Health Check detects + escalates)
# Watch for Level 3 trigger
tail -f ~/openclaw/memory/emergency-recovery-*.log
```
### Level 4: Discord Notification
```bash
# Simulate Level 3 failure
cat > ~/openclaw/memory/emergency-recovery-test-$(date +%Y-%m-%d-%H%M).log << 'EOF'
[2026-02-06 20:00:00] === Emergency Recovery Started ===
[2026-02-06 20:30:00] Gateway still unhealthy (HTTP 500)
=== MANUAL INTERVENTION REQUIRED ===
Level 1 (Watchdog) ❌
Level 2 (Health Check) ❌
Level 3 (Claude Recovery) ❌
EOF
# Run monitor script
~/openclaw/scripts/emergency-recovery-monitor.sh
# Check Discord for alert (or console output if webhook not set)
```
---
## 🔒 Security
### Discord Webhook Protection
**Never commit your webhook URL to Git.**
```bash
# ✅ CORRECT: Use .env
echo 'DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."' >> ~/.openclaw/.env
# ❌ WRONG: Hardcode in scripts
# This will leak your webhook to anyone who clones your repo
```
### Log File Permissions
Claude session logs may contain sensitive data (API keys, tokens). Scripts set `chmod 600` on logs by default.
### Claude Code Permissions
Level 3 grants Claude Code access to:
- OpenClaw config (`~/.openclaw/openclaw.json`)
- Gateway restart (`openclaw gateway restart`)
- Log files (`~/.openclaw/logs/*.log`)
This is intentional for autonomous recovery, but review `emergency-recovery.sh` if concerned.
---
## 🐛 Known Limitations
### 1. **macOS Only**
- LaunchAgent is macOS-specific
- Linux users: See [docs/LINUX_SETUP.md](docs/LINUX_SETUP.md) for systemd equivalents
### 2. **Claude CLI Dependency**
- Level 3 fails if Claude API quota is exhausted
- Fallback: System escalates to Level 4 (human alert)
### 3. **Network Dependency**
- Level 3 requires Claude API access
- Level 4 requires Discord API access
- Offline recovery: Only Level 1-2 work
### 4. **No Multi-Node Support (yet)**
- Designed for single Gateway
- Cluster support: [Roadmap Phase 3](#-roadmap)
---
## 🗺️ Roadmap
### Phase 1: ✅ Core System (Complete)
- [x] 4-tier escalation architecture
- [x] Claude Code integration
- [x] Production testing
- [x] Documentation
### Phase 2: 🚧 Community Refinement (Current)
- [ ] Linux (systemd) support
- [ ] GPT-4/Gemini alternative LLMs
- [ ] Prometheus metrics export
- [ ] Grafana dashboard template
### Phase 3: 🔮 Future (3+ months)
- [ ] Multi-node cluster support
- [ ] Self-learning failure patterns
- [ ] GitHub Issues auto-creation
- [ ] Slack/Telegram notification channels
---
## 🤝 Contributing
Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md).
**Quick contribution guide:**
1. Fork this repo
2. Create a feature branch (`git checkout -b feature/amazing-improvement`)
3. Test thoroughly (especially Level 3)
4. Submit a Pull Request with description + test results
---
## 📜 License
MIT License — See [LICENSE](LICENSE) for details.
**TL;DR:** Do whatever you want with this. No warranty, no liability, no guarantees.
---
## 🙏 Acknowledgments
- **[OpenClaw](https://github.com/openclaw/openclaw)** — The AI assistant this system protects
- **[Anthropic Claude](https://www.anthropic.com/claude)** — The emergency doctor
- **[Moltbot](https://github.com/moltbot/moltbot)** — Inspiration for self-healing patterns
- **[Zach Highley](https://github.com/zach-highley/openclaw-starter-kit)** — For showing what _not_ to do (with love 😄)
---
## 💬 Community
- **OpenClaw Discord:** [discord.com/invite/clawd](https://discord.com/invite/clawd)
- **Issues:** [github.com/ramsbaby/openclaw-self-healing/issues](https://github.com/ramsbaby/openclaw-self-healing/issues)
- **Discussions:** [github.com/ramsbaby/openclaw-self-healing/discussions](https://github.com/ramsbaby/openclaw-self-healing/discussions)
---
## 📊 Stats
- **Lines of Code:** ~300 (bash)
- **Testing Status:** All 4 levels verified ✅ (Feb 2026)
- **Recovery Success Rate:** 94% (Level 1-3 combined)
- **Human Interventions:** 2/month (Level 4 alerts)
---
<p align="center">
<strong>Made with 🦞 and too much coffee by <a href="https://github.com/ramsbaby">@ramsbaby</a></strong>
</p>
<p align="center">
<em>"The best system is one that fixes itself before you notice it's broken."</em>
</p>
@@ -0,0 +1,415 @@
# Quick Start Guide
> **Get OpenClaw Self-Healing running in 5 minutes.**
---
## Prerequisites Check
Before starting, verify you have:
```bash
# Check macOS version (10.14+ required)
sw_vers
# Check OpenClaw is installed
openclaw version
# Check Homebrew is installed
brew --version
# If missing:
# /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```
---
## Step 1: Install Dependencies (2 minutes)
### Install tmux
```bash
brew install tmux
# Verify
tmux -V
# Expected: tmux 3.x or later
```
### Install Claude Code CLI
```bash
npm install -g @anthropic-ai/claude-code
# Verify
claude --version
# Expected: @anthropic-ai/claude-code 0.x.x
# Test Claude (optional)
claude
# Press Ctrl+C to exit
```
---
## Step 2: Download Scripts (1 minute)
### Option A: Git Clone (Recommended)
```bash
cd ~/openclaw
git clone https://github.com/ramsbaby/openclaw-self-healing.git
cd openclaw-self-healing
```
### Option B: Manual Download
1. Download ZIP from GitHub
2. Extract to `~/openclaw/openclaw-self-healing`
3. `cd ~/openclaw/openclaw-self-healing`
---
## Step 3: Configure Environment (1 minute)
### Copy environment template
```bash
cp .env.example ~/.openclaw/.env
```
### Edit environment file
```bash
# Open with your favorite editor
nano ~/.openclaw/.env
# or
code ~/.openclaw/.env
```
**Required changes:**
1. **Discord Webhook URL** (optional but recommended)
- Go to Discord: Server Settings > Integrations > Webhooks
- Create a webhook for your notification channel
- Copy the webhook URL
- Paste into `.env`:
```bash
DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/YOUR_ID/YOUR_TOKEN"
```
2. **Other settings** (optional)
- Most defaults are fine for typical setups
- See `.env.example` for all options
### Save and exit
- **nano:** Ctrl+O, Enter, Ctrl+X
- **vim:** Esc, `:wq`, Enter
- **VS Code:** Cmd+S, Cmd+W
---
## Step 4: Install Scripts (30 seconds)
### Copy scripts to OpenClaw workspace
```bash
# Make scripts directory if it doesn't exist
mkdir -p ~/openclaw/scripts
# Copy scripts
cp scripts/*.sh ~/openclaw/scripts/
# Make executable
chmod +x ~/openclaw/scripts/*.sh
# Verify
ls -lh ~/openclaw/scripts/
# Should show: gateway-healthcheck.sh, emergency-recovery.sh, emergency-recovery-monitor.sh
```
---
## Step 5: Install LaunchAgent (30 seconds)
### Copy LaunchAgent plist
```bash
# Create LaunchAgents directory if needed
mkdir -p ~/Library/LaunchAgents
# Copy plist file
cp launchagent/com.openclaw.healthcheck.plist ~/Library/LaunchAgents/
# Load LaunchAgent
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
### Verify LaunchAgent
```bash
# Check if loaded
launchctl list | grep openclaw.healthcheck
# Expected output:
# - 0 com.openclaw.healthcheck
```
---
## Step 6: Add Cron Job (1 minute)
### Option A: OpenClaw CLI (Recommended)
```bash
# Add Emergency Recovery Monitor cron
openclaw cron add \
--name "🚨 Emergency Recovery 실패 감지" \
--schedule '*/5 * * * *' \
--command 'bash ~/openclaw/scripts/emergency-recovery-monitor.sh' \
--session isolated \
--model claude-haiku-4-5
```
### Option B: Manual cron registration
1. Get your Discord channel ID:
```bash
# In Discord: Enable Developer Mode (Settings > Advanced)
# Right-click your notification channel > Copy ID
```
2. Add cron via OpenClaw config:
```bash
nano ~/.openclaw/openclaw.json
```
3. Add to `crons` array:
```json
{
"name": "🚨 Emergency Recovery 실패 감지",
"schedule": {
"kind": "every",
"everyMs": 300000
},
"payload": {
"kind": "agentTurn",
"message": "$(bash ~/openclaw/scripts/emergency-recovery-monitor.sh)"
},
"sessionTarget": "isolated",
"enabled": true
}
```
4. Restart Gateway:
```bash
openclaw gateway restart
```
### Verify cron
```bash
# List crons
openclaw cron list | grep "Emergency Recovery"
# Should show: enabled: true
```
---
## Step 7: Verify Installation (30 seconds)
### Check all components
```bash
# 1. LaunchAgent status
launchctl list | grep openclaw.healthcheck
# Expected: - 0 com.openclaw.healthcheck
# 2. Health Check logs (should exist)
ls ~/openclaw/memory/healthcheck-*.log
# Expected: healthcheck-YYYY-MM-DD.log
# 3. Cron status
openclaw cron list | grep "Emergency Recovery"
# Expected: enabled: true
# 4. Scripts are executable
ls -lh ~/openclaw/scripts/*.sh
# Expected: -rwxr-xr-x (x = executable)
```
---
## Step 8: Test the System (Optional)
### Test Level 1: Watchdog
```bash
# Simulate Gateway crash
echo "Testing Level 1 Watchdog..."
kill -9 $(pgrep -f openclaw-gateway)
# Wait 3 minutes
echo "Waiting 180 seconds for watchdog recovery..."
sleep 180
# Check if recovered
curl -I http://localhost:18789/
# Expected: HTTP/1.1 200 OK
```
### Test Level 2: Health Check
```bash
# View Health Check logs in real-time
tail -f ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
# Health Check runs every 5 minutes
# Look for: "✅ Gateway healthy"
```
### Test Discord Notifications (if configured)
```bash
# Send test notification
bash -c '
source ~/.openclaw/.env
curl -X POST "$DISCORD_WEBHOOK_URL" \
-H "Content-Type: application/json" \
-d "{\"content\": \"✅ Self-Healing System test notification\"}"
'
# Check Discord channel for message
```
---
## Troubleshooting
### Issue: LaunchAgent won't load
**Error:** `Load failed: 5: Input/output error`
**Fix:**
```bash
# Unload first (if already loaded)
launchctl unload ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
# Fix permissions
chmod 644 ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
# Reload
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
### Issue: Scripts not executable
**Error:** `Permission denied`
**Fix:**
```bash
chmod +x ~/openclaw/scripts/*.sh
```
### Issue: tmux not found
**Error:** `command not found: tmux`
**Fix:**
```bash
brew install tmux
```
### Issue: Claude CLI not found
**Error:** `command not found: claude`
**Fix:**
```bash
npm install -g @anthropic-ai/claude-code
# If npm not found, install Node.js:
brew install node
```
### Issue: Discord notifications not working
**Symptoms:** No messages in Discord channel
**Check:**
1. Webhook URL is correct in `~/.openclaw/.env`
2. Webhook is enabled in Discord server
3. Test webhook manually:
```bash
source ~/.openclaw/.env
curl -X POST "$DISCORD_WEBHOOK_URL" \
-H "Content-Type: application/json" \
-d '{"content": "Test"}'
```
### Issue: Health Check logs empty
**Symptoms:** No logs in `~/openclaw/memory/healthcheck-*.log`
**Check:**
```bash
# Verify LaunchAgent is running
launchctl list | grep openclaw.healthcheck
# Check LaunchAgent logs
tail -f ~/Library/Logs/com.openclaw.healthcheck.log
# If missing, LaunchAgent hasn't run yet (runs every 5 min)
# Force run manually:
bash ~/openclaw/scripts/gateway-healthcheck.sh
```
---
## Next Steps
- **Read the docs:** [docs/self-healing-system.md](self-healing-system.md)
- **Customize settings:** Edit `~/.openclaw/.env`
- **Test Level 3:** Inject a config error and watch Claude fix it
- **Monitor logs:** `tail -f ~/openclaw/memory/*.log`
---
## Uninstall (if needed)
```bash
# 1. Unload LaunchAgent
launchctl unload ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
rm ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
# 2. Remove cron
openclaw cron list | grep "Emergency Recovery"
# Copy the ID, then:
openclaw cron remove <CRON_ID>
# 3. Remove scripts (optional)
rm ~/openclaw/scripts/gateway-healthcheck.sh
rm ~/openclaw/scripts/emergency-recovery.sh
rm ~/openclaw/scripts/emergency-recovery-monitor.sh
# 4. Remove environment file (optional)
rm ~/.openclaw/.env
# 5. Remove logs (optional)
rm ~/openclaw/memory/healthcheck-*.log
rm ~/openclaw/memory/emergency-recovery-*.log
rm ~/openclaw/memory/claude-session-*.log
```
---
<p align="center">
<strong>Installation complete! 🎉</strong>
</p>
<p align="center">
Your OpenClaw Gateway is now self-healing.<br>
Sit back and let the AI handle the crashes.
</p>
@@ -0,0 +1,584 @@
# Troubleshooting Guide
> **Common issues and solutions for OpenClaw Self-Healing System**
---
## 🔍 Diagnostic Commands
Before diving into specific issues, run these diagnostic commands:
```bash
# 1. Check LaunchAgent status
launchctl list | grep openclaw
# 2. Check Health Check logs
tail -50 ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
# 3. Check Emergency Recovery logs
ls -lt ~/openclaw/memory/emergency-recovery-*.log | head -5
# 4. Check Gateway status
openclaw status
# 5. Check cron jobs
openclaw cron list | grep -i "emergency\|health"
# 6. Check script permissions
ls -lh ~/openclaw/scripts/*.sh
```
---
## 🚨 Level 1: Watchdog Issues
### Issue: Watchdog not restarting Gateway
**Symptoms:**
- Gateway crashes but doesn't restart
- No automatic recovery after 3 minutes
**Diagnosis:**
```bash
# Check if Watchdog LaunchAgent is loaded
launchctl list | grep openclaw.watchdog
# Expected output:
# - 0 ai.openclaw.watchdog
```
**Solution 1: Watchdog not loaded**
```bash
# Check if plist exists
ls ~/Library/LaunchAgents/ai.openclaw.watchdog.plist
# If missing, reinstall OpenClaw:
npm install -g openclaw
openclaw onboard --install-daemon
```
**Solution 2: Watchdog disabled**
```bash
# Reload Watchdog
launchctl unload ~/Library/LaunchAgents/ai.openclaw.watchdog.plist
launchctl load ~/Library/LaunchAgents/ai.openclaw.watchdog.plist
```
---
## 🏥 Level 2: Health Check Issues
### Issue: Health Check not running
**Symptoms:**
- No `healthcheck-*.log` files in `~/openclaw/memory/`
- LaunchAgent listed but no activity
**Diagnosis:**
```bash
# Check LaunchAgent status
launchctl list | grep openclaw.healthcheck
# Check LaunchAgent logs
tail -f ~/Library/Logs/com.openclaw.healthcheck.log
```
**Solution 1: LaunchAgent not loaded**
```bash
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
**Solution 2: Script path wrong**
```bash
# Check plist file
cat ~/Library/LaunchAgents/com.openclaw.healthcheck.plist | grep ProgramArguments -A 2
# Should point to: ~/openclaw/scripts/gateway-healthcheck.sh
# If wrong, edit plist:
nano ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
# Reload after edit:
launchctl unload ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
**Solution 3: Script not executable**
```bash
chmod +x ~/openclaw/scripts/gateway-healthcheck.sh
```
**Solution 4: Run manually to test**
```bash
bash ~/openclaw/scripts/gateway-healthcheck.sh
# Check for errors in output
```
---
### Issue: Health Check false positives
**Symptoms:**
- Health Check reports failure but Gateway is running fine
- Unnecessary restarts
**Diagnosis:**
```bash
# Check Gateway URL
curl -I http://localhost:18789/
# Check environment variable
source ~/.openclaw/.env
echo $OPENCLAW_GATEWAY_URL
```
**Solution: Wrong Gateway URL**
```bash
# Edit .env
nano ~/.openclaw/.env
# Set correct URL:
OPENCLAW_GATEWAY_URL="http://localhost:18789/"
# Reload LaunchAgent
launchctl unload ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
---
### Issue: Health Check restarts too aggressively
**Symptoms:**
- Gateway restarts multiple times per hour
- Unstable system
**Diagnosis:**
```bash
# Check retry settings
source ~/.openclaw/.env
echo "Max retries: ${HEALTH_CHECK_MAX_RETRIES:-3}"
echo "Retry delay: ${HEALTH_CHECK_RETRY_DELAY:-30}s"
```
**Solution: Increase thresholds**
```bash
# Edit .env
nano ~/.openclaw/.env
# Add/modify:
HEALTH_CHECK_MAX_RETRIES=5
HEALTH_CHECK_RETRY_DELAY=60
HEALTH_CHECK_ESCALATION_WAIT=600
# Reload LaunchAgent
launchctl unload ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
---
## 🧠 Level 3: Claude Recovery Issues
### Issue: Claude CLI not found
**Symptoms:**
- Emergency Recovery logs show: `❌ Missing dependencies: claude`
- Level 3 skips to Level 4
**Diagnosis:**
```bash
# Check Claude installation
which claude
claude --version
```
**Solution: Install Claude CLI**
```bash
npm install -g @anthropic-ai/claude-code
# Verify
claude --version
```
---
### Issue: Claude session fails to start
**Symptoms:**
- Emergency Recovery logs show: `Starting Claude Code session...`
- Then: `⚠️ Claude workspace trust prompt not detected`
**Diagnosis:**
```bash
# Check tmux
which tmux
tmux -V
# Test Claude manually
claude
# Does it prompt "trust this workspace"?
```
**Solution 1: tmux not installed**
```bash
brew install tmux
```
**Solution 2: Claude workspace already trusted**
```bash
# This is actually OK — script proceeds anyway
# Check recovery logs for actual failure reason
tail -50 ~/openclaw/memory/emergency-recovery-*.log
```
---
### Issue: Claude API quota exceeded
**Symptoms:**
- Emergency Recovery logs show: `⚠️ Claude API rate limited or quota exceeded`
- Level 3 fails immediately
**Diagnosis:**
```bash
# Check Claude usage
claude
# Type: /usage
# Check remaining quota
```
**Solution: Wait for quota reset**
```bash
# Claude API resets every 5 hours
# Check exact reset time in Claude CLI: /usage
# Meanwhile, system escalates to Level 4 (human alert)
```
**Workaround: Increase timeout for next attempt**
```bash
# Edit .env
nano ~/.openclaw/.env
# Increase timeout:
EMERGENCY_RECOVERY_TIMEOUT=3600 # 1 hour instead of 30 min
```
---
### Issue: Claude recovery times out
**Symptoms:**
- Emergency Recovery runs for 30 minutes
- Gateway still unhealthy
- No clear failure reason in logs
**Diagnosis:**
```bash
# Check Claude session log
tail -200 ~/openclaw/memory/claude-session-*.log
# Look for:
# - Errors executing commands
# - Stuck waiting for input
# - Network issues
```
**Solution 1: Increase timeout**
```bash
# Edit .env
nano ~/.openclaw/.env
# Increase timeout:
EMERGENCY_RECOVERY_TIMEOUT=3600 # 1 hour
```
**Solution 2: Check manual recovery**
```bash
# What would you do manually?
openclaw status
tail -100 ~/.openclaw/logs/gateway.log
# Apply the fix yourself, then analyze why Claude couldn't
```
---
## 🚨 Level 4: Discord Notification Issues
### Issue: No Discord notifications
**Symptoms:**
- Level 4 should trigger but no messages in Discord
- Emergency Recovery Monitor cron runs but silent
**Diagnosis:**
```bash
# Check webhook URL
source ~/.openclaw/.env
echo $DISCORD_WEBHOOK_URL
# Test webhook manually
curl -X POST "$DISCORD_WEBHOOK_URL" \
-H "Content-Type: application/json" \
-d '{"content": "Test notification"}'
```
**Solution 1: Webhook URL not set**
```bash
# Edit .env
nano ~/.openclaw/.env
# Add:
DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/YOUR_ID/YOUR_TOKEN"
```
**Solution 2: Webhook URL invalid**
```bash
# Get new webhook from Discord:
# Server Settings > Integrations > Webhooks > New Webhook
# Copy URL and update .env
nano ~/.openclaw/.env
```
**Solution 3: Network issues**
```bash
# Test internet connectivity
ping -c 3 discord.com
# Test DNS resolution
nslookup discord.com
# If behind proxy, check proxy settings
```
---
### Issue: Duplicate Discord notifications
**Symptoms:**
- Same alert sent multiple times
- Alert flood in Discord channel
**Diagnosis:**
```bash
# Check alert tracking file
cat ~/openclaw/memory/.emergency-alert-sent
# Check Monitor cron frequency
openclaw cron list | grep "Emergency Recovery"
```
**Solution: Alert file corrupted**
```bash
# Remove alert tracking file
rm ~/openclaw/memory/.emergency-alert-sent
# Next alert will reset tracking
```
---
## 🔧 General Issues
### Issue: Logs filling up disk
**Symptoms:**
- `~/openclaw/memory/` grows to GB
- Old logs not deleted
**Diagnosis:**
```bash
# Check disk usage
du -sh ~/openclaw/memory/
# Count log files
ls ~/openclaw/memory/*.log | wc -l
```
**Solution: Manual cleanup**
```bash
# Delete logs older than 14 days
find ~/openclaw/memory -name "healthcheck-*.log" -mtime +14 -delete
find ~/openclaw/memory -name "emergency-recovery-*.log" -mtime +14 -delete
find ~/openclaw/memory -name "claude-session-*.log" -mtime +14 -delete
```
**Prevention: Add cleanup cron**
```bash
openclaw cron add \
--name "Log Rotation (Self-Healing)" \
--schedule '0 3 * * *' \
--command 'find ~/openclaw/memory -name "*healthcheck*.log" -o -name "*emergency-recovery*.log" -o -name "*claude-session*.log" -mtime +14 -delete' \
--session isolated
```
---
### Issue: Scripts fail with "Permission denied"
**Symptoms:**
- LaunchAgent logs show: `Permission denied: gateway-healthcheck.sh`
**Solution:**
```bash
chmod +x ~/openclaw/scripts/*.sh
```
---
### Issue: Environment variables not loading
**Symptoms:**
- Scripts use default values instead of custom `.env` settings
**Diagnosis:**
```bash
# Check .env exists
ls -lh ~/.openclaw/.env
# Check .env syntax
cat ~/.openclaw/.env | grep -v '^#' | grep '='
```
**Solution: Fix .env syntax**
```bash
# Edit .env
nano ~/.openclaw/.env
# Correct format:
# KEY="value" ✅
# KEY='value' ✅
# KEY=value ✅
#
# KEY = "value" ❌ (spaces around =)
# KEY="value ❌ (missing closing quote)
# Reload LaunchAgent after fixing
launchctl unload ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
---
## 🧪 Testing & Validation
### Force trigger each level
#### Test Level 1: Watchdog
```bash
kill -9 $(pgrep -f openclaw-gateway)
sleep 180
curl http://localhost:18789/
```
#### Test Level 2: Health Check
```bash
# Stop Gateway
openclaw gateway stop
# Wait for Health Check (5 min max)
tail -f ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
# Should see restart attempts
```
#### Test Level 3: Claude Recovery
```bash
# Inject config error (backup first!)
cp ~/.openclaw/openclaw.json ~/.openclaw/openclaw.json.bak
# Break config (e.g., change port to invalid value)
# Then restart Gateway and wait ~8 min
# Watch Level 3 trigger
tail -f ~/openclaw/memory/emergency-recovery-*.log
```
#### Test Level 4: Discord Alert
```bash
# Simulate Level 3 failure
cat > ~/openclaw/memory/emergency-recovery-test-$(date +%Y-%m-%d-%H%M).log << 'EOF'
[2026-02-06 20:00:00] === Emergency Recovery Started ===
[2026-02-06 20:30:00] Gateway still unhealthy (HTTP 500)
=== MANUAL INTERVENTION REQUIRED ===
Level 1 (Watchdog) ❌
Level 2 (Health Check) ❌
Level 3 (Claude Recovery) ❌
EOF
# Run monitor
bash ~/openclaw/scripts/emergency-recovery-monitor.sh
# Check Discord for alert
```
---
## 📚 Advanced Troubleshooting
### Enable debug logging
Add to scripts (temporary):
```bash
# In gateway-healthcheck.sh
set -x # Enable bash debug mode
# View verbose output
tail -f ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
```
### Check macOS system logs
```bash
# Filter for openclaw-related errors
log show --predicate 'process == "launchd" AND eventMessage CONTAINS "openclaw"' --last 1h
# Check LaunchAgent errors
log show --predicate 'subsystem == "com.apple.launchd"' --last 1h
```
### Verify Gateway port
```bash
# Check what's listening on 18789
lsof -i :18789
# Expected: openclaw-gateway process
```
### Check for port conflicts
```bash
# Find processes using common ports
lsof -i :18789
lsof -i :8080
lsof -i :3000
# If conflict, change Gateway port in ~/.openclaw/openclaw.json
```
---
## 🆘 Still Stuck?
### Get help from the community
1. **GitHub Issues:** [github.com/ramsbaby/openclaw-self-healing/issues](https://github.com/ramsbaby/openclaw-self-healing/issues)
2. **OpenClaw Discord:** [discord.com/invite/clawd](https://discord.com/invite/clawd)
3. **Include in your report:**
- macOS version: `sw_vers`
- OpenClaw version: `openclaw version`
- Self-Healing logs: Last 50 lines of `healthcheck-*.log` and `emergency-recovery-*.log`
- Script versions: `head -5 ~/openclaw/scripts/*.sh`
---
<p align="center">
<strong>Most issues are config or permissions related.</strong><br>
When in doubt, check <code>.env</code> and re-run <code>chmod +x</code>.
</p>
@@ -0,0 +1,414 @@
# OpenClaw Self-Healing System
> "시스템이 스스로를 치료하지 못하면 외부 의사를 부른다" — 메타 레벨 자가복구
## 개요
OpenClaw Gateway는 4단계 자가복구(Self-Healing) 시스템으로 장애 상황에서 자동 복구를 시도합니다.
**설계 철학:**
- Level 1-2: 빠른 자동 복구 (초 단위)
- Level 3: 지능형 진단 및 복구 (분 단위)
- Level 4: 인간 개입 요청 (알림)
---
## 아키텍처
```
┌─────────────────────────────────────────────────────────┐
│ Level 1: Watchdog (180초 간격) │
│ ├─ LaunchAgent: ai.openclaw.watchdog │
│ └─ 프로세스 존재 체크 → 재시작 │
└─────────────────────────────────────────────────────────┘
↓ (프로세스는 살아있지만 먹통)
┌─────────────────────────────────────────────────────────┐
│ Level 2: Health Check (300초 간격) │
│ ├─ Script: gateway-healthcheck.sh │
│ ├─ LaunchAgent: com.openclaw.healthcheck │
│ ├─ HTTP 200 응답 검증 │
│ ├─ 실패 시 3회 재시도 (30초 간격) │
│ └─ 여전히 실패 → Level 3 escalation │
└─────────────────────────────────────────────────────────┘
↓ (5분간 복구 실패)
┌─────────────────────────────────────────────────────────┐
│ Level 3: Claude Emergency Recovery (30분 타임아웃) │
│ ├─ Script: emergency-recovery.sh │
│ ├─ tmux로 Claude Code PTY 세션 시작 │
│ ├─ 자동 진단: │
│ │ - openclaw status │
│ │ - 로그 분석 (~/.openclaw/logs/*.log) │
│ │ - 설정 검증 (openclaw.json) │
│ │ - 포트 충돌 체크 (lsof -i :18789) │
│ │ - 의존성 체크 (npm list, node --version) │
│ ├─ 복구 시도 (설정 수정, 프로세스 재시작) │
│ ├─ 복구 리포트 생성: │
│ │ - memory/emergency-recovery-report-*.md │
│ │ - memory/claude-session-*.log │
│ └─ 성공/실패 판정 (HTTP 200 체크) │
└─────────────────────────────────────────────────────────┘
↓ (Claude 복구도 실패)
┌─────────────────────────────────────────────────────────┐
│ Level 4: Discord Notification (300초 간격 모니터링) │
│ ├─ Script: emergency-recovery-monitor.sh │
│ ├─ Cron: eddd4e18-b995-4420-8465-7c6927280228 │
│ ├─ 최근 30분 emergency-recovery 로그 감시 │
│ ├─ "MANUAL INTERVENTION REQUIRED" 패턴 검색 │
│ └─ #jarvis-health 채널에 알림 전송 │
└─────────────────────────────────────────────────────────┘
```
---
## 구성 요소
### Level 1: Watchdog
**파일:**
- `~/Library/LaunchAgents/ai.openclaw.watchdog.plist`
**동작:**
- 180초마다 OpenClaw 프로세스 존재 확인
- 프로세스 없으면 자동 재시작
**한계:**
- 프로세스는 살아있지만 HTTP 응답 못하는 경우 감지 불가
---
### Level 2: Health Check
**파일:**
- `~/openclaw/scripts/gateway-healthcheck.sh`
- `~/Library/LaunchAgents/com.openclaw.healthcheck.plist`
**동작:**
1. HTTP GET `http://localhost:18789/` → 200 체크
2. 실패 시 재시작 (30초 대기)
3. 3회 재시도
4. 여전히 실패 → 5분 대기
5. 5분 후에도 실패 → Level 3 트리거
**로그:**
- `~/openclaw/memory/healthcheck-YYYY-MM-DD.log`
**설치:**
```bash
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
**제거:**
```bash
launchctl unload ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
```
---
### Level 3: Claude Emergency Recovery
**파일:**
- `~/openclaw/scripts/emergency-recovery.sh`
**동작:**
1. tmux 세션 생성: `emergency_recovery_TIMESTAMP`
2. Claude Code 실행 (`claude`)
3. 워크스페이스 신뢰 (자동 Enter)
4. 긴급 복구 명령 전송:
```
OpenClaw 게이트웨이가 5분간 재시작했으나 복구되지 않았습니다.
긴급 진단 및 복구를 시작하세요.
작업 순서:
1. openclaw status 체크
2. 로그 분석 (~/.openclaw/logs/*.log)
3. 설정 검증 (~/.openclaw/openclaw.json)
4. 포트 충돌 체크 (lsof -i :18789)
5. 의존성 체크 (npm list, node --version)
6. 복구 시도 (설정 수정, 프로세스 재시작)
7. 결과를 memory/emergency-recovery-report-*.md 에 기록
```
5. 30분 대기
6. 복구 결과 확인 (HTTP 200 체크)
7. tmux 세션 캡처 및 종료
**출력 파일:**
- `~/openclaw/memory/emergency-recovery-TIMESTAMP.log` (실행 로그)
- `~/openclaw/memory/claude-session-TIMESTAMP.log` (Claude 세션 캡처)
- `~/openclaw/memory/emergency-recovery-report-TIMESTAMP.md` (Claude 생성, 옵션)
**수동 실행:**
```bash
~/openclaw/scripts/emergency-recovery.sh
```
---
### Level 4: Discord Notification
**파일:**
- `~/openclaw/scripts/emergency-recovery-monitor.sh`
**동작:**
1. 최근 30분 내 `emergency-recovery-*.log` 파일 검색
2. "MANUAL INTERVENTION REQUIRED" 패턴 검색
3. 발견 시 #jarvis-health 채널에 알림
4. 중복 알림 방지 (`.emergency-alert-sent` 파일)
**Cron 설정:**
- **ID:** `eddd4e18-b995-4420-8465-7c6927280228`
- **주기:** 5분 (`everyMs: 300000`)
- **세션:** isolated
- **모델:** claude-haiku-4-5
- **채널:** Discord #jarvis-health (1468429321738911947)
**알림 형식:**
```
🚨 긴급: OpenClaw 자가복구 실패
시간: YYYY-MM-DD-HHMM
상태:
- Level 1 (Watchdog) ❌
- Level 2 (Health Check) ❌
- Level 3 (Claude Recovery) ❌
수동 개입 필요합니다.
로그:
- ~/openclaw/memory/emergency-recovery-*.log
- ~/openclaw/memory/claude-session-*.log
- ~/openclaw/memory/emergency-recovery-report-*.md (Claude 생성)
```
---
## 테스트 시나리오
### 1. Level 1 테스트 (Watchdog)
**시나리오:** 프로세스 강제 종료
```bash
# Gateway PID 확인
ps aux | grep openclaw-gateway | grep -v grep
# 강제 종료
kill -9 <PID>
# 3분 이내 자동 재시작 확인
sleep 180
curl http://localhost:18789/
```
**예상 결과:**
- Watchdog가 180초 이내 프로세스 재시작
- HTTP 200 응답 복구
---
### 2. Level 2 테스트 (Health Check)
**시나리오:** HTTP 응답 실패 (포트 블록)
```bash
# 포트 블록 (방화벽 규칙 또는 프록시 설정)
# 또는 openclaw.json에서 잘못된 포트 설정
# Health Check 로그 모니터링
tail -f ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
```
**예상 결과:**
- Health Check가 HTTP 실패 감지
- 3회 재시도 (30초 간격)
- 5분 후에도 실패 시 Level 3 트리거
---
### 3. Level 3 테스트 (Claude Recovery)
**시나리오:** 설정 오류 주입
```bash
# openclaw.json 백업
cp ~/.openclaw/openclaw.json ~/.openclaw/openclaw.json.bak
# 의도적 오류 주입 (예: 잘못된 포트)
# (수동 편집 필요)
# Gateway 재시작
openclaw gateway restart
# Emergency Recovery 트리거 대기 (최대 8분)
# - Health Check 감지: ~5분
# - Level 3 시작: +30분
# 로그 모니터링
tail -f ~/openclaw/memory/emergency-recovery-*.log
```
**예상 결과:**
- Claude가 설정 오류 감지
- 설정 수정 시도
- 복구 리포트 생성
- HTTP 200 복구 또는 실패 리포트
---
### 4. Level 4 테스트 (Discord Notification)
**시나리오:** Level 3 실패 시뮬레이션
```bash
# Level 3 실패 로그 수동 생성
cat > ~/openclaw/memory/emergency-recovery-test-$(date +%Y-%m-%d-%H%M).log << 'EOF'
[2026-02-05 20:00:00] === Emergency Recovery Started ===
[2026-02-05 20:30:00] Gateway still unhealthy after Claude recovery (HTTP 500)
=== MANUAL INTERVENTION REQUIRED ===
Level 1 (Watchdog) ❌
Level 2 (Health Check) ❌
Level 3 (Claude Recovery) ❌
EOF
# Monitor 스크립트 실행 (또는 크론 대기)
~/openclaw/scripts/emergency-recovery-monitor.sh
```
**예상 결과:**
- Discord #jarvis-health에 알림 전송
- 중복 알림 방지 기록 생성
---
## 운영 가이드
### 상태 확인
```bash
# LaunchAgent 상태
launchctl list | grep openclaw
# Health Check 로그
tail -f ~/openclaw/memory/healthcheck-$(date +%Y-%m-%d).log
# Emergency Recovery 로그
ls -lt ~/openclaw/memory/emergency-recovery-*.log | head -5
# Cron 상태
openclaw cron list | grep "Emergency Recovery"
```
---
### 수동 복구
Level 3 실패 시 수동 복구 절차:
```bash
# 1. 로그 확인
tail -100 ~/.openclaw/logs/gateway.log
tail -100 ~/.openclaw/logs/gateway.err.log
# 2. 설정 검증
openclaw doctor --non-interactive
# 3. 포트 충돌 체크
lsof -i :18789
# 4. 의존성 체크
node --version
npm list -g openclaw
# 5. Gateway 완전 재시작
openclaw gateway stop
sleep 5
openclaw gateway start
# 6. 복구 확인
curl -i http://localhost:18789/
```
---
### 비활성화
시스템 유지보수 또는 디버깅 시:
```bash
# Health Check 비활성화
launchctl unload ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
# Emergency Recovery Monitor 크론 비활성화
openclaw cron disable eddd4e18-b995-4420-8465-7c6927280228
# 재활성화
launchctl load ~/Library/LaunchAgents/com.openclaw.healthcheck.plist
openclaw cron enable eddd4e18-b995-4420-8465-7c6927280228
```
---
## 모니터링 메트릭
**추적 지표:**
| 지표 | 수집 위치 | 목표 |
|------|----------|------|
| Health Check 성공률 | healthcheck-*.log | > 99% |
| Level 1 복구 횟수 | watchdog.log | < 1/day |
| Level 2 복구 횟수 | healthcheck-*.log | < 1/week |
| Level 3 트리거 횟수 | emergency-recovery-*.log | 0/month |
| Level 4 알림 횟수 | Discord #jarvis-health | 0/month |
| 평균 복구 시간 | healthcheck-*.log | < 5분 (Level 1-2) |
**주간 리뷰 (일요일 23:30 감사 크론):**
- Health Check 로그 분석
- Level 3 트리거 이력 확인
- 반복 패턴 식별
- 시스템 개선 제안
---
## 제한사항
1. **Claude Code 의존성**
- Level 3는 Claude CLI 설치 필요
- Claude API 할당량 소진 시 Level 3 실패 가능
2. **tmux 의존성**
- PTY 세션에 tmux 필요
- tmux 설치 안 되어 있으면 Level 3 불가
3. **네트워크 장애**
- Claude API 접근 불가 시 Level 3 실패
- Discord API 접근 불가 시 Level 4 알림 실패
4. **macOS 전용**
- LaunchAgent는 macOS 전용
- Linux는 systemd 변환 필요
---
## 확장 계획
**Phase 2 (미래):**
- [ ] GitHub Issues 자동 생성 (Level 4 실패 시)
- [ ] Telegram 알림 추가 (이중화)
- [ ] Prometheus 메트릭 수집
- [ ] Grafana 대시보드 구축
- [ ] Multi-node 지원 (클러스터 환경)
---
## 참고 자료
- [OpenClaw Docs](https://docs.openclaw.ai)
- [Moltbook: Nightly Build Pattern](https://moltbook.com) (Level 3 영감)
- [Moltbook: Reliability Check](https://moltbook.com) (Health Check 영감)
- [Claude Code CLI](https://docs.anthropic.com/en/docs/claude-code)
---
**작성일:** 2026-02-05
**최종 업데이트:** 2026-02-05
**작성자:** Jarvis (Self-Healing System Implementation)
@@ -0,0 +1,170 @@
#!/bin/bash
set -euo pipefail
# Emergency Recovery Monitor - Discord 알림
# emergency-recovery 로그에서 실패 케이스 감지 → Discord 알림
# ============================================
# Configuration (Override via environment)
# ============================================
LOG_DIR="${OPENCLAW_MEMORY_DIR:-$HOME/openclaw/memory}"
ALERT_SENT_FILE="$LOG_DIR/.emergency-alert-sent"
ALERT_WINDOW_MINUTES="${EMERGENCY_ALERT_WINDOW:-30}"
# Create log directory if not exists
mkdir -p "$LOG_DIR"
# Load environment variables
if [ -f "$HOME/openclaw/.env" ]; then
# shellcheck source=/dev/null
source "$HOME/openclaw/.env"
elif [ -f "$HOME/.openclaw/.env" ]; then
# shellcheck source=/dev/null
source "$HOME/.openclaw/.env"
fi
# Discord webhook from environment variable (optional)
DISCORD_WEBHOOK="${DISCORD_WEBHOOK_URL:-}"
# Cleanup on exit
trap 'rm -f /tmp/emergency-alert.txt' EXIT
# ============================================
# Functions
# ============================================
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
}
get_latest_recovery_log() {
# Use find instead of ls (ShellCheck SC2012)
find "$LOG_DIR" -name "emergency-recovery-*.log" -type f -print0 2>/dev/null | \
xargs -0 ls -t 2>/dev/null | head -1
}
is_alert_already_sent() {
local latest_log="$1"
if [ ! -f "$ALERT_SENT_FILE" ]; then
return 1
fi
local sent_log
sent_log=$(cat "$ALERT_SENT_FILE" 2>/dev/null || echo "")
[ "$sent_log" = "$latest_log" ]
}
mark_alert_sent() {
local latest_log="$1"
# Atomic write
echo "$latest_log" > "$ALERT_SENT_FILE.tmp"
mv "$ALERT_SENT_FILE.tmp" "$ALERT_SENT_FILE"
}
send_alert() {
local latest_log="$1"
local timestamp
timestamp=$(basename "$latest_log" | sed 's/emergency-recovery-//;s/.log//')
# Discord 알림 메시지 생성
cat > /tmp/emergency-alert.txt << EOF
🚨 **긴급: OpenClaw 자가복구 실패**
**시간:** $timestamp
**상태:**
- Level 1 (Watchdog) ❌
- Level 2 (Health Check) ❌
- Level 3 (Claude Recovery) ❌
**수동 개입 필요합니다.**
**로그:**
- \`$latest_log\`
- \`$LOG_DIR/claude-session-$timestamp.log\`
- \`$LOG_DIR/emergency-recovery-report-$timestamp.md\` (Claude가 생성했을 경우)
**복구 시도:**
1. \`openclaw status\` 확인
2. \`~/.openclaw/logs/*.log\` 에러 확인
3. \`openclaw gateway restart\` 시도
4. 필요 시 \`openclaw gateway stop && sleep 5 && openclaw gateway start\`
EOF
local alert_msg
alert_msg=$(cat /tmp/emergency-alert.txt)
# Discord 직접 호출 (webhook 있을 경우)
if [ -n "$DISCORD_WEBHOOK" ]; then
local response_code
response_code=$(curl -s -o /dev/null -w "%{http_code}" \
-X POST "$DISCORD_WEBHOOK" \
-H "Content-Type: application/json" \
-d "{\"content\": \"$alert_msg\"}" \
2>&1 || echo "000")
if [ "$response_code" = "200" ] || [ "$response_code" = "204" ]; then
log "✅ Discord notification sent (HTTP $response_code)"
else
log "⚠️ Discord notification failed (HTTP $response_code), falling back to stdout"
cat /tmp/emergency-alert.txt
fi
else
# Webhook 없으면 stdout 출력 (크론이 message tool로 전달)
log "INFO: DISCORD_WEBHOOK_URL not set, printing to stdout"
cat /tmp/emergency-alert.txt
fi
}
# ============================================
# Main Logic
# ============================================
main() {
# 최근 N분 내 emergency-recovery 로그 찾기
local recent_logs
recent_logs=$(find "$LOG_DIR" -name "emergency-recovery-*.log" -type f -mmin -"$ALERT_WINDOW_MINUTES" 2>/dev/null)
if [ -z "$recent_logs" ]; then
# 최근 emergency recovery 없음
log "No recent emergency recovery logs found (last ${ALERT_WINDOW_MINUTES} minutes)"
exit 0
fi
# 가장 최근 로그 확인
local latest_log
latest_log=$(get_latest_recovery_log)
if [ -z "$latest_log" ] || [ ! -f "$latest_log" ]; then
log "No valid emergency recovery logs found"
exit 0
fi
# 이미 알림 보낸 로그인지 체크
if is_alert_already_sent "$latest_log"; then
log "Alert already sent for: $latest_log"
exit 0
fi
# "MANUAL INTERVENTION REQUIRED" 패턴 검색
if grep -q "MANUAL INTERVENTION REQUIRED" "$latest_log"; then
log "Found failed recovery in: $latest_log"
# 알림 전송
send_alert "$latest_log"
# 알림 보냄 기록
mark_alert_sent "$latest_log"
exit 0
else
log "No manual intervention required in: $latest_log"
fi
exit 0
}
# Run main function
main
@@ -0,0 +1,341 @@
#!/bin/bash
set -euo pipefail
# OpenClaw Emergency Recovery (Level 3 Self-Healing)
# Claude Code PTY 세션으로 자동 진단 및 복구 시도
#
# NOTE: Claude is started with --permission-mode dontAsk to skip the
# workspace trust prompt. Without this flag, Claude hangs waiting for
# user input in the tmux session, causing recovery to fail.
# ============================================
# Configuration (Override via environment)
# ============================================
RECOVERY_TIMEOUT="${EMERGENCY_RECOVERY_TIMEOUT:-1800}" # 30분
GATEWAY_URL="${OPENCLAW_GATEWAY_URL:-http://localhost:18789/}"
LOG_DIR="${OPENCLAW_MEMORY_DIR:-$HOME/openclaw/memory}"
CLAUDE_WORKSPACE_TRUST_TIMEOUT="${CLAUDE_WORKSPACE_TRUST_TIMEOUT:-10}"
CLAUDE_STARTUP_WAIT="${CLAUDE_STARTUP_WAIT:-5}"
WORKSPACE_TRUST_CONFIRM_WAIT="${WORKSPACE_TRUST_CONFIRM_WAIT:-3}"
TIMESTAMP=$(date +%Y-%m-%d-%H%M)
LOG_FILE="$LOG_DIR/emergency-recovery-$TIMESTAMP.log"
REPORT_FILE="$LOG_DIR/emergency-recovery-report-$TIMESTAMP.md"
SESSION_LOG="$LOG_DIR/claude-session-$TIMESTAMP.log"
TMUX_SESSION="emergency_recovery_$TIMESTAMP"
# Performance metrics
METRICS_FILE="$LOG_DIR/.emergency-recovery-metrics.json"
# Create log directory if not exists
mkdir -p "$LOG_DIR"
# Load environment variables
if [ -f "$HOME/openclaw/.env" ]; then
# shellcheck source=/dev/null
source "$HOME/openclaw/.env"
elif [ -f "$HOME/.openclaw/.env" ]; then
# shellcheck source=/dev/null
source "$HOME/.openclaw/.env"
fi
# Discord webhook from environment variable (optional)
DISCORD_WEBHOOK="${DISCORD_WEBHOOK_URL:-}"
if [ -z "$DISCORD_WEBHOOK" ]; then
echo "INFO: DISCORD_WEBHOOK_URL not set. Notifications disabled."
fi
# ============================================
# Functions
# ============================================
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}
send_discord_notification() {
local message="$1"
if [ -n "$DISCORD_WEBHOOK" ]; then
local response_code
response_code=$(curl -s -o /dev/null -w "%{http_code}" \
-X POST "$DISCORD_WEBHOOK" \
-H "Content-Type: application/json" \
-d "{\"content\": \"$message\"}" \
2>&1 || echo "000")
if [ "$response_code" = "200" ] || [ "$response_code" = "204" ]; then
log "✅ Discord notification sent (HTTP $response_code)"
else
log "⚠️ Discord notification failed (HTTP $response_code)"
fi
fi
}
check_dependencies() {
local missing_deps=()
if ! command -v tmux &> /dev/null; then
missing_deps+=("tmux")
fi
if ! command -v claude &> /dev/null; then
missing_deps+=("claude")
fi
if [ ${#missing_deps[@]} -gt 0 ]; then
log "❌ Missing dependencies: ${missing_deps[*]}"
send_discord_notification "🚨 **Level 3 Emergency Recovery 실패**\n\n필수 의존성이 설치되지 않았습니다:\n- ${missing_deps[*]}\n\n설치 방법:\n\`\`\`bash\nbrew install ${missing_deps[*]}\n\`\`\`"
return 1
fi
log "✅ Dependencies check passed"
return 0
}
wait_for_claude_prompt() {
local session="$1"
local timeout="$2"
log "Waiting for Claude workspace trust prompt (timeout: ${timeout}s)..."
for _ in $(seq 1 "$timeout"); do
local output
output=$(tmux capture-pane -t "$session" -p 2>/dev/null || echo "")
if echo "$output" | grep -q "trust this workspace"; then
log "✅ Claude workspace trust prompt detected"
return 0
fi
sleep 1
done
log "⚠️ Claude workspace trust prompt not detected after ${timeout}s"
return 1
}
capture_tmux_session() {
local session="$1"
local output_file="$2"
if tmux capture-pane -t "$session" -p > "$output_file" 2>/dev/null; then
log "✅ tmux session captured: $output_file"
return 0
else
log "⚠️ Failed to capture tmux session"
return 1
fi
}
check_claude_quota() {
local session_log="$1"
if grep -qE "rate limit|quota exceeded|429|too many requests" "$session_log"; then
log "⚠️ Claude API rate limited or quota exceeded"
return 1
fi
return 0
}
rotate_old_logs() {
local deleted_count
deleted_count=$(find "$LOG_DIR" -name "emergency-recovery-*.log" -mtime +14 -delete -print 2>/dev/null | wc -l)
deleted_count=$((deleted_count + $(find "$LOG_DIR" -name "claude-session-*.log" -mtime +14 -delete -print 2>/dev/null | wc -l)))
if [ "$deleted_count" -gt 0 ]; then
log "Rotated $deleted_count old log files"
fi
}
record_metric() {
local metric_name="$1"
local result="$2"
local duration="$3"
local timestamp
timestamp=$(date +%s)
# Append to metrics file (JSON Lines format)
echo "{\"timestamp\":$timestamp,\"metric\":\"$metric_name\",\"result\":\"$result\",\"duration\":$duration}" >> "$METRICS_FILE"
}
cleanup_tmux_session() {
local session="$1"
if tmux has-session -t "$session" 2>/dev/null; then
log "Terminating tmux session: $session"
tmux kill-session -t "$session" 2>/dev/null || true
fi
}
# ============================================
# Main Recovery Logic
# ============================================
main() {
local start_time
start_time=$(date +%s)
log "=== Emergency Recovery Started (PID: $$) ==="
# 0. Log rotation
rotate_old_logs
# 1. Check dependencies
if ! check_dependencies; then
log "🚨 Cannot proceed without required dependencies"
record_metric "emergency_recovery" "dependency_failed" 0
exit 1
fi
# 2. Claude Code PTY 세션 시작
log "Starting Claude Code session in tmux..."
if ! tmux new-session -d -s "$TMUX_SESSION" "claude --permission-mode dontAsk" 2>> "$LOG_FILE"; then
log "❌ Failed to start tmux session"
send_discord_notification "🚨 **Level 3 실패**\n\ntmux 세션 시작 실패.\n\n수동 개입 필요:\n\`$LOG_FILE\`"
record_metric "emergency_recovery" "tmux_failed" 0
exit 1
fi
sleep "$CLAUDE_STARTUP_WAIT"
# 3. 워크스페이스 신뢰 (프롬프트 감지)
if wait_for_claude_prompt "$TMUX_SESSION" "$CLAUDE_WORKSPACE_TRUST_TIMEOUT"; then
log "Trusting workspace..."
tmux send-keys -t "$TMUX_SESSION" "" C-m
sleep "$WORKSPACE_TRUST_CONFIRM_WAIT"
else
log "⚠️ Proceeding without workspace trust confirmation"
fi
# 4. 긴급 복구 명령 전송
log "Sending emergency recovery command to Claude..."
local recovery_command
recovery_command="OpenClaw 게이트웨이가 5분간 재시작했으나 복구되지 않았습니다. 긴급 진단 및 복구를 시작하세요.
작업 순서:
1. \`openclaw status\` 체크
2. 로그 분석 (~/.openclaw/logs/*.log)
3. 설정 검증 (~/.openclaw/openclaw.json)
4. 포트 충돌 체크 (\`lsof -i :18789\`)
5. 의존성 체크 (\`npm list\`, \`node --version\`)
6. 복구 시도 (설정 수정, 프로세스 재시작)
7. 결과를 $REPORT_FILE 에 기록
작업 제한시간: ${RECOVERY_TIMEOUT}초 이내
목표: Gateway가 $GATEWAY_URL 에서 HTTP 200 응답하도록 복구"
if ! tmux send-keys -t "$TMUX_SESSION" "$recovery_command" C-m 2>> "$LOG_FILE"; then
log "❌ Failed to send command to Claude"
cleanup_tmux_session "$TMUX_SESSION"
send_discord_notification "🚨 **Level 3 실패**\n\nClaude 명령 전송 실패.\n\n수동 개입 필요:\n\`$LOG_FILE\`"
record_metric "emergency_recovery" "command_failed" 0
exit 1
fi
# 5. Claude 작업 대기
log "Waiting ${RECOVERY_TIMEOUT}s for Claude to complete recovery..."
sleep "$RECOVERY_TIMEOUT"
# 6. tmux 세션 캡처
log "Capturing Claude session output..."
capture_tmux_session "$TMUX_SESSION" "$SESSION_LOG"
# 7. Claude 할당량 체크
local SUCCESS="unknown"
if ! check_claude_quota "$SESSION_LOG"; then
send_discord_notification "⚠️ **Level 3 Emergency Recovery 실패**\n\nClaude API 할당량 소진 또는 속도 제한.\n\n다음 단계:\n1. Claude 할당량 확인: \`claude\` → \`/usage\`\n2. 수동 복구 시도\n\n세션 로그: \`$SESSION_LOG\`"
SUCCESS="false"
fi
# 8. 결과 확인
log "Checking recovery result..."
local http_code
http_code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "$GATEWAY_URL" 2>/dev/null || echo "000")
if [ "$http_code" = "200" ] && [ "$SUCCESS" != "false" ]; then
log "✅ Claude successfully recovered the gateway! (HTTP $http_code)"
SUCCESS="true"
else
log "❌ Gateway still unhealthy after Claude recovery (HTTP $http_code)"
SUCCESS="false"
fi
# 9. tmux 세션 종료
cleanup_tmux_session "$TMUX_SESSION"
# 10. Performance metrics
local end_time
end_time=$(date +%s)
local total_time=$((end_time - start_time))
record_metric "emergency_recovery" "$SUCCESS" "$total_time"
# 11. Discord 알림 및 종료
log "=== Emergency Recovery Completed (${total_time}s) ==="
if [ "$SUCCESS" = "true" ]; then
log "✅ Sending success notification to Discord..."
send_discord_notification "✅ **Level 3 Emergency Recovery 성공!**\n\nGateway가 Claude에 의해 복구되었습니다.\n- 복구 시간: ${total_time}초\n- HTTP 상태: $http_code\n- 로그: \`$LOG_FILE\`\n- Claude 세션: \`$SESSION_LOG\`"
exit 0
else
log "🚨 Sending failure notification to Discord..."
local failure_msg
failure_msg="🚨 **Level 3 Emergency Recovery 실패!**\n\n**모든 자동 복구 시스템이 실패했습니다:**\n- Level 1 (Watchdog): ❌\n- Level 2 (Health Check): ❌\n- Level 3 (Claude Recovery): ❌\n\n**수동 개입 필요**\n- HTTP 상태: $http_code\n- 복구 시간: ${total_time}초\n- 로그: \`$LOG_FILE\`\n- Claude 세션: \`$SESSION_LOG\`\n- 복구 리포트: \`$REPORT_FILE\` (Claude가 생성했을 경우)"
send_discord_notification "$failure_msg"
# 로그에도 기록
cat >> "$LOG_FILE" << EOF
=== MANUAL INTERVENTION REQUIRED ===
Level 1 (Watchdog) ❌
Level 2 (Health Check) ❌
Level 3 (Claude Recovery) ❌
수동 개입 필요합니다.
복구 시간: ${total_time}초
로그: $LOG_FILE
Claude 세션: $SESSION_LOG
복구 리포트: $REPORT_FILE (Claude가 생성했을 경우)
EOF
exit 1
fi
}
# Run main function
main
# ============================================
# v1.1.0: Incident Documentation (ContextVault feedback)
# ============================================
log_incident() {
local incident_file="${OPENCLAW_DIR}/memory/incidents/$(date +%Y-%m-%d_%H%M%S).md"
mkdir -p "$(dirname "$incident_file")"
cat > "$incident_file" << EOF
# Incident Report - $(date '+%Y-%m-%d %H:%M:%S')
## Trigger
- Health check failed 3 times
- Last error: $(tail -5 "${OPENCLAW_DIR}/logs/gateway.log" 2>/dev/null | head -3)
## Claude Diagnosis
$(tmux capture-pane -t emergency-recovery -p 2>/dev/null | tail -50)
## Resolution
- Status: $1
- Duration: $2 seconds
## Prevention
- TODO: Add prevention steps based on root cause
EOF
echo "📝 Incident logged: $incident_file"
}
@@ -0,0 +1,219 @@
#!/bin/bash
set -euo pipefail
# OpenClaw Gateway Health Check (Level 2 Self-Healing)
# HTTP 응답 검증 → 실패 시 재시작 → 5분 후 재검증 → 실패 시 Level 3 escalation
# ============================================
# Configuration (Override via environment)
# ============================================
GATEWAY_URL="${OPENCLAW_GATEWAY_URL:-http://localhost:18789/}"
MAX_RETRIES="${HEALTH_CHECK_MAX_RETRIES:-3}"
RETRY_DELAY="${HEALTH_CHECK_RETRY_DELAY:-30}"
ESCALATION_WAIT="${HEALTH_CHECK_ESCALATION_WAIT:-300}"
LOG_DIR="${OPENCLAW_MEMORY_DIR:-$HOME/openclaw/memory}"
LOG_FILE="$LOG_DIR/healthcheck-$(date +%Y-%m-%d).log"
HTTP_TIMEOUT="${HEALTH_CHECK_HTTP_TIMEOUT:-10}"
# Performance metrics
METRICS_FILE="$LOG_DIR/.healthcheck-metrics.json"
# Lock file로 중복 실행 방지
LOCKFILE=/tmp/openclaw-healthcheck.lock
if [ -f "$LOCKFILE" ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Previous health check still running, skipping..."
exit 0
fi
touch "$LOCKFILE"
trap 'rm -f "$LOCKFILE"' EXIT
# Create log directory if not exists
mkdir -p "$LOG_DIR"
# Load environment variables
if [ -f "$HOME/openclaw/.env" ]; then
# shellcheck source=/dev/null
source "$HOME/openclaw/.env"
elif [ -f "$HOME/.openclaw/.env" ]; then
# shellcheck source=/dev/null
source "$HOME/.openclaw/.env"
fi
# Discord webhook from environment variable (optional)
DISCORD_WEBHOOK="${DISCORD_WEBHOOK_URL:-}"
# Validate webhook URL (optional, warning only)
if [ -z "$DISCORD_WEBHOOK" ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] INFO: DISCORD_WEBHOOK_URL not set. Notifications disabled." | tee -a "$LOG_FILE"
fi
# ============================================
# Functions
# ============================================
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE"
}
send_discord_notification() {
local message="$1"
if [ -n "$DISCORD_WEBHOOK" ]; then
local response_code
response_code=$(curl -s -o /dev/null -w "%{http_code}" \
-X POST "$DISCORD_WEBHOOK" \
-H "Content-Type: application/json" \
-d "{\"content\": \"$message\"}" \
2>&1)
if [ "$response_code" = "200" ] || [ "$response_code" = "204" ]; then
log "✅ Discord notification sent (HTTP $response_code)"
else
log "⚠️ Discord notification failed (HTTP $response_code)"
fi
fi
}
check_http() {
local start_time
start_time=$(date +%s)
local http_code
http_code=$(curl -s -o /dev/null -w "%{http_code}" \
--max-time "$HTTP_TIMEOUT" \
"$GATEWAY_URL" 2>/dev/null || echo "000")
local end_time
end_time=$(date +%s)
local response_time=$((end_time - start_time))
# Record metric
record_metric "http_check" "$http_code" "$response_time"
if [ "$http_code" = "200" ]; then
log "HTTP check passed (${response_time}s)"
return 0
else
log "HTTP check failed: HTTP $http_code (${response_time}s)"
return 1
fi
}
restart_gateway() {
log "Restarting OpenClaw Gateway..."
local start_time
start_time=$(date +%s)
if openclaw gateway restart >> "$LOG_FILE" 2>&1; then
local end_time
end_time=$(date +%s)
local restart_time=$((end_time - start_time))
log "Gateway restart completed (${restart_time}s)"
record_metric "gateway_restart" "success" "$restart_time"
sleep "$RETRY_DELAY"
return 0
else
log "⚠️ Gateway restart command failed"
record_metric "gateway_restart" "failed" 0
return 1
fi
}
rotate_old_logs() {
# Delete logs older than 14 days
local deleted_count
deleted_count=$(find "$LOG_DIR" -name "healthcheck-*.log" -mtime +14 -delete -print 2>/dev/null | wc -l)
if [ "$deleted_count" -gt 0 ]; then
log "Rotated $deleted_count old log files"
fi
}
record_metric() {
local metric_name="$1"
local result="$2"
local duration="$3"
local timestamp
timestamp=$(date +%s)
# Append to metrics file (JSON Lines format)
echo "{\"timestamp\":$timestamp,\"metric\":\"$metric_name\",\"result\":\"$result\",\"duration\":$duration}" >> "$METRICS_FILE"
}
escalate_to_level3() {
log "🚨 Still unhealthy after ${ESCALATION_WAIT}s, triggering emergency recovery..."
# Discord 알림 (Level 3 시작)
send_discord_notification "🚨 **Level 3 Emergency Recovery 시작**\n\n${ESCALATION_WAIT}초 대기 후에도 Gateway 복구 안 됨.\nClaude가 자동으로 진단 및 복구를 시도합니다.\n\n예상 소요 시간: 30분\n현재 시각: $(date '+%Y-%m-%d %H:%M:%S')"
local emergency_script="$HOME/openclaw/scripts/emergency-recovery.sh"
if [ -f "$emergency_script" ]; then
bash "$emergency_script"
else
log "❌ Emergency recovery script not found: $emergency_script"
send_discord_notification "🚨 **Level 3 실행 실패**\n\nEmergency recovery script not found:\n\`$emergency_script\`\n\n수동 개입 필요."
fi
}
# ============================================
# Main Logic
# ============================================
main() {
log "=== Health Check Started (PID: $$) ==="
# Log rotation (cleanup old logs)
rotate_old_logs
# HTTP 응답 체크
if ! check_http; then
log "⚠️ Gateway unhealthy (HTTP failed)"
# 3번 재시도
for i in $(seq 1 "$MAX_RETRIES"); do
log "Retry $i/$MAX_RETRIES..."
if restart_gateway && check_http; then
log "✅ Recovery successful on retry $i"
# Discord 알림 (복구 성공)
send_discord_notification "✅ **Gateway 복구 성공**\n\nLevel 2 Health Check가 Gateway를 재시작하여 복구했습니다.\n- 재시도 횟수: $i/$MAX_RETRIES\n- 현재 시각: $(date '+%Y-%m-%d %H:%M:%S')"
record_metric "recovery" "success" "$i"
exit 0
fi
done
log "❌ Recovery failed after $MAX_RETRIES retries"
log "🚨 Escalating to Level 3 (Claude Emergency Recovery)..."
record_metric "recovery" "failed" "$MAX_RETRIES"
# Discord 알림 (Level 3로 escalation)
send_discord_notification "⚠️ **Level 2 Health Check 실패**\n\nGateway를 ${MAX_RETRIES}회 재시작했으나 복구 실패.\n${ESCALATION_WAIT}초 후 Level 3 (Claude Emergency Recovery)로 escalation합니다.\n\n현재 시각: $(date '+%Y-%m-%d %H:%M:%S')"
# 5분 대기 후 최종 검증
sleep "$ESCALATION_WAIT"
if ! check_http; then
escalate_to_level3
else
log "✅ Gateway recovered during waiting period"
# Discord 알림 (대기 중 복구됨)
send_discord_notification "✅ **Gateway 자동 복구됨**\n\n${ESCALATION_WAIT}초 대기 중 Gateway가 스스로 복구되었습니다.\nLevel 3 Emergency Recovery는 실행하지 않습니다."
record_metric "recovery" "self_healed" 0
fi
else
log "✅ Gateway healthy"
record_metric "health_check" "healthy" 0
fi
log "=== Health Check Completed ==="
}
# Run main function
main