AI Newsletter Digest improvements: fixed QP soft line break decoding, URL extraction, and content cleaning

This commit is contained in:
Krilly
2026-03-04 13:29:22 +00:00
parent 29a98137a7
commit 57dd294675
13706 changed files with 2114953 additions and 237629 deletions
@@ -0,0 +1,146 @@
# OpenClaw Hardening: Lessons Learned
**Date:** 2026-02-08
**Source:** Real-world hardening session with OpenClaw 2026.2.6-3
## What We Learned The Hard Way
### ❌ Config Fields That DON'T Work
These fields cause `Config validation failed` errors:
```json
{
"logging": {
"redactSensitive": "all", // ❌ Invalid input
"enabled": true // ❌ Unrecognized key
},
"agents": {
"defaults": {
"tools": { // ❌ Unrecognized key
"deny": ["exec", "browser", "cron", "process", "gateway"]
},
"sandbox": { // ❌ Unrecognized key (in defaults)
"mode": "all",
"scope": "agent"
}
}
}
}
```
**Why?** OpenClaw's config schema doesn't support these fields. The original hardening guide was based on assumptions, not tested reality.
### ✅ What Actually Works
1. **File Permissions** (tested, works):
```bash
chmod 700 ~/.openclaw
chmod 600 ~/.openclaw/openclaw.json
chmod 700 ~/.openclaw/credentials
```
2. **Built-in Security Tools** (use these instead):
```bash
openclaw doctor --fix # Validates and fixes config
openclaw security audit --deep # Comprehensive security scan
openclaw health # Gateway and connection health
openclaw models status # Auth and model status
```
3. **Network Security** (check with):
```bash
netstat -an | grep 18789 | grep LISTEN
# Secure: 127.0.0.1 or ::1
# Exposed: 0.0.0.0
```
4. **Git Tracking** (rollback capability):
```bash
cd ~/.openclaw && git init
git add openclaw.json && git commit -m "config: baseline"
```
## Security Audit Results
After hardening a real installation:
```
Summary: 0 critical · 2 warn · 1 info
```
**Typical warnings (non-critical):**
- `gateway.trusted_proxies_missing` - OK if localhost-only
- `channels.discord.dm.scope_main_multiuser` - Optional session isolation
- `fs.credentials_dir.perms_readable` - Fixed with `chmod 700`
## The Reality: OpenClaw is Secure by Default
OpenClaw already provides:
- ✅ Localhost-only binding (`gateway.bind: "loopback"`)
- ✅ Token-based authentication required
- ✅ Secrets encrypted at rest
- ✅ OAuth flows with PKCE
- ✅ Session isolation
- ✅ CSRF protections
- ✅ Rate limiting
**Your job:** Verify these are working, lock down file permissions, maintain good operational security practices.
## Workflow That Works
```bash
# 1. Check network exposure
netstat -an | grep 18789 | grep LISTEN
# 2. Lock file permissions
chmod 700 ~/.openclaw
chmod 600 ~/.openclaw/openclaw.json
chmod 700 ~/.openclaw/credentials
# 3. Run security audit
openclaw security audit --deep
# 4. Fix any issues
openclaw doctor --fix
# 5. Git-track for rollback
cd ~/.openclaw && git init
git add openclaw.json && git commit -m "security: baseline config"
# 6. Verify health
openclaw health
openclaw models status
```
## Don't Waste Time On
- ❌ Manual config edits for tool restrictions (schema doesn't support)
- ❌ Manual logging config (not supported)
- ❌ Sandbox mode in defaults (not supported)
- ❌ Complex security configs (already built-in)
## Focus On
- ✅ File permissions (700/600)
- ✅ Network binding verification
- ✅ Regular security audits (`openclaw security audit --deep`)
- ✅ API key rotation (90-day cycle)
- ✅ Git-tracking config changes
- ✅ Operational security (dedicated accounts, separate machines)
## Command Reference
| Task | Command | Expected Result |
|------|---------|----------------|
| Check network | `netstat -an \| grep 18789` | 127.0.0.1 (not 0.0.0.0) |
| Validate config | `openclaw doctor --fix` | "Doctor complete." |
| Security scan | `openclaw security audit --deep` | 0 critical |
| Check health | `openclaw health` | "Discord: ok" |
| Auth status | `openclaw models status` | Lists auth providers |
## Key Insight
**Stop adding manual security configs. Start verifying built-in security.**
OpenClaw is secure by design. The hardening process is about verification and operational security, not configuration hacking.
@@ -0,0 +1,232 @@
# OpenClaw Hardening Guide
Based on the AI SAFE2 framework. Focuses on proven, working security measures.
## ⚠️ IMPORTANT: OpenClaw Security Reality
**OpenClaw already has strong security defaults built-in:**
- Secure authentication required by default
- Strong account/workspace isolation
- CSRF protections for state-changing requests
- Secrets encrypted at rest
- Private-by-default networking (localhost binding)
- Secure OAuth flows (state/PKCE)
**This guide helps you verify and enhance what's already there.**
## Prerequisites
- SSH access to server running OpenClaw (or local tmux session)
- Basic command line knowledge
## Step 1: Verify Network Security (2 min)
```bash
# Check current binding
netstat -an | grep -E "8080|18789|8888" | grep LISTEN
# SECURE if you see:
# tcp4 0 0 127.0.0.1.18789 *.* LISTEN
# tcp6 0 0 ::1.18789 *.* LISTEN
# EXPOSED if you see:
# tcp4 0 0 0.0.0.0.18789 *.* LISTEN ← BAD!
# If exposed, restart bound to localhost only:
pkill -f "openclaw gateway"
openclaw gateway --bind 127.0.0.1 --port 18789
# Access via SSH tunnel from laptop:
ssh -L 18789:127.0.0.1:18789 user@server
```
**Note:** OpenClaw's default `gateway.bind: "loopback"` config already binds to localhost. Most installations are secure by default.
## Step 2: Lock File Permissions (1 min)
```bash
chmod 700 ~/.openclaw
chmod 600 ~/.openclaw/openclaw.json
chmod 600 ~/.openclaw/*.log 2>/dev/null || true
chmod 600 ~/.openclaw/.env 2>/dev/null || true
chmod 700 ~/.openclaw/credentials 2>/dev/null || true
```
**Verify:**
```bash
ls -la ~/.openclaw/ | head -5
# Should show: drwx------ for .openclaw
# Should show: -rw------- for openclaw.json
```
## Step 3: Run Security Audit (2 min)
⚠️ **SKIP manual config edits for tool restrictions.** OpenClaw's config schema doesn't support the following fields:
- ❌ `logging.redactSensitive`
- ❌ `logging.enabled`
- ❌ `agents.defaults.tools`
- ❌ `agents.defaults.sandbox`
These fields will cause config validation errors. OpenClaw has built-in security controls that work differently.
**Instead, run the built-in security tools:**
```bash
# Validate config
openclaw doctor --fix
# Run deep security audit
openclaw security audit --deep
```
**Target result:** `0 critical · 0-3 warn · 1 info`
**Common warnings (non-critical):**
- `gateway.trusted_proxies_missing` - OK if localhost-only
- `channels.discord.dm.scope_main_multiuser` - Optional session isolation
- `fs.credentials_dir.perms_readable` - Fixed by Step 2
## Step 4: Rotate Secrets (3 min)
**Use OpenClaw's built-in authentication commands (safe, no shell modification):**
1. Generate new API keys from each provider console
2. Update via OpenClaw's secure method:
```bash
# Interactive authentication (recommended)
openclaw models auth paste-token
# Or use environment variables (set these in your terminal session)
export ZAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-..."
```
3. Delete old keys from provider console
**⚠️ IMPORTANT:** Do NOT modify shell startup files (`~/.bashrc`, `~/.zshrc`) directly. Instead:
- Use OpenClaw's `openclaw models auth` command for permanent storage
- Or set environment variables in your current terminal session only
**Verify auth status:**
```bash
openclaw models status
# Check "Auth overview" section
# Look for OAuth expiration dates
```
## Step 5: Git-Track Config for Rollback (2 min)
```bash
cd ~/.openclaw
# Initialize git if not already done
git init 2>/dev/null || true
# Create .gitignore
printf 'agents/*/sessions/\nagents/*/agent/*.jsonl\n*.log\n' > .gitignore
# Commit current config
git add .gitignore openclaw.json agents/*/agent/auth-profiles.json agents/*/agent/models.json
git commit -m "security: baseline hardened config"
# View commit history for rollback
git log --oneline
```
**To rollback:**
```bash
cd ~/.openclaw
git log --oneline # Find commit hash
git checkout <commit-hash> -- openclaw.json
openclaw doctor --fix # Validate after rollback
```
## Step 6: Optional Backups
**⚠️ MANUAL SETUP REQUIRED:** This skill does NOT automatically set up cron jobs or modify system schedules for security reasons.
If you want automated backups, manually create a backup script and schedule it yourself:
1. Create backup script manually: `~/backup-openclaw.sh`
2. Make it executable: `chmod +x ~/backup-openclaw.sh`
3. **Manually** add to crontab using `crontab -e`
**Example backup script content** (create this yourself):
```bash
#!/bin/bash
DATE=$(date +%Y%m%d)
mkdir -p ~/backups
tar -czf ~/backups/openclaw-$DATE.tar.gz ~/.openclaw/openclaw.json ~/.openclaw/agents/
find ~/backups/ -name "openclaw-*.tar.gz" -mtime +30 -delete
```
**This skill will NOT create or schedule this for you.** You must do this manually if desired.
## Optional: Isolate Discord DM Sessions
If you have multiple users DMing your bot, add to `openclaw.json`:
```json
{
"session": {
"dmScope": "per-channel-peer"
}
}
```
This prevents context leakage between different DM senders.
## Verification Checklist
After hardening, verify with:
```bash
# 1. Network security
netstat -an | grep 18789 | grep LISTEN
# Should show: 127.0.0.1 or ::1 (NOT 0.0.0.0)
# 2. File permissions
ls -la ~/.openclaw/ | head -5
# Should show: drwx------ for directory
# 3. Security audit
openclaw security audit --deep
# Target: 0 critical · 0-2 warn · 1 info
# 4. Config validity
openclaw doctor --fix
# Should complete without errors
# 5. Gateway health
openclaw health
# Should show: Discord: ok, Agents: main
```
## Safety Rules
1. **Human approval** for external comms (email, social media, purchases)
2. **Dedicated bot accounts** — never use personal credentials
3. **Separate machine** — don't run OpenClaw on your personal laptop with sensitive files
4. **Rotate keys** every 90 days minimum, immediately after any suspected compromise
## What OpenClaw Already Provides
You don't need to manually configure these - they're built-in:
✅ Secure authentication required by default
✅ Strong account/workspace isolation across all actions
✅ CSRF protections for state-changing requests
✅ Strict origin checks to block cross-site attacks
✅ WebSocket origin validation to prevent hijacking
✅ Rate limiting and abuse prevention on sensitive endpoints
✅ Secrets encrypted at rest
✅ Secrets transmitted securely and never logged
✅ Short-lived/rotating access credentials where applicable
✅ Private-by-default networking
✅ Tight allowlists for any browser-accessible control surfaces
✅ Secure OAuth flows (state/PKCE) for supported providers
✅ Security-focused HTTP headers (CSP, clickjacking, etc.)
✅ Least-privilege runtime (non-root) for services
**Your job:** Verify these are working, lock down file permissions, and maintain good operational security practices.
@@ -0,0 +1,142 @@
# Provider Configuration Guide
## Built-in Providers
These need only auth setup + model selection. No `models.providers` config required.
### Z.AI (GLM models)
```bash
openclaw onboard --auth-choice zai-api-key
openclaw models set zai/glm-4.7
openclaw models fallbacks add zai/glm-4.6
```
Config snippet:
```json
{ "agents": { "defaults": { "model": { "primary": "zai/glm-4.7" } } } }
```
Models: `glm-4.7`, `glm-4.6`
Auth: Bearer token via ZAI_API_KEY or auth-profiles
### Anthropic
```bash
openclaw onboard --auth-choice anthropic-api-key
openclaw models set anthropic/claude-opus-4-6
```
Models: `claude-opus-4-6`, `claude-sonnet-4-5`, `claude-haiku-4-5`
### OpenAI
```bash
openclaw models set openai/gpt-5.1
```
### OpenAI Codex (OAuth)
```bash
openclaw models auth login --provider openai-codex
openclaw models set openai-codex/gpt-5.3-codex
```
### OpenRouter
```bash
openclaw models set openrouter/<org>/<model>
```
Free models: use `openclaw models scan` to discover and rank.
### Ollama (local)
```bash
ollama pull llama3.3
openclaw models set ollama/llama3.3
```
Auto-detected at `http://127.0.0.1:11434/v1`.
## Custom Providers (OpenAI-compatible)
For NVIDIA NIM, LM Studio, vLLM, etc. — requires `models.providers` in openclaw.json.
Write config safely via python3 (never heredoc in tmux):
```python
python3 -c "
import json
# Read existing config
with open('/Users/<user>/.openclaw/openclaw.json') as f:
config = json.load(f)
# Add custom provider
config.setdefault('models', {})['mode'] = 'merge'
config['models'].setdefault('providers', {})['nvidia'] = {
'baseUrl': 'https://integrate.api.nvidia.com/v1',
'apiKey': '<key>',
'api': 'openai-completions',
'models': [{
'id': 'moonshotai/kimi-k2.5',
'name': 'Kimi K2.5 (NVIDIA NIM)',
'reasoning': True,
'input': ['text'],
'contextWindow': 131072,
'maxTokens': 8192
}]
}
# Write back
with open('/Users/<user>/.openclaw/openclaw.json', 'w') as f:
json.dump(config, f, indent=2)
"
```
### NVIDIA NIM (free tier)
- Base URL: `https://integrate.api.nvidia.com/v1`
- API: `openai-completions`
- Warning: Free tier is often congested (150+ queue). Not practical for agent workflows.
### LM Studio (local)
- Base URL: `http://localhost:1234/v1`
- API: `openai-completions`
## Planning vs Execution Pattern
Set stronger model as primary, lighter as fallback:
```bash
openclaw models set zai/glm-4.7 # Planning (stronger)
openclaw models fallbacks add zai/glm-4.6 # Execution (lighter)
```
Users switch in-session with `/model zai/glm-4.6`.
For dedicated agents with separate models:
```json
{
"agents": {
"list": [
{ "id": "planner", "model": "zai/glm-4.7", "workspace": "~/.openclaw/workspace-planner" },
{ "id": "executor", "model": "zai/glm-4.6", "workspace": "~/.openclaw/workspace-executor" }
]
}
}
```
## Common Model Reference
| Provider | Model IDs | Notes |
|----------|-----------|-------|
| zai | glm-4.7, glm-4.6 | Z.AI, Bearer auth, ~5s response |
| anthropic | claude-opus-4-6, claude-sonnet-4-5 | API key or CLI token |
| openai | gpt-5.1, gpt-5.2 | API key |
| openai-codex | gpt-5.3-codex | OAuth device flow |
| nvidia | moonshotai/kimi-k2.5 | Free but congested |
| openrouter | varies | Many free; use `openclaw models scan` |
| ollama | llama3.3, etc. | Local, no auth |
@@ -0,0 +1,87 @@
# Remote Connection Guide
⚠️ **PREREQUISITE**: This guide assumes you already have SSH access to your remote machine. If you don't, please set up SSH access manually before proceeding.
## Step 1: Determine Connection Method
Ask the user:
> How do you connect to your remote machine?
> 1. Tailscale (recommended — zero-config mesh VPN)
> 2. Direct SSH to a VPS (public IP)
> 3. Local network (same LAN)
## Step 2: Connect via Tailscale
If the user has Tailscale installed on both machines:
```bash
# Check Tailscale is running
tailscale status
# SSH to remote using Tailscale hostname
ssh <user>@<hostname>.tail<tailnet>.ts.net
```
**Note**: If Tailscale is not installed, please install it manually following the official Tailscale documentation: https://tailscale.com/download
## Step 2 (alt): Connect via Direct SSH
```bash
# Test connection (assumes SSH is already configured)
ssh <user>@<ip-address>
```
**Note**: This guide assumes SSH authentication is already configured on your system. We recommend using SSH key-based authentication for security, which you should set up manually outside of this skill.
## Step 3: Start tmux Session on Remote
```bash
# SSH in and start tmux
ssh <user>@<remote-address>
tmux new-session -s openclaw
# Or attach to existing
ssh <user>@<remote-address> -t 'tmux attach-session -t openclaw || tmux new-session -s openclaw'
```
## Step 4: Use tmux from Local Agent
Once the user has an SSH connection, interact via tmux from the local machine:
```bash
# If SSH session has tmux running locally that forwards to remote:
tmux send-keys -t <local-session> 'openclaw --version' Enter
sleep 2 && tmux capture-pane -t <local-session> -p -S -5
# If using SSH directly in tmux:
tmux send-keys -t <session> 'ssh <user>@<remote> "openclaw --version"' Enter
```
## Step 5: Verify OpenClaw Installation
```bash
which openclaw && openclaw --version
```
If not installed:
```bash
# macOS (Homebrew)
brew install openclaw
# Linux (npm)
npm install -g openclaw
# Verify
openclaw --version
```
## Troubleshooting
| Problem | Solution |
|---------|----------|
| Connection refused | Check SSH is running: `sudo systemctl status sshd` |
| Permission denied | Check key permissions: `chmod 600 ~/.ssh/id_ed25519` |
| Tailscale not connecting | Run `tailscale up --reset` on both machines |
| tmux not found | Install: `brew install tmux` (mac) or `apt install tmux` (linux) |
| ENOENT uv_cwd in tmux | Run `cd ~` first — previous cwd was deleted |