AI Newsletter Digest improvements: fixed QP soft line break decoding, URL extraction, and content cleaning

skills/youtube-full/SKILL.md

@@ -0,0 +1,209 @@
+---
+name: youtube-full
+description: Complete YouTube toolkit — transcripts, search, channels, playlists, and metadata all in one skill. Use when you need comprehensive YouTube access, want to search and then get transcripts, browse channel content, work with playlists, or need the full suite of YouTube data endpoints. The all-in-one YouTube skill for agents.
+homepage: https://transcriptapi.com
+user-invocable: true
+metadata: {"openclaw":{"emoji":"🎯","requires":{"env":["TRANSCRIPT_API_KEY"],"bins":["node"],"config":["~/.openclaw/openclaw.json"]},"primaryEnv":"TRANSCRIPT_API_KEY"}}
+---
+
+# YouTube Full
+
+Complete YouTube toolkit via [TranscriptAPI.com](https://transcriptapi.com). Everything in one skill.
+
+## Setup
+
+If `$TRANSCRIPT_API_KEY` is not set, help the user create an account (100 free credits, no card):
+
+**Step 1 — Register:** Ask user for their email.
+
+```bash
+node ./scripts/tapi-auth.js register --email USER_EMAIL
+```
+
+→ OTP sent to email. Ask user: _"Check your email for a 6-digit verification code."_
+
+**Step 2 — Verify:** Once user provides the OTP:
+
+```bash
+node ./scripts/tapi-auth.js verify --token TOKEN_FROM_STEP_1 --otp CODE
+```
+
+> API key saved to `~/.openclaw/openclaw.json`. See **File Writes** below for details. Existing file is backed up before modification.
+
+Manual option: [transcriptapi.com/signup](https://transcriptapi.com/signup) → Dashboard → API Keys.
+
+## File Writes
+
+The verify and save-key commands save the API key to `~/.openclaw/openclaw.json` (sets `skills.entries.transcriptapi.apiKey` and `enabled: true`). **Existing file is backed up to `~/.openclaw/openclaw.json.bak` before modification.**
+
+To use the API key in terminal/CLI outside the agent, add to your shell profile manually:
+`export TRANSCRIPT_API_KEY=<your-key>`
+
+## API Reference
+
+Full OpenAPI spec: [transcriptapi.com/openapi.json](https://transcriptapi.com/openapi.json) — consult this for the latest parameters and schemas.
+
+## Transcript — 1 credit
+
+```bash
+curl -s "https://transcriptapi.com/api/v2/youtube/transcript\
+?video_url=VIDEO_URL&format=text&include_timestamp=true&send_metadata=true" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+| Param               | Required | Default | Values                          |
+| ------------------- | -------- | ------- | ------------------------------- |
+| `video_url`         | yes      | —       | YouTube URL or 11-char video ID |
+| `format`            | no       | `json`  | `json`, `text`                  |
+| `include_timestamp` | no       | `true`  | `true`, `false`                 |
+| `send_metadata`     | no       | `false` | `true`, `false`                 |
+
+**Response** (`format=json`):
+
+```json
+{
+  "video_id": "dQw4w9WgXcQ",
+  "language": "en",
+  "transcript": [{ "text": "...", "start": 18.0, "duration": 3.5 }],
+  "metadata": { "title": "...", "author_name": "...", "author_url": "..." }
+}
+```
+
+## Search — 1 credit
+
+```bash
+# Videos
+curl -s "https://transcriptapi.com/api/v2/youtube/search?q=QUERY&type=video&limit=20" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+
+# Channels
+curl -s "https://transcriptapi.com/api/v2/youtube/search?q=QUERY&type=channel&limit=10" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+| Param   | Required | Default | Validation         |
+| ------- | -------- | ------- | ------------------ |
+| `q`     | yes      | —       | 1-200 chars        |
+| `type`  | no       | `video` | `video`, `channel` |
+| `limit` | no       | `20`    | 1-50               |
+
+## Channels
+
+All channel endpoints accept `channel` — an `@handle`, channel URL, or `UC...` channel ID. No need to resolve first.
+
+### Resolve handle — FREE
+
+```bash
+curl -s "https://transcriptapi.com/api/v2/youtube/channel/resolve?input=@TED" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+Response: `{"channel_id": "UC...", "resolved_from": "@TED"}`
+
+### Latest 15 videos — FREE
+
+```bash
+curl -s "https://transcriptapi.com/api/v2/youtube/channel/latest?channel=@TED" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+Returns exact `viewCount` and ISO `published` timestamps.
+
+### All channel videos — 1 credit/page
+
+```bash
+# First page (100 videos)
+curl -s "https://transcriptapi.com/api/v2/youtube/channel/videos?channel=@NASA" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+
+# Next pages
+curl -s "https://transcriptapi.com/api/v2/youtube/channel/videos?continuation=TOKEN" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+Provide exactly one of `channel` or `continuation`. Response includes `continuation_token` and `has_more`.
+
+### Search within channel — 1 credit
+
+```bash
+curl -s "https://transcriptapi.com/api/v2/youtube/channel/search\
+?channel=@TED&q=QUERY&limit=30" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+## Playlists — 1 credit/page
+
+Accepts `playlist` — a YouTube playlist URL or playlist ID.
+
+```bash
+# First page
+curl -s "https://transcriptapi.com/api/v2/youtube/playlist/videos?playlist=PL_ID" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+
+# Next pages
+curl -s "https://transcriptapi.com/api/v2/youtube/playlist/videos?continuation=TOKEN" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+Valid ID prefixes: `PL`, `UU`, `LL`, `FL`, `OL`. Response includes `playlist_info`, `results`, `continuation_token`, `has_more`.
+
+## Credit Costs
+
+| Endpoint        | Cost     |
+| --------------- | -------- |
+| transcript      | 1        |
+| search          | 1        |
+| channel/resolve | **free** |
+| channel/latest  | **free** |
+| channel/videos  | 1/page   |
+| channel/search  | 1        |
+| playlist/videos | 1/page   |
+
+## Validation Rules
+
+| Field      | Rule                                                    |
+| ---------- | ------------------------------------------------------- |
+| `channel`  | `@handle`, channel URL, or `UC...` ID                   |
+| `playlist` | Playlist URL or ID (`PL`/`UU`/`LL`/`FL`/`OL` prefix)   |
+| `q`        | 1-200 chars                                             |
+| `limit`    | 1-50                                                    |
+
+## Errors
+
+| Code | Meaning          | Action                                |
+| ---- | ---------------- | ------------------------------------- |
+| 401  | Bad API key      | Check key                             |
+| 402  | No credits       | transcriptapi.com/billing             |
+| 404  | Not found        | Resource doesn't exist or no captions |
+| 408  | Timeout          | Retry once after 2s                   |
+| 422  | Validation error | Check param format                    |
+| 429  | Rate limited     | Wait, respect Retry-After             |
+
+## Typical Workflows
+
+**Research workflow:** search → pick videos → fetch transcripts
+
+```bash
+# 1. Search
+curl -s "https://transcriptapi.com/api/v2/youtube/search\
+?q=machine+learning+explained&limit=5" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+# 2. Transcript
+curl -s "https://transcriptapi.com/api/v2/youtube/transcript\
+?video_url=VIDEO_ID&format=text&include_timestamp=true&send_metadata=true" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+**Channel monitoring:** latest (free) → transcript
+
+```bash
+# 1. Latest uploads (free — pass @handle directly)
+curl -s "https://transcriptapi.com/api/v2/youtube/channel/latest?channel=@TED" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+# 2. Transcript of latest
+curl -s "https://transcriptapi.com/api/v2/youtube/transcript\
+?video_url=VIDEO_ID&format=text&include_timestamp=true&send_metadata=true" \
+  -H "Authorization: Bearer $TRANSCRIPT_API_KEY"
+```
+
+Free tier: 100 credits, 300 req/min. Starter ($5/mo): 1,000 credits.

skills/youtube-full/_meta.json

@@ -0,0 +1,6 @@
+{
+  "ownerId": "kn76tn13s33tbyv3x1p4thbmv5809t0m",
+  "slug": "youtube-full",
+  "version": "1.4.1",
+  "publishedAt": 1770817069239
+}

skills/youtube-full/scripts/tapi-auth.js

@@ -0,0 +1,496 @@
+#!/usr/bin/env node
+
+// ============================================================================
+// TranscriptAPI CLI — Passwordless Account Setup (ClawHub Edition)
+//
+// Minimal version for ClawHub registry. Only writes to ~/.openclaw/openclaw.json.
+// For shell RC writes, use the standard version in skills/*/scripts/tapi-auth.js.
+//
+// Authentication flow:
+//   1. User provides email → server creates account and returns a short-lived
+//      session token (JWT, expires in ~30 min). No password is involved.
+//   2. Server sends a one-time 6-digit verification code to the email.
+//   3. User provides the code → server verifies and returns an API key.
+//   4. API key is saved to ~/.openclaw/openclaw.json for agent runtime access.
+//
+// Source: https://transcriptapi.com | Docs: https://docs.transcriptapi.com
+// ============================================================================
+
+const VERSION = "3.0.0";
+const BASE_URL = "https://transcriptapi.com/api/auth";
+
+// ============================================================================
+// Utilities
+// ============================================================================
+
+const fs = require("fs");
+const path = require("path");
+const os = require("os");
+
+function parseArgs(args) {
+  const result = { _: [] };
+  for (let i = 0; i < args.length; i++) {
+    const arg = args[i];
+    if (arg.startsWith("--")) {
+      const key = arg.slice(2);
+      const next = args[i + 1];
+      if (next && !next.startsWith("--")) {
+        result[key] = next;
+        i++;
+      } else {
+        result[key] = true;
+      }
+    } else {
+      result._.push(arg);
+    }
+  }
+  return result;
+}
+
+function isHumanMode(args) {
+  return !!args.human;
+}
+
+function err(msg, humanMode = false) {
+  if (humanMode) {
+    console.error(`Error: ${msg}`);
+  } else {
+    console.error(JSON.stringify({ error: msg }));
+  }
+  process.exit(1);
+}
+
+function out(msg, humanMode = false, data = null) {
+  if (humanMode) {
+    console.log(msg);
+  } else {
+    console.log(JSON.stringify(data || { message: msg }));
+  }
+}
+
+async function httpRequest(url, options = {}) {
+  const response = await fetch(url, options);
+  let body;
+  const contentType = response.headers.get("content-type") || "";
+  if (contentType.includes("application/json")) {
+    body = await response.json();
+  } else {
+    body = await response.text();
+  }
+  return { status: response.status, ok: response.ok, body };
+}
+
+// ============================================================================
+// API Functions
+// ============================================================================
+
+async function registerCli(email, name) {
+  const payload = { email };
+  if (name) payload.name = name;
+
+  const res = await httpRequest(`${BASE_URL}/register-cli`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify(payload),
+  });
+
+  if (!res.ok) {
+    if (res.status === 409) {
+      throw new Error("Account already exists with this email");
+    }
+    const msg = res.body?.detail || res.body?.message || JSON.stringify(res.body);
+    throw new Error(`Registration failed: ${msg}`);
+  }
+
+  return res.body;
+}
+
+async function verifyCli(sessionToken, otp) {
+  const res = await httpRequest(`${BASE_URL}/verify-cli`, {
+    method: "POST",
+    headers: {
+      Authorization: `Bearer ${sessionToken}`,
+      "Content-Type": "application/json",
+    },
+    body: JSON.stringify({ otp }),
+  });
+
+  if (!res.ok) {
+    const msg = res.body?.detail || res.body?.message || "Verification failed";
+    throw new Error(msg);
+  }
+
+  return res.body;
+}
+
+async function getApiKeys(token) {
+  const res = await httpRequest(`${BASE_URL}/api-keys`, {
+    method: "GET",
+    headers: { Authorization: `Bearer ${token}` },
+  });
+
+  if (!res.ok) {
+    const msg = res.body?.detail || res.body?.message || "Failed to get API keys";
+    throw new Error(msg);
+  }
+
+  return res.body;
+}
+
+async function createApiKey(token, name = "default") {
+  const res = await httpRequest(`${BASE_URL}/api-keys`, {
+    method: "POST",
+    headers: {
+      Authorization: `Bearer ${token}`,
+      "Content-Type": "application/json",
+    },
+    body: JSON.stringify({ name }),
+  });
+
+  if (!res.ok) {
+    const msg = res.body?.detail || res.body?.message || "Failed to create API key";
+    throw new Error(msg);
+  }
+
+  return res.body;
+}
+
+async function getMe(token) {
+  const res = await httpRequest(`${BASE_URL}/me`, {
+    method: "GET",
+    headers: { Authorization: `Bearer ${token}` },
+  });
+
+  if (!res.ok) {
+    const msg = res.body?.detail || res.body?.message || "Failed to get user info";
+    throw new Error(msg);
+  }
+
+  return res.body;
+}
+
+async function getEmailVerificationStatus(token) {
+  const res = await httpRequest(`${BASE_URL}/email-verification-status`, {
+    method: "GET",
+    headers: { Authorization: `Bearer ${token}` },
+  });
+
+  if (!res.ok) {
+    const msg = res.body?.detail || res.body?.message || "Failed to get verification status";
+    throw new Error(msg);
+  }
+
+  return res.body;
+}
+
+// ============================================================================
+// File System Helpers — OpenClaw config only
+// ============================================================================
+
+function ensureDir(dir) {
+  if (!fs.existsSync(dir)) {
+    fs.mkdirSync(dir, { recursive: true });
+  }
+}
+
+function backupFile(filePath) {
+  if (fs.existsSync(filePath)) {
+    const backupPath = filePath + ".bak";
+    fs.copyFileSync(filePath, backupPath);
+    return backupPath;
+  }
+  return null;
+}
+
+// Save API key to ~/.openclaw/openclaw.json only.
+// Returns { files, warnings }.
+function saveApiKeyToConfigs(key) {
+  const home = os.homedir();
+  const filesWritten = [];
+  const warnings = [];
+
+  const openclawConfigPath = path.join(home, ".openclaw", "openclaw.json");
+
+  try {
+    ensureDir(path.join(home, ".openclaw"));
+    backupFile(openclawConfigPath);
+
+    let config = {};
+    if (fs.existsSync(openclawConfigPath)) {
+      const configContent = fs.readFileSync(openclawConfigPath, "utf8");
+      config = JSON.parse(configContent);
+    }
+
+    if (!config.skills) config.skills = {};
+    if (!config.skills.entries) config.skills.entries = {};
+    if (!config.skills.entries.transcriptapi) {
+      config.skills.entries.transcriptapi = {};
+    }
+    config.skills.entries.transcriptapi.apiKey = key;
+    config.skills.entries.transcriptapi.enabled = true;
+
+    fs.writeFileSync(openclawConfigPath, JSON.stringify(config, null, 2));
+    filesWritten.push({ path: openclawConfigPath, action: "updated", type: "openclaw-config" });
+  } catch (e) {
+    warnings.push(`Could not update ${openclawConfigPath}: ${e.message}`);
+  }
+
+  return { files: filesWritten, warnings };
+}
+
+// ============================================================================
+// Resolve a session token
+// ============================================================================
+
+async function resolveToken(args, humanMode) {
+  if (args.token) {
+    return args.token;
+  }
+  return null;
+}
+
+// ============================================================================
+// Commands
+// ============================================================================
+
+async function cmdRegister(args) {
+  const human = isHumanMode(args);
+  const email = args.email;
+  const name = args.name;
+
+  if (!email) err("--email is required", human);
+
+  const tempDomains = ["tempmail", "guerrilla", "10minute", "throwaway", "mailinator", "temp-mail", "fakeinbox", "trashmail"];
+  const emailLower = email.toLowerCase();
+  if (tempDomains.some(d => emailLower.includes(d))) {
+    err("Temporary/disposable emails are not allowed. Please use a real email address.", human);
+  }
+
+  try {
+    const result = await registerCli(email, name);
+    const sessionToken = result.access_token;
+
+    if (human) {
+      console.log(`\n  Account created. Verification code sent to ${email}.`);
+      console.log(`\n  Ask user: "Check your email for a 6-digit verification code."`);
+      console.log(`\n  Then run: node tapi-auth.js verify --token ${sessionToken} --otp CODE`);
+    } else {
+      out("", false, {
+        success: true,
+        email,
+        access_token: sessionToken,
+        access_token_note: "Short-lived server session token for the verify step. Not stored.",
+        next_step: "verify",
+        action_required: "ask_user_for_otp",
+        user_prompt: `Check your email (${email}) for a 6-digit verification code.`,
+        next_command: `node ./scripts/tapi-auth.js verify --token ${sessionToken} --otp <CODE>`
+      });
+    }
+  } catch (e) {
+    err(e.message, human);
+  }
+}
+
+async function cmdVerify(args) {
+  const human = isHumanMode(args);
+  const otp = args.otp;
+
+  const token = await resolveToken(args, human);
+  if (!token) err("--token is required", human);
+  if (!otp) err("--otp is required", human);
+
+  try {
+    const result = await verifyCli(token, otp);
+    const keyValue = result.api_key;
+
+    const saved = saveApiKeyToConfigs(keyValue);
+
+    if (human) {
+      console.log(`\n  Email verified!`);
+      console.log(`\n  API Key: ${keyValue}`);
+      console.log(`\n  Key saved to:`);
+      saved.files.forEach((f) => console.log(`    ${f.path}`));
+      if (saved.warnings.length > 0) {
+        console.log(`\n  Warnings:`);
+        saved.warnings.forEach((w) => console.log(`    ${w}`));
+      }
+      console.log(`\n  To use in terminal/CLI, add to your shell profile:`);
+      console.log(`    export TRANSCRIPT_API_KEY=${keyValue}`);
+    } else {
+      out("", false, {
+        success: true,
+        verified: true,
+        api_key: keyValue,
+        saved: { files: saved.files, warnings: saved.warnings },
+        manual_export: `export TRANSCRIPT_API_KEY=${keyValue}`,
+      });
+    }
+  } catch (e) {
+    err(e.message, human);
+  }
+}
+
+async function cmdGetKey(args) {
+  const human = isHumanMode(args);
+
+  const token = await resolveToken(args, human);
+  if (!token) err("--token is required", human);
+
+  try {
+    let keys = await getApiKeys(token);
+    let activeKey = keys.find((k) => k.is_active);
+
+    if (!activeKey) {
+      const newKey = await createApiKey(token);
+      activeKey = newKey;
+    }
+
+    const keyValue = activeKey.key;
+    out(keyValue, human, { api_key: keyValue });
+  } catch (e) {
+    err(e.message, human);
+  }
+}
+
+async function cmdSaveKey(args) {
+  const human = isHumanMode(args);
+  const key = args.key;
+
+  if (!key) err("--key is required", human);
+  if (!key.startsWith("sk_")) err("Key should start with sk_", human);
+
+  try {
+    const saved = saveApiKeyToConfigs(key);
+
+    if (human) {
+      console.log("API key saved:\n");
+      saved.files.forEach((f) => console.log(`  ${f.path}`));
+      if (saved.warnings.length > 0) {
+        console.log("\n  Warnings:");
+        saved.warnings.forEach((w) => console.log(`    ${w}`));
+      }
+      console.log(`\n  To use in terminal/CLI, add to your shell profile:`);
+      console.log(`    export TRANSCRIPT_API_KEY=${key}`);
+    } else {
+      out("", false, {
+        success: true,
+        files: saved.files,
+        warnings: saved.warnings,
+        manual_export: `export TRANSCRIPT_API_KEY=${key}`,
+      });
+    }
+  } catch (e) {
+    err(e.message, human);
+  }
+}
+
+async function cmdStatus(args) {
+  const human = isHumanMode(args);
+
+  const token = await resolveToken(args, human);
+  if (!token) err("--token is required", human);
+
+  try {
+    const me = await getMe(token);
+    const keys = await getApiKeys(token);
+    let verificationStatus;
+    try {
+      verificationStatus = await getEmailVerificationStatus(token);
+    } catch {
+      verificationStatus = { verified: me.is_verified || false };
+    }
+
+    const activeKeys = keys.filter((k) => k.is_active);
+
+    if (human) {
+      console.log("Account Status");
+      console.log("==============");
+      console.log(`Email:    ${me.email}`);
+      console.log(`Name:     ${me.name || "(not set)"}`);
+      console.log(`Verified: ${me.is_verified ? "Yes" : "No"}`);
+      console.log(`API Keys: ${keys.length} total, ${activeKeys.length} active`);
+    } else {
+      out("", false, {
+        email: me.email,
+        name: me.name,
+        is_verified: me.is_verified,
+        verification_status: verificationStatus,
+        api_keys_count: keys.length,
+        active_keys_count: activeKeys.length,
+      });
+    }
+  } catch (e) {
+    err(e.message, human);
+  }
+}
+
+function cmdHelp() {
+  console.log(`
+tapi-auth.js v${VERSION} - TranscriptAPI Account Setup (ClawHub Edition)
+
+  Creates a TranscriptAPI account and sets up an API key. No passwords
+  are involved — the server sends a one-time verification code to your
+  email, and once verified, the API key is saved to the OpenClaw config
+  (~/.openclaw/openclaw.json) for agent runtime access.
+
+  For terminal/CLI usage, manually add to your shell profile:
+    export TRANSCRIPT_API_KEY=<your-key>
+
+USAGE:
+
+  1. Register:  node ./scripts/tapi-auth.js register --email USER_EMAIL
+     → Sends a 6-digit code to your email. Returns a session token.
+     → Ask user: "Check your email for a 6-digit verification code."
+
+  2. Verify:    node ./scripts/tapi-auth.js verify --token TOKEN --otp CODE
+     → Verifies the code, saves API key to OpenClaw config. Done.
+
+COMMANDS:
+  register    Create account, sends verification code   --email (required), --name
+  verify      Verify code, auto-save API key            --token, --otp
+  get-key     Retrieve existing API key                 --token
+  save-key    Manually save an API key                  --key
+  status      Check account status                      --token
+
+FLAGS:
+  --human     Human-readable output (default is JSON)
+`);
+}
+
+// ============================================================================
+// Main
+// ============================================================================
+
+async function main() {
+  const args = parseArgs(process.argv.slice(2));
+  const command = args._[0];
+
+  switch (command) {
+    case "register":
+      await cmdRegister(args);
+      break;
+    case "verify":
+      await cmdVerify(args);
+      break;
+    case "get-key":
+      await cmdGetKey(args);
+      break;
+    case "save-key":
+      await cmdSaveKey(args);
+      break;
+    case "status":
+      await cmdStatus(args);
+      break;
+    case "help":
+    case undefined:
+      cmdHelp();
+      break;
+    default:
+      err(`Unknown command: ${command}. Run 'node tapi-auth.js help' for usage.`);
+  }
+}
+
+main().catch((e) => {
+  console.error(JSON.stringify({ error: e.message }));
+  process.exit(1);
+});