Heartbeat

<Why_This_Exists> Developers often miss slow-building problems: disk filling up, memory leaks in dev servers, failing CI pipelines they forgot about, new Sentry errors from last night's deploy, or calendar conflicts for upcoming meetings. Without proactive monitoring, these issues become emergencies. Heartbeat runs collectors on a schedule (or on-demand), evaluates against thresholds, and pushes alerts to Telegram so the developer knows about problems even when they are away from the terminal. It provides the "ops awareness" that solo developers lack. </Why_This_Exists>

<Execution_Policy>

• Run all 7 collectors in parallel for maximum speed (independent data sources)
• Timeout per collector: 15 seconds (prevent one slow collector from blocking the report)
• If a collector fails: log the error, mark that section as "unavailable", continue with others
• Max report frequency: no more than 1 full heartbeat per 5 minutes (prevent spam)
• Alert deduplication: same alert not re-sent within 30 minutes
• Critical alerts: always send to Telegram immediately
• Warning alerts: batch and send every 15 minutes or with next heartbeat
• Info level: include in report only, no Telegram notification </Execution_Policy>

Collector 1 - System Metrics:

• Bash: top -l 1 -s 0 | head -12 for CPU and memory
• Bash: df -h / for disk usage
• Bash: uptime for load average and uptime
• Bash: sysctl hw.memsize for total memory
• Output: CPU%, memory%, disk%, load average, uptime

Collector 2 - Dev Environment:

• Bash: node --version, npm --version, python3 --version
• Bash: git status --porcelain | wc -l for uncommitted changes
• Bash: npm test 2>&1 | tail -5 or project test command (if configured)
• Bash: npx tsc --noEmit 2>&1 | tail -10 for TypeScript errors
• Output: tool versions, uncommitted file count, test status, type errors

Collector 3 - GitHub CI:

• Bash: gh run list --limit 5 --json status,conclusion,name,createdAt
• Bash: gh pr list --json number,title,mergeable,reviews
• Output: recent CI run results, open PR status, merge conflicts

Collector 4 - Sentry Errors:

• Bash: curl -s -H "Authorization: Bearer $SENTRY_AUTH_TOKEN" "https://sentry.io/api/0/projects/$SENTRY_ORG/$SENTRY_PROJECT/issues/?query=is:unresolved&limit=5"
• Output: unresolved error count, top error titles, first/last seen

Collector 5 - Calendar:

• sc_osascript: query Calendar.app for today's events via AppleScript
• Script: tell application "Calendar" to get summary of events of calendar "Work" whose start date >= (current date) and start date <= ((current date) + 1 * days)
• Output: today's remaining events with times

Collector 6 - Process Health:

• Bash: ps aux | head -20 sorted by CPU usage
• Bash: lsof -i -P | grep LISTEN for open ports
• Bash: check if key services are running (node, docker, postgres, etc.)
• Output: top CPU processes, listening ports, service status

Collector 7 - Custom Collectors:

• Read user-defined collector scripts from ~/superclaw/collectors/
• Each script is a shell script that outputs JSON: {"status":"ok|warn|critical","metric":value,"message":"..."}
• Execute each script with 10-second timeout
• Output: custom metric results

2. Phase 2 - Evaluate Against Thresholds: Compare collected metrics to alert rules

   • Default thresholds (overridable in config):

   Metric	Warning	Critical
   CPU %	> 70%	> 90%
   Memory %	> 75%	> 90%
   Disk %	> 80%	> 95%
   Load Average	> 4.0	> 8.0
   Uncommitted Files	> 20	> 50
   Failed CI Runs	>= 1	>= 3
   Unresolved Sentry	>= 5	>= 20
   TypeScript Errors	>= 1	>= 10

   • Each metric evaluated independently
   • Overall status = worst individual status (any critical -> overall critical)

3. Phase 3 - Generate Report: Format results into a structured heartbeat report

   • Format:

   === SuperClaw Heartbeat ===
   Time: 2026-02-12 10:30:00
   Overall: OK | WARN | CRITICAL
   
   [System]
   CPU: 23% (ok)  |  Memory: 61% (ok)  |  Disk: 45% (ok)
   Load: 1.2  |  Uptime: 14d 3h
   
   [Dev Environment]
   Node: v22.1.0  |  TypeScript Errors: 0 (ok)
   Uncommitted: 3 files  |  Tests: passing
   
   [GitHub CI]
   Last 5 runs: 4 passed, 1 failed
   Open PRs: 2 (1 needs review)
   
   [Sentry]
   Unresolved: 3 issues (warn)
   Top: "TypeError: Cannot read property 'id' of undefined" (12 events)
   
   [Calendar]
   14:00 - Team standup (30min)
   16:00 - Design review (1hr)
   
   [Processes]
   Top CPU: node (8.2%), postgres (3.1%), docker (2.4%)
   Listening: :3000 (node), :5432 (postgres), :6379 (redis)
   
   [Alerts]
   WARN: 1 failed CI run on main branch
   WARN: 3 unresolved Sentry issues
   

4. Phase 4 - Alert on Critical/Warn: Send notifications for threshold violations

   • Critical: immediately send to Telegram via sc_send_message
   • Warning: include in report, send to Telegram if configured for warn-level
   • Info: include in report only
   • Alert message format: "[CRITICAL] CPU at 92% on hostname | Heartbeat 10:30"
   • Deduplicate: skip if same alert was sent within the last 30 minutes

5. Phase 5 - Send to Telegram (if configured):

   • Tool: sc_send_message with channel="telegram"
   • Full report sent as formatted text (truncated to 4096 chars if needed)
   • Critical alerts sent as separate urgent messages
   • Respect quiet hours if configured (no non-critical alerts between 22:00-08:00)

6. Phase 6 - Store Results for Trending:

   • Write heartbeat result to ~/superclaw/heartbeat/history/YYYY-MM-DD-HH-mm.json
   • Keep last 7 days of history (auto-prune older files)
   • Format: JSON with all collector results, thresholds, alert states
   • Enables "show me CPU trend for the last week" queries

7. Phase 7 - Schedule Next Run (if periodic monitoring requested):

   • Add to system crontab with the cron expression (e.g., "*/30 * * * *" for every 30 minutes)
   • Verify the job was added via crontab -l </Steps>

<Tool_Usage> Messaging (2 tools):

• sc_telegram_status -- Check if Telegram bot is configured before attempting alerts; no params
• sc_send_message -- Send heartbeat report or alert to Telegram; params: channel (string, "telegram"), text (string, formatted report/alert)

Scheduling:

• Bash -- Manage system crontab via crontab -l, crontab -e, crontab <file> for scheduling periodic heartbeat runs

System Data (via Bash):

• top -l 1 -s 0 | head -12 -- CPU and memory usage snapshot
• df -h / -- Disk usage for root volume
• uptime -- System uptime and load averages
• ps aux --sort=-%cpu | head -10 -- Top CPU-consuming processes
• lsof -i -P | grep LISTEN -- Listening network ports

GitHub Data (via Bash with gh CLI):

• gh run list --limit 5 --json status,conclusion,name,createdAt -- Recent CI runs
• gh pr list --json number,title,mergeable -- Open pull requests

Calendar Data (via SuperClaw):

• sc_osascript -- Query Calendar.app via AppleScript for today's events

Notification Fallback:

• sc_notify -- Send macOS notification if Telegram is unavailable; params: title, message </Tool_Usage>

<Escalation_And_Stop_Conditions>

• Stop if all collectors fail -- system may be in a broken state, inform user to check manually
• Stop if crontab modification fails repeatedly -- check permissions
• Escalate if critical alerts persist across 3+ consecutive heartbeats -- problem is not self-resolving
• Escalate if disk usage is above 95% -- immediate user action required
• Escalate if GitHub CI has been failing for more than 24 hours -- may indicate a broken main branch
• Warn if Sentry collector fails with auth error -- token may have expired
• Warn if Calendar collector returns permission error -- Automation permission needed for Calendar.app
• Fallback to sc_notify (macOS notification) if Telegram is unreachable for alerts </Escalation_And_Stop_Conditions>

<Final_Checklist>

• All enabled collectors executed (failed ones marked as "unavailable", not blocking)
• Thresholds evaluated for every collected metric
• Report formatted with clear status indicators (ok/warn/critical)
• Critical alerts sent to Telegram immediately
• Warning alerts included in report and sent if configured
• Results stored to history file for trending
• Cron schedule verified if periodic monitoring was requested
• No collector timeout exceeded 15 seconds </Final_Checklist>

Heartbeat configuration in ~/superclaw/superclaw.json: