Service Monitoring Setup

Monitoring Architecture Decision Tree

Is the workload ephemeral (runs and exits)?
├─ YES → Use Healthchecks.io (Dead Man's Switch)
│   ├─ Job pings on success
│   ├─ No ping within timeout → Alert
│   └─ Free tier: 20 checks, user-defined timeouts
│
└─ NO → Is it a persistent HTTP endpoint?
    └─ YES → Use UptimeRobot (HTTP polling)
        ├─ Service pings endpoint every N minutes
        ├─ No response → Alert
        └─ Free tier: 50 monitors, 5-minute intervals

Recommended Architecture: Use BOTH services for dual-pipeline monitoring

• Healthchecks.io: Cloud Run Jobs (data collection jobs)
• UptimeRobot: VM HTTP endpoints (persistent services)

Quick Start

Healthchecks.io (Dead Man's Switch)

# /// script
# dependencies = ["requests"]
# ///

import os
from scripts.healthchecks_client import HealthchecksClient

# Get API key from Doppler
api_key = os.popen(
    "doppler secrets get HEALTHCHECKS_API_KEY --project claude-config --config dev --plain"
).read().strip()

client = HealthchecksClient(api_key)

# Create check for Cloud Run Job
result = client.create_check(
    name="Ethereum Collector Job",
    timeout=7200,  # 2 hours
    grace=600,     # 10 minutes grace period
    tags="production ethereum",
    channels="*"   # All notification channels
)

ping_url = result["ping_url"]
print(f"Add to Cloud Run Job environment: HEALTHCHECK_PING_URL={ping_url}")

# In your Cloud Run Job script:
# import requests
# requests.get(os.getenv("HEALTHCHECK_PING_URL"))  # On success
# requests.get(f"{os.getenv('HEALTHCHECK_PING_URL')}/fail")  # On failure

UptimeRobot (HTTP Monitoring)

# /// script
# dependencies = ["requests"]
# ///

import os
from scripts.uptimerobot_client import UptimeRobotClient

# Get API key from Doppler
api_key = os.popen(
    "doppler secrets get UPTIMEROBOT_API_KEY --project claude-config --config dev --plain"
).read().strip()

client = UptimeRobotClient(api_key)

# Create HTTP monitor for VM service
result = client.create_monitor(
    friendly_name="Production API Endpoint",
    url="https://your-vm-ip:8000/health",
    type=1,  # HTTP(S)
    interval=300,  # 5 minutes (free tier)
    alert_contacts=client.get_pushover_contact_id()  # Type 9 Pushover contact
)

print(f"Monitor created: {result['monitor']['id']}")

UptimeRobot Operations

Authentication

UptimeRobot API v2 uses POST data authentication:

def _request(self, endpoint: str, data: Dict) -> Dict:
    data["api_key"] = self.api_key
    data["format"] = "json"
    response = requests.post(f"{self.base_url}/{endpoint}", data=data)
    response.raise_for_status()
    result = response.json()
    if result.get("stat") != "ok":
        raise Exception(f"UptimeRobot API error: {result}")
    return result

Free Tier Capabilities

• Monitors: 50 HTTP/HTTPS monitors
• Check Interval: 5 minutes (minimum)
• Alert Contacts: Unlimited (email, Pushover, Slack, webhook)
• Limitations:
  • Heartbeat monitoring requires Pro ($7/mo)
  • Rate limiting: 10 req/min (429 error with Retry-After: 47s header)
  • Use exponential backoff for bulk operations

Common Operations

List Monitors:

monitors = client.get_monitors()
for monitor in monitors:
    print(f"{monitor['friendly_name']}: {monitor['url']} - {monitor['status']}")

Create Monitor:

result = client.create_monitor(
    friendly_name="API Health Check",
    url="https://api.example.com/health",
    type=1,  # HTTP(S)
    interval=300,  # 5 minutes
    alert_contacts=pushover_id
)

Delete Monitor:

client.delete_monitor(monitor_id="801762241")

Get Pushover Contact ID:

pushover_id = client.get_pushover_contact_id()
if not pushover_id:
    print("⚠️ Pushover not configured - see Pushover Integration Setup")

Monitor Types

• 1 = HTTP(S) - Checks endpoint response
• 2 = Keyword - Checks for specific text in response
• 3 = Ping - ICMP ping
• 4 = Port - TCP port check

Healthchecks.io Operations

Authentication

Healthchecks.io API v3 uses X-Api-Key header authentication:

headers = {
    "X-Api-Key": api_key,
    "Content-Type": "application/json"
}
response = requests.get(f"{base_url}/checks/", headers=headers)

Free Tier Capabilities

• Checks: 20 checks
• Timeout: User-defined (any duration)
• Grace Period: User-defined buffer before alert
• Integrations: Email, Pushover, Slack, Discord, webhooks (all free)
• Ping Mechanism: Simple HTTP GET to unique ping URL

Common Operations

List Checks:

checks = client.get_checks()
for check in checks:
    print(f"{check['name']}: {check['status']} - {check['ping_url']}")

Create Check:

result = client.create_check(
    name="Daily Backup Job",
    timeout=86400,  # 24 hours
    grace=3600,     # 1 hour grace
    tags="backup production",
    channels=pushover_id  # Or "*" for all channels
)
ping_url = result["ping_url"]

Ping Check (Success):

# From your job/script
import requests
requests.get(ping_url)

Ping Check (Failure):

requests.get(f"{ping_url}/fail")

Delete Check:

client.delete_check(check_uuid="6a991157-552d-4c2c-b972-d43de0a96bff")

Dead Man's Switch Pattern

Perfect for ephemeral workloads (Cloud Run Jobs, cron jobs):

Job Lifecycle:
1. Job starts
2. Job executes work
3. Job pings Healthchecks.io on success
4. If no ping within timeout → Alert

Advantages:
- No always-on endpoint needed
- Works with ephemeral infrastructure
- Simple integration (one HTTP GET)
- Catches job crashes, hangs, or scheduling failures

Pushover Integration Setup

Important: Pushover integration must be configured via web UI first. API can only list and assign existing integrations, not create them.

Prerequisites

1. Create Pushover account at https://pushover.net (30-day trial, then $5 one-time)
2. Install Pushover app on mobile device
3. Note your User Key from Pushover dashboard

UptimeRobot Pushover Setup

1. Go to https://uptimerobot.com/dashboard
2. Click "My Settings" → "Alert Contacts"
3. Click "Add Alert Contact"
4. Select "Pushover" (may appear as type 9 in API)
5. Enter your Pushover User Key
6. Complete verification
7. Verify with: client.get_pushover_contact_id() (should return numeric ID)

API Note: Pushover contacts appear as type=9 in UptimeRobot API responses.

Healthchecks.io Pushover Setup

1. Go to https://healthchecks.io/projects
2. Navigate to "Integrations" → "Add Integration"
3. Select "Pushover"
4. Enter your Pushover User Key and API Token/Key
5. Save integration
6. Verify with: client.get_pushover_channel_id() (should return UUID)

API Note: Pushover channels use kind code "po" (abbreviated), not "pushover".

Common Issue: If get_pushover_contact_id() or get_pushover_channel_id() returns None, Pushover is not configured. Complete setup steps above via web UI.

Troubleshooting

UptimeRobot Issues

429 Too Many Requests:

• Free tier has rate limits (10 req/min empirically validated)
• Response includes Retry-After header (observed: 47 seconds)
• Response includes X-RateLimit-Remaining header (counts down from 9 to 0)
• Add exponential backoff between operations
• Use bulk operations where available

Rate Limit Example:

import time
from requests.exceptions import HTTPError