Railway Observability

Quick Start

1. Access Built-in Observability Dashboard

Navigate to your Railway project:

Railway Dashboard → Project → Service → Metrics Tab

What you see:

• CPU usage (% of allocated resources)
• Memory usage (MB/GB over time)
• Disk usage (storage consumption)
• Network I/O (ingress/egress)
• 30-day metric retention

2. Configure Custom Dashboard Widgets

Add and customize metric widgets:

Metrics Tab → Add Widget → Select Metric Type

Available widgets:

• Line charts (time-series metrics)
• Bar charts (deployment comparisons)
• Gauges (current values)
• Tables (multi-metric views)

Customization:

• Drag-and-drop layout
• Adjustable time ranges (1h, 6h, 24h, 7d, 30d)
• Multi-replica aggregation
• Deployment markers on charts

3. Set Up Alerting (Pro Plan Required)

Configure alerts for threshold violations:

Service Settings → Alerts → Create Alert Rule

Alert types:

• CPU threshold (e.g., > 80% for 5 minutes)
• Memory threshold (e.g., > 90% for 10 minutes)
• Disk usage (e.g., > 85%)
• Custom metric thresholds

Notification channels:

• Email (default)
• Discord webhook
• Slack webhook
• Custom webhook (JSON payload)

4. Integrate External OTEL Collectors

Export metrics to external systems:

Service Settings → Observability → OTEL Integration

Configure environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector:4318
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <token>
OTEL_SERVICE_NAME=my-railway-service

See references/otel-integration.md for complete setup.

5. Deploy Full Grafana Stack

Use Railway template for complete observability:

# Option 1: Deploy via Railway Dashboard
# Template ID: 8TLSQD (Grafana Stack)
# Includes: Grafana, Prometheus, Loki, Tempo, Alloy

# Option 2: Deploy via script
.claude/skills/railway-observability/scripts/deploy-grafana-stack.sh

Stack components:

• Grafana (visualization)
• Prometheus (metrics)
• Loki (logs)
• Tempo (traces)
• Alloy (collector)

Workflow

Step 1: Access Built-in Metrics

Railway provides instant metrics without configuration.

Navigate to metrics:

Project → Service → Metrics

Available metrics:

• CPU: % usage, allocated cores
• Memory: MB used, % of limit
• Disk: GB used, read/write IOPS
• Network: MB/s ingress/egress

Retention: 30 days for all metrics

Step 2: Customize Dashboard

Create personalized monitoring views.

Add widgets:

1. Click "Add Widget"
2. Select metric type
3. Configure aggregation
4. Set time range
5. Position via drag-and-drop

Best practices:

• Group related metrics (CPU + Memory)
• Use deployment markers for correlation
• Configure multi-replica views
• Set appropriate time ranges

Step 3: Configure Alerts (Pro Plan)

Set up proactive monitoring.

Create alert rule:

Service → Settings → Alerts → New Rule

Alert configuration:

Metric: CPU Usage
Condition: Greater than 80%
Duration: 5 minutes
Notification: Slack webhook

Webhook payload example:

{
  "service": "backend-production",
  "metric": "cpu_usage",
  "threshold": 80,
  "current": 87.5,
  "timestamp": "2025-11-26T10:30:00Z"
}

See references/dashboard-widgets.md for all alert types.

Step 4: Integrate OTEL Exporters

Send metrics to external systems.

Configure Alloy collector:

# Use template from templates/alloy-config.river
# Deploy as Railway service
# Configure OTEL endpoints

Environment setup:

# In your Railway service
OTEL_EXPORTER_OTLP_ENDPOINT=http://alloy:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_METRICS_EXPORTER=otlp
OTEL_LOGS_EXPORTER=otlp
OTEL_TRACES_EXPORTER=otlp

Verify integration:

# Check Alloy logs
railway logs -s alloy

# Should show: "Successfully received OTLP metrics"

See references/otel-integration.md for complete guide.

Step 5: Deploy Grafana Observability Stack

Full monitoring solution on Railway.

Deploy stack:

# Run deployment script
cd .claude/skills/railway-observability/scripts
./deploy-grafana-stack.sh

# Or deploy manually via Railway Dashboard
# Template: 8TLSQD (Grafana Stack)

Stack includes:

• Grafana (port 3000) - Dashboards
• Prometheus (port 9090) - Metrics storage
• Loki (port 3100) - Log aggregation
• Tempo (port 3200) - Distributed tracing
• Alloy (port 4318) - OTLP collector

Access Grafana:

URL: https://<grafana-service>.up.railway.app
Username: admin
Password: (set during deployment)

Architecture

Progressive Disclosure Structure

1. SKILL.md (this file): Quick start and workflow
2. references/: Detailed documentation
   • metrics-reference.md - Complete metrics catalog
   • dashboard-widgets.md - Widget configuration guide
   • otel-integration.md - External integration setup
3. scripts/: Automation tools
   • deploy-grafana-stack.sh - Deploy observability stack
4. templates/: Configuration files
   • alloy-config.river - Grafana Alloy collector config

Key Features

Built-in Metrics (Always Available)

No configuration required - Metrics collected automatically for all services.

Dashboard Customization

Drag-and-drop widgets:

• Resize and reposition
• Multiple chart types
• Custom time ranges
• Deployment markers

Multi-replica support:

• Aggregated views (avg, min, max, sum)
• Per-replica breakdowns
• Replica scaling events

Time range options:

• Last 1 hour
• Last 6 hours
• Last 24 hours
• Last 7 days
• Last 30 days
• Custom range

Alerting (Pro Plan Only)

Threshold alerts:

• CPU > X% for Y minutes
• Memory > X MB for Y minutes
• Disk > X% capacity
• Network > X MB/s

Notification channels:

# Email
[email protected]

# Discord webhook