Debug Local O2

Log Search

POST /api/{org}/_search — SQL-based search across any stream.

{
  "query": {
    "sql": "SELECT * FROM {stream} WHERE {conditions}",
    "start_time": 1674789786006000,
    "end_time": 1674793386006000,
    "from": 0,
    "size": 100
  }
}

• Times are microseconds (Unix seconds × 1,000,000)
• SQL is PostgreSQL-compatible
• str_match(field, 'term') — full-text search within a field
• match_all('term') — match any field containing term
• size: -1 — return all results (no pagination)

Response:

{ "took": 155, "hits": [...], "total": 27179, "from": 0, "size": 10, "scan_size": 28943 }

Trace Search

CRITICAL: Trace queries require ?type=traces on the search endpoint. Without it, O2 searches the log stream instead and fields like start_time will not be found.

Overview — GET /api/{org}/{stream}/traces/latest

Query params: start_time, end_time, from, size, filter (all required except filter).

Returns high-level trace list with trace IDs, durations, and service names.

Span details — POST /api/{org}/_search?type=traces with SQL against the trace stream:

SELECT * FROM default WHERE trace_id = '{id}' ORDER BY start_time

Use size: -1 to retrieve all spans for a trace.

Log fields: _timestamp, body, severity (numeric: 1=TRACE, 5=DEBUG, 9=INFO, 13=WARN, 17=ERROR, 21=FATAL), service_name, trace_id, span_id, instrumentation_library_name

Trace fields: trace_id, span_id, operation_name, service_name, duration, start_time, end_time, span_kind, span_status (UNSET/ERROR), http_route, http_response_status_code, status_code, status_message

Other Endpoints

Example Scripts

Pre-written queries for common debugging scenarios. Run directly via bash .agents/skills/debug-local-o2/scripts/<script>.sh, or compose your own curl -s ... | jq -c ... commands as needed.

Common Queries

Logs

-- Error+ logs (severity >= 17)
SELECT * FROM default WHERE severity >= 17 ORDER BY _timestamp DESC

-- Logs for a specific user
SELECT * FROM default WHERE str_match(body, '[email protected]')

-- Inngest function logs
SELECT * FROM default WHERE str_match(service_name, 'inngest')

-- Logs with a specific attribute
SELECT * FROM default WHERE match_all('draft_id')

Traces (remember ?type=traces)

-- All spans for a trace
SELECT * FROM default WHERE trace_id = '{id}' ORDER BY start_time

-- Slow spans (duration in microseconds)
SELECT * FROM default WHERE duration > 1000000 ORDER BY duration DESC

-- Error spans
SELECT * FROM default WHERE span_status = 'ERROR' ORDER BY start_time DESC

-- Spans for a specific HTTP route
SELECT * FROM default WHERE http_route = '/dashboard' ORDER BY start_time DESC

Noise Filtering

The Inngest dev server polls /api/inngest continuously for auto-discovery, generating significant trace/log noise. Exclude it from queries:

-- Logs: exclude Inngest polling
SELECT * FROM default WHERE NOT str_match(body, '/api/inngest') ...

-- Traces: exclude Inngest polling spans
SELECT * FROM default WHERE http_route != '/api/inngest' ...

Debugging Workflow

Recommended approach — trace-first:

1. Find relevant traces — use recent-traces.sh or filter by time window
2. Inspect spans — use trace-spans.sh {trace_id} to see the full span tree
3. Correlate with logs — use search-by-trace.sh {trace_id} to find log records emitted during that trace
4. Drill into specifics — compose ad-hoc queries targeting specific fields, services, or error conditions

Manual Curl Template

When composing ad-hoc queries, first compute the timestamp with date +%s, then embed it in the curl request:

# Step 1: Get current epoch (run separately)
date +%s

# Step 2: Use the output (e.g., 1708700000) to compose the query
curl -s -u "[email protected]:password" \
  -X POST "http://localhost:5080/api/default/_search" \
  -H 'Content-Type: application/json' \
  -d '{"query":{"sql":"SELECT * FROM default WHERE ...","start_time":1708399700000000,"end_time":1708700000000000,"from":0,"size":50}}' \
  | jq -c '.hits'

Use jq -c (compact output) to minimize context usage. Use jq -c '.hits[]' to stream individual records.