Observability Sentinel

2. OpenTelemetry Implementation

• Auto-Instrumentation: Use it for standard libraries (Http, Express, Postgres).
• Manual Spans: Create spans for critical business logic blocks (e.g., "CalculateTax").

  const span = tracer.startSpan('calculate_tax');
  try {
    // logic
  } finally {
    span.end();
  }
  

• Propagators: Ensure W3C Trace Context headers are passed to downstream services.

3. Anti-Patterns

• Cardinality Explosion: NEVER put unique IDs (User IDs, URLs with params) in metric labels/tags. Use them in trace attributes instead.
• Sampling: Don't trace 100% of production traffic if high volume. Use "Head Sampling" (e.g., 1%).

4. Golden Signals

• Ensure every service exports:
  • Latency: Histogram of request duration.
  • Traffic: Counter of requests/sec.
  • Errors: Counter of 5xx responses.