Java Observability Tracing

Standard propagation: W3C Trace Context

• Use traceparent and tracestate headers for cross-service propagation.
• Ensure outbound HTTP and messaging producers inject propagation headers.
• Ensure inbound handlers extract propagation headers.

Baggage policy

• Use baggage sparingly: it propagates everywhere and can increase cost.
• Never put secrets/PII in baggage.
• Define an allowlist of baggage keys if you enable it.

Instrumentation approach (recommended default)

Option A: Auto-instrumentation first

• Use the OpenTelemetry Java agent for fast baseline coverage:
  • inbound HTTP server spans
  • outbound HTTP client spans
  • common libraries (JDBC, messaging) Pros: fastest adoption, fewer code changes.
    Cons: may need manual spans for domain-specific operations.

Option B: Manual instrumentation for key domains

• Add explicit spans for:
  • business workflows
  • critical background jobs
  • complex async pipelines
• Use semantic attributes and stable span names.

Span design rules

Span naming

• Use low-cardinality names:
  • HTTP GET /v1/orders/{id}
  • DB SELECT orders
  • MQ consume orders.created
• Avoid embedding IDs in span names.

Attributes (tagging)

• Use semantic conventions where available.
• Keep attributes bounded; avoid IDs in attributes unless necessary and approved.

Events and errors

• Record exceptions as span events (but do not leak secrets).
• Use status to represent error outcome.

Sampling strategy (cost vs value)

Default recommended: Parent-based + TraceIdRatio

• Keep sampling consistent across services.
• Start with a low rate in prod (e.g., 1% to 10%), increase during incidents.
• Always sample errors if your system supports tail sampling (advanced).

Operational knobs

• Support runtime configuration for sampling (env-based).
• Document how to temporarily increase sampling for debugging.

Export and resource identity

• Export via OTLP (HTTP/gRPC) to your collector/backend.
• Set resource attributes:
  • service.name, service.version, deployment.environment. These must be consistent across services to enable cross-service queries.

Correlation with logs

• Inject traceId and spanId into logs:
  • via MDC or OpenTelemetry-aware logging appenders.
• This enables “logs <-> traces” pivoting during incidents.

Key spans map (template)

A minimal “span map” for a typical backend request:

1. Inbound HTTP span:
   • name: HTTP <method> <route>
2. Authentication/authorization span (optional, internal):
   • name: auth.verify
3. DB span:
   • name: DB <operation> <table>
4. Outbound dependency call span:
   • name: HTTP <method> <upstream>
5. Business workflow span:
   • name: orders.create (manual)
6. Response finalize span events:
   • errors, retries, timeouts as span events

Testing and validation

• Smoke test: confirm traces exported in staging.
• Propagation test: multi-service request produces a single trace.
• Attribute linting: reject forbidden high-cardinality attributes.
• Performance check: ensure overhead acceptable at target sampling rate.

Outputs / artifacts

• docs/tracing.md (propagation, sampling, span naming)
• observability/span-naming.md
• observability/attribute-policy.md
• Code changes:
  • OTel agent config or SDK bootstrap
  • manual spans in critical workflows
  • log correlation integration
• Tests:
  • propagation tests
  • policy tests for forbidden attributes

Definition of Done (DoD)

• Inbound and outbound requests generate spans.
• W3C propagation works across services.
• Sampling configured and documented.
• Resource identity set consistently.
• Logs include traceId/spanId for correlation.
• Attribute policy prevents cardinality/security issues.

Guardrails (What NOT to do)

• Never propagate secrets/PII via baggage or attributes.
• Avoid high-cardinality attributes (IDs, full URLs, raw payloads).
• Do not create a span for every tiny function (noise and overhead).
• Avoid inconsistent sampling across services.

Common failure modes & fixes

• Symptom: broken traces across services -> Fix: ensure traceparent extraction/injection on all hops.
• Symptom: tracing cost too high -> Fix: reduce sampling; tighten attribute policy; focus manual spans.
• Symptom: cannot find root cause -> Fix: add manual spans for domain workflows; record key events.

References (see references/)

• references/w3c-trace-context.md
• references/w3c-baggage-policy.md
• references/otel-java-agent.md
• references/otel-semconv-http.md
• references/sampling-playbook.md