Grafana Tempo

Configurar Tempo con el backend de almacenamiento y limites apropiados:

# tempo.yml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio.storage:9000
      access_key: ${MINIO_ACCESS_KEY}
      secret_key: ${MINIO_SECRET_KEY}
      insecure: true
    wal:
      path: /var/tempo/wal
    block:
      bloom_filter_false_positive: 0.05
    pool:
      max_workers: 100
      queue_depth: 10000

compactor:
  compaction:
    block_retention: 72h

metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write

Instrumentar los microservicios del pipeline KYC con OpenTelemetry:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

provider = TracerProvider()
processor = BatchSpanProcessor(
    OTLPSpanExporter(endpoint="tempo:4317", insecure=True)
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

FastAPIInstrumentor.instrument_app(app)

Crear spans personalizados para cada etapa del pipeline de verificacion:

tracer = trace.get_tracer("kyc-pipeline")

async def verify_identity(session_id: str):
    with tracer.start_as_current_span("kyc.verification",
        attributes={"session.id": session_id}) as root_span:

        with tracer.start_as_current_span("kyc.liveness_check"):
            liveness_result = await liveness_module.check(session_id)
            trace.get_current_span().set_attribute("liveness.score", liveness_result.score)

        with tracer.start_as_current_span("kyc.document_processing"):
            doc_result = await doc_module.process(session_id)

        with tracer.start_as_current_span("kyc.face_match"):
            match_result = await face_match.compare(session_id)
            trace.get_current_span().set_attribute("face_match.similarity", match_result.score)

        with tracer.start_as_current_span("kyc.antifraud_analysis"):
            fraud_result = await antifraud.analyze(session_id)

        with tracer.start_as_current_span("kyc.decision"):
            decision = await decision_engine.evaluate(session_id)
            root_span.set_attribute("verification.result", decision.status)

Configurar Grafana para usar Tempo como datasource de trazas:

datasources:
  - name: Tempo
    type: tempo
    url: http://tempo:3200
    access: proxy
    jsonData:
      tracesToMetrics:
        datasourceUid: prometheus
        tags:
          - key: "service.name"
            value: "module"
      tracesToLogs:
        datasourceUid: loki
        filterByTraceID: true
        tags:
          - key: "session.id"
      serviceMap:
        datasourceUid: prometheus

Habilitar el generador de metricas de Tempo para obtener RED metrics automaticas:

metrics_generator:
  processor:
    service_graphs:
      dimensions:
        - session.status
    span_metrics:
      dimensions:
        - session.status
        - verification.result

Crear queries de ejemplo en Grafana para buscar trazas del pipeline KYC:

# TraceQL: buscar verificaciones rechazadas con latencia alta
{ span.verification.result = "rejected" && duration > 5s }

# TraceQL: buscar sesiones con score de liveness bajo
{ span.liveness.score < 0.5 && name = "kyc.liveness_check" }

# TraceQL: buscar errores en face_match
{ name = "kyc.face_match" && status = error }