Observability

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "Microsoft.EntityFrameworkCore.Database.Command": "Warning",
        "System": "Warning"
      }
    }
  }
}

Log level Warning for EF Core SQL commands — never log SQL at Information in production (may expose query parameters containing PII).

Log level standards

PII-free logging rules

NEVER log:

• User names, email addresses
• SIN, DL number, student number, or other personal identifiers
• Connection strings, tokens, passwords, or Vault values
• Full HTTP request/response bodies (may contain form data)
• Query string parameters that may carry tokens (?code=, ?token=)

Safe to log:

• User ID (GUID / sub claim) — opaque identifier only
• HTTP method + path (no query string)
• HTTP status code
• Duration (ms)
• Correlation ID / trace ID

Structured logging pattern

// Use structured properties, not string interpolation
_logger.LogInformation("Created work item {WorkItemId} for project {ProjectId}",
    item.Id, item.ProjectId);

// NOT:
_logger.LogInformation($"Created work item {item.Id}");

Health Checks

Package

dotnet add package AspNetCore.HealthChecks.MySql

Registration in Program.cs

// HEALTH CHECKS — /api/health (liveness) and /api/health/details (readiness)
builder.Services.AddHealthChecks()
    .AddMySql(
        connectionString: builder.Configuration.GetConnectionString("DefaultConnection")!,
        name: "database",
        tags: ["ready"]);

// ...
app.MapHealthChecks("/api/health");
app.MapHealthChecks("/api/health/details", new HealthCheckOptions
{
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse,
    Predicate      = _ => true,
});

Containerfile health check

HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
  CMD wget -qO- http://localhost:8080/api/health || exit 1

Metrics — Prometheus

Emerald uses a Prometheus pull model. Annotate pods so Prometheus discovers the metrics endpoint.

Pod annotations (Helm values.yaml)

podAnnotations:
  prometheus.io/scrape: "true"
  prometheus.io/path:   "/metrics"
  prometheus.io/port:   "8080"

.NET metrics endpoint (ASP.NET Core)

dotnet add package prometheus-net.AspNetCore

// METRICS — expose /metrics for Prometheus scrape
app.UseHttpMetrics();  // request duration, count, in-flight
app.MapMetrics();      // GET /metrics

Custom metrics example

// create once at class level (static) — Prometheus counters are global
private static readonly Counter _itemsCreated =
    Metrics.CreateCounter("app_items_created_total", "Number of work items created.",
        labelNames: ["project_id"]);

// in service method — increment counter when an item is created
_itemsCreated.WithLabels(projectId.ToString()).Inc();

Tracing — OpenTelemetry

Packages

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.Console   # dev only

Registration

// TRACING — OpenTelemetry with OTLP export (or console in dev)
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .SetResourceBuilder(ResourceBuilder.CreateDefault()
                .AddService(serviceName: "my-app-api"))
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddEntityFrameworkCoreInstrumentation();

        if (builder.Environment.IsDevelopment())
            tracing.AddConsoleExporter();
        else
            tracing.AddOtlpExporter(opt =>
                opt.Endpoint = new Uri(builder.Configuration["Otlp:Endpoint"]!));
    });

Alert Baseline (document in securityNextSteps.md)

OBSERVABILITY_KNOWLEDGE

confirmed_facts:
  - "CompactJsonFormatter writes single-line JSON to stdout — required for OpenShift log aggregation"
  - "Prometheus on Emerald uses pod annotations to discover /metrics endpoints"
  - "EF Core SQL logging at Information level may expose query parameters — use Warning"
  - "OpenTelemetry AddEntityFrameworkCoreInstrumentation requires EFCore instrumentation package"
  - "Health check at /api/health is used by OpenShift liveness probe; /api/health/details for readiness"
common_pitfalls:
  - "Never log PII — user emails, names, or identifying numbers in structured properties"
  - "Metrics.CreateCounter must be static — creating per-request instances causes memory leaks"
  - "prometheus-net endpoint conflicts with ASP.NET Core minimal API route if MapMetrics() called after MapControllers()"