Agent Observability Report Skill

Both UDTFs take 4 arguments: (database_name VARCHAR, schema_name VARCHAR, agent_name VARCHAR, agent_type VARCHAR)

Example: TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS_NORMALIZED('{DATABASE}', '{SCHEMA}', '{AGENT_NAME}', 'CORTEX AGENT'))

Event Types

Workflow

Step 1: Validate Data Source

Goal: Confirm access and collect parameters.

-- List available agents in the account
SHOW AGENTS IN ACCOUNT;

Use SHOW AGENTS (or SHOW AGENTS IN DATABASE {db} / SHOW AGENTS IN SCHEMA {db}.{schema}) to discover agent names. Then validate data access for a specific agent:

SELECT COUNT(*) AS event_count
FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS('{DATABASE}', '{SCHEMA}', '{AGENT_NAME}', 'CORTEX AGENT'))
WHERE RECORD:"name"::STRING = 'CORTEX_AGENT_REQUEST'
  AND TIMESTAMP >= CURRENT_DATE - 30

Parameters to collect:

• Agent Name
• Database Name (from discovery query)
• Schema Name (from discovery query)
• Time Range (default: 7 days)
• Report depth (summary/detailed/full)

CHECKPOINT: If UDTF not accessible or returns error:

• Inform user that AI Observability UDTFs are not available
• Verify user has MONITOR/OWNERSHIP on the AGENT object and CORTEX_USER database role
• Do NOT offer to set it up; user should contact their Snowflake administrator
• Stop workflow

Step 2: Select Metric Categories

Goal: Let user choose which metric categories to collect.

USER PROMPT (REQUIRED): Ask user to select categories:

Note: Allow multi-select. Default to "All categories" if user doesn't specify.

⚠️ MANDATORY STOPPING POINT: Wait for user selection before proceeding.

Step 3: Collect Metrics

Goal: Materialize observability events once, then run analytical queries against the materialized table.

Step 3a: Materialize Events

Create a temporary materialized table from a single UDTF call. This avoids repeated UDTF invocations (which are slow) by scanning the data once:

CREATE OR REPLACE TEMPORARY TABLE _obs_events AS
SELECT *
FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS('{DATABASE}', '{SCHEMA}', '{AGENT_NAME}', 'CORTEX AGENT'))
WHERE TIMESTAMP >= CURRENT_DATE - {DAYS};

Important: If temp tables are not accessible across queries (session isolation), use a permanent table with a fully qualified name instead:

CREATE OR REPLACE TABLE {USER_DB}.{USER_SCHEMA}._obs_events AS
SELECT *
FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS('{DATABASE}', '{SCHEMA}', '{AGENT_NAME}', 'CORTEX AGENT'))
WHERE TIMESTAMP >= CURRENT_DATE - {DAYS};

Step 3b: Run Analytical Queries

Load queries from references/queries.md for selected categories only. All queries reference _obs_events (or the fully qualified name) instead of calling the UDTF directly.

Run queries for all selected categories in parallel for maximum speed.

Step 3c: Cleanup

After all queries complete, drop the materialized table:

DROP TABLE IF EXISTS _obs_events;

CHECKPOINT: If category fails, note in data_limitations and continue.

Step 4: Calculate Derived Metrics

Compute from raw data per references/metrics_catalog.md:

• Percentiles (P50, P90, P95, P99)
• Rates (success_rate, error_rate, feedback_rate)
• Ratios (cache_hit_rate, input_output_ratio)
• Trends (daily_trend, week_over_week)

Step 5: Validate Insights

Apply statistical rigor per references/insight-quality-guardrails.md:

Language rules: Use correlational language ("associated with"), not causal ("causes").

Step 6: Compile Report

{
  "_meta": {
    "skill": "agent-observability-report",
    "version": "1.0",
    "generated_at": "ISO8601",
    "data_source": "SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS_NORMALIZED"
  },
  "usage_adoption": { "raw": {}, "derived": {} },
  "performance_latency": { "raw": {}, "derived": {}, "percentiles": {} },
  "token_economics": { "raw": {}, "derived": {} },
  "quality_reliability": { "raw": {}, "derived": {} },
  "user_feedback": { "raw": {}, "derived": {}, "confidence": "low|moderate|high" },
  "tool_execution": { "raw": {}, "derived": {}, "by_tool": [] },
  "conversation_behavior": { "raw": {}, "derived": {} },
  "key_insights": [],
  "data_limitations": [],
  "recommendations": []
}

Step 7: Select Output Format

Goal: Let user choose how to receive the report.

USER PROMPT (REQUIRED): Ask user for output format:

⚠️ MANDATORY STOPPING POINT: Wait for user selection before proceeding.

Step 8: Deliver Report

If "Display in chat":

Present full report in conversation:

Agent: {AGENT_NAME}
Period: {DAYS} days

USAGE: X requests, Y users
PERFORMANCE: Avg X ms, P90 Y ms
QUALITY: X% success rate
FEEDBACK: X% satisfaction (n=Y)

[Full metrics for each selected category]
[Key insights]
[Recommendations]

If "Save as JSON":

1. Save report to: Reports/{YYYY-MM-DD}/{agent}_observability_{days}days.json
2. Confirm file path to user

Follow-up: Offer deep dive, compare periods, or switch output format

Query Pattern

Preferred (normalized UDTF with top-level columns):

SELECT {metrics}
FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS_NORMALIZED('{DATABASE}', '{SCHEMA}', '{AGENT_NAME}', 'CORTEX AGENT'))
WHERE SPAN_NAME = '{EVENT_TYPE}'
  AND TIMESTAMP >= CURRENT_DATE - {DAYS}

Fallback (raw UDTF):

SELECT {metrics}
FROM TABLE(SNOWFLAKE.LOCAL.GET_AI_OBSERVABILITY_EVENTS('{DATABASE}', '{SCHEMA}', '{AGENT_NAME}', 'CORTEX AGENT'))
WHERE RECORD:"name"::STRING = '{EVENT_TYPE}'
  AND TIMESTAMP >= CURRENT_DATE - {DAYS}

Troubleshooting

Stopping Points

• ✋ Step 1: If event table not accessible (stop workflow)
• ✋ Step 2: Category selection (wait for user)
• ✋ Step 7: Output format selection (wait for user)

References

• references/metrics_catalog.md - 110 metrics across 8 categories
• references/queries.md - SQL query templates
• references/insight-quality-guardrails.md - Statistical validation rules