This guides how you analyze the factory's performance and propose changes. Every proposal must be backed by evidence from the telemetry database — not intuition, not best practices, not "I think this would be better."

Querying Telemetry

The telemetry database is at eval/factory.db (SQLite). Use Bash to query it.

Common Queries

Agent failure rates:

SELECT agent, 
       COUNT(*) as total_runs,
       SUM(CASE WHEN verdict='fail' THEN 1 ELSE 0 END) as failures,
       ROUND(100.0 * SUM(CASE WHEN verdict='fail' THEN 1 ELSE 0 END) / COUNT(*), 1) as fail_rate
FROM agent_runs 
GROUP BY agent 
ORDER BY fail_rate DESC;

Token usage by agent (cost optimization):

SELECT agent, model,
       AVG(output_tokens) as avg_tokens,
       MIN(output_tokens) as min_tokens,
       MAX(output_tokens) as max_tokens
FROM agent_runs
GROUP BY agent, model;

Querying Telemetry

The telemetry database is at eval/factory.db (SQLite). Use Bash to query it.

Common Queries

Agent failure rates:

SELECT agent, 
       COUNT(*) as total_runs,
       SUM(CASE WHEN verdict='fail' THEN 1 ELSE 0 END) as failures,
       ROUND(100.0 * SUM(CASE WHEN verdict='fail' THEN 1 ELSE 0 END) / COUNT(*), 1) as fail_rate
FROM agent_runs 
GROUP BY agent 
ORDER BY fail_rate DESC;

Token usage by agent (cost optimization):

SELECT agent, model,
       AVG(output_tokens) as avg_tokens,
       MIN(output_tokens) as min_tokens,
       MAX(output_tokens) as max_tokens
FROM agent_runs
GROUP BY agent, model;

Improvement Methodology

Querying Telemetry

Common Queries

Improvement Methodology

Querying Telemetry

Common Queries

Evidence Standards

What Counts as a Pattern

Citing Evidence

Reading Transcripts

Risk Classification

Safe (auto-apply to branch)

Needs Review (document in PR body only)

Dangerous (document in PR body with warning)

PR Format

Improvement Schema

Deep Research

Data Analyst

Academic Researcher

Data Scientist

Biopython

Binary Analysis Patterns