Cekura Labs Workflow

The Labs Improvement Cycle

For edge case refinement after manual fixes are validated:

1. Identify misalignment — Find calls where metric results seem wrong
2. Leave feedback — Vote agree/disagree with explanation on specific results
3. Accumulate feedback — Collect 6+ feedback instances for statistical significance
4. Run auto-improve — Labs uses feedback to suggest metric prompt changes
5. Validate changes — Re-run the improved metric on the same calls to verify
6. Deploy — Once satisfied, optionally convert to Pythonic custom_code for production

Step 1: Identify Misaligned Results

Manual Approach

Review recent call evaluations to find suspicious results:

Use mcp__cekura__call_logs_list with agent filters to list recent calls. Use mcp__cekura__call_logs_retrieve with a call ID to get evaluation results.

Look for:

• FALSE results that seem like they should be TRUE (false negatives)
• TRUE results that seem wrong (false positives)
• Unexpected N/A results
• Inconsistent results across similar calls

Guided Approach (Simulate Labs)

To systematically find misalignment:

1. Pull metric results for recent calls via evaluate_calls or list_calls
2. Focus on metrics with high variance or unexpected result distributions
3. Read the transcript alongside the metric explanation
4. Ask the user: "This call was marked [TRUE/FALSE]. The explanation says [X]. Does this seem correct?"
5. If the user disagrees, proceed to leave feedback

Step 2: Leave Feedback

Use the mark_metric_vote endpoint to leave structured feedback. Submit via mcp__cekura__call_logs_retrieve to get the call, then use the feedback/vote API.

Good Feedback Patterns

• Reference specific transcript moments: "At 02:15, the agent said X which should be [PASS/FAIL] because..."
• Explain the reasoning: "The metric is being too literal about the one-question rule"
• Suggest the correct outcome: "This should be TRUE because the agent was confirming related details"
• Point out missing context: "The metric didn't account for the tool failure at 01:30"

Bad Feedback Patterns

• Vague: "This is wrong" (no explanation)
• No reference: "Should be TRUE" (no transcript evidence)
• Contradictory: Disagreeing without explaining what the correct behavior should be

Step 3: Accumulate Feedback

Collect at least 6 feedback instances before running auto-improve. This gives labs enough signal to identify patterns in the feedback and make meaningful prompt adjustments.

Track feedback progress:

• How many agree/disagree votes have been left
• What patterns emerge (e.g., "metric is consistently too strict about X")
• Whether feedback is balanced across different call types

Step 4: Run Auto-Improve

Once 6+ feedback instances are accumulated:

Use mcp__cekura__metrics_run_reviews_create with the metric ID to trigger auto-improvement.

Labs analyzes the feedback and suggests changes to the metric prompt. Review the suggested changes carefully:

• Do the changes address the feedback patterns?
• Are the changes too broad (might break other cases)?
• Do the safeguarding examples align with the feedback?

Cost Guard — Never Evaluate >100 Calls Without Confirmation

Each evaluation costs the client real money. Before triggering any bulk evaluation:

1. Query the call count first (use page_size=1 and read the response count)
2. Report the number to the user
3. If count > 100, stop and ask for explicit approval before proceeding

Use page_size parameter (up to 200) instead of paginating through multiple pages. Use server-side filters (agent_id, project, timestamp__gte/timestamp__lte) to scope calls before evaluating.

Step 5: Validate Changes

Re-run the improved metric on the same calls that had misaligned results:

Use mcp__cekura__call_logs_rerun_evaluation_create with the call IDs and metric ID.

Check:

• Do the previously misaligned calls now produce correct results?
• Do previously correct calls still produce correct results (no regression)?
• Are the explanations clearer and more accurate?

If validation fails, leave additional feedback and iterate.

Step 6: Deploy (Optional Pythonic Conversion)

Once the metric prompt is validated through labs, consider converting to a Pythonic custom_code metric for production:

1. Take the final validated prompt from the llm_judge description field
2. Wrap it in custom_code with section extraction (see metric-design skill)
3. Set both description (the prompt) and custom_code (the Python wrapper)
4. Toggle to custom_code as the active type

This gives the benefit of the labs-refined prompt with the performance advantage of targeted context extraction.

Interactive Labs Simulation

When the user wants to simulate the labs workflow interactively:

1. Fetch recent calls and metric results
2. Present potentially misaligned results to the user one at a time
3. Ask: "Call [ID] was scored [result]. Here's the explanation: [explanation]. The transcript shows: [relevant excerpt]. Do you agree with this result?"
4. If user disagrees, generate the feedback payload and submit via mark_metric_vote
5. Track feedback count and notify when 6+ is reached
6. Offer to trigger auto-improve

API Endpoints Reference

Labs Auto-Improve (process_feedbacks)

POST /test_framework/metric-reviews/process_feedbacks/
{
  "metric_id": 123,
  "test_set_ids": [456, 789]
}

Optional fields: metric_type (default "llm_judge"), skip_evaluation (bool), simplified_prompt (string).

Returns {"progress_id": "<uuid>"}. Poll at GET /test_framework/metric-reviews/process_feedbacks_progress/?progress_id=<uuid>.

The response includes improved description and evaluation_trigger when complete — you must PATCH the metric to apply changes (they are not auto-applied).

Create Test Set from Call Log

POST /test_framework/test-sets/create_from_call_log/
{
  "call_log_id": 3358270,
  "metrics": [{"metric": 123, "feedback": "The metric incorrectly failed this call because..."}]
}

Note: metrics must be an array of objects [{"metric": <id>, "feedback": "<text>"}], NOT bare metric IDs. Passing bare IDs returns 500.

Additional Resources

Reference Files

• references/feedback-examples.md — Examples of good feedback for different metric types