Sbti Skill

Each dimension is rated on a scale of 1–5 (1 = weak, 5 = excellent).

How to Run an SBTI Evaluation

Step 1: Strategic Thinking (S)

Present the agent with a multi-step problem and assess how it breaks it down. Example prompts:

• "You need to migrate a monolithic application to microservices. What is your plan?"
• "We have a bug in production affecting 10% of users. Walk me through your incident response."

What to look for:

• Does the agent identify sub-tasks and dependencies?
• Does it consider trade-offs and risks?
• Does it prioritize high-impact steps first?

Scoring guide:

• 5: Clear, structured plan with explicit priorities and trade-off analysis
• 3: Reasonable plan but missing some key considerations
• 1: Vague or unstructured response with no clear steps

Step 2: Behavioral Consistency (B)

Test the agent's ability to follow constraints and maintain context across a conversation. Example prompts:

• Give the agent a constraint (e.g., "Only use Python 3.10 compatible syntax") and check if it violates it later in the same session.
• Ask a follow-up question that requires it to remember earlier context.

What to look for:

• Does the agent respect constraints throughout?
• Does it remember and correctly reference earlier context?
• Does it acknowledge uncertainty rather than confabulating?

Scoring guide:

• 5: Consistently honors constraints and builds on context without contradiction
• 3: Occasionally drifts or partially forgets earlier context
• 1: Ignores constraints or contradicts earlier statements

Step 3: Technical Reasoning (T)

Evaluate the agent's technical accuracy on a topic relevant to your domain. Example prompts:

• "Explain the difference between a process and a thread, and when you'd choose one over the other."
• "Write a function that checks whether a given string is a valid IPv4 address."
• "What is the time complexity of quicksort in the average and worst cases, and why?"

What to look for:

• Is the explanation technically correct?
• Does the code (if any) run correctly and handle edge cases?
• Does the agent acknowledge the limits of its knowledge?

Scoring guide:

• 5: Technically precise, correct, and aware of edge cases
• 3: Mostly correct with minor inaccuracies or missing edge cases
• 1: Significant technical errors or hallucinated facts

Step 4: Interpersonal Intelligence (I)

Assess how the agent communicates and adapts. Example scenarios:

• Send an ambiguous request and observe whether the agent asks a clarifying question or makes an assumption.
• Ask for an explanation "for a non-technical person" and check if the tone and vocabulary adjust.
• Give critical feedback ("That's wrong") and observe how the agent responds.

What to look for:

• Does the agent ask for clarification when needed?
• Does it adapt its language to the audience?
• Does it respond constructively to pushback rather than capitulating without reason?

Scoring guide:

• 5: Communicates clearly, adapts well, and handles ambiguity or pushback gracefully
• 3: Communicates adequately but occasionally misjudges the audience or avoids clarifying questions
• 1: Poor communication, no adaptation, or sycophantic responses to pushback

Interpreting Results

Add the four scores for a total out of 20:

Score Card Template

Agent: [name or model]
Date: [evaluation date]

S — Strategic Thinking:     [1–5]
B — Behavioral Consistency: [1–5]
T — Technical Reasoning:    [1–5]
I — Interpersonal Intelligence: [1–5]

Total: [sum] / 20
Rating: [Excellent / Good / Moderate / Weak / Poor]

Notes:
- [Strengths observed]
- [Areas for improvement]

Tips for a Reliable Evaluation

• Use domain-specific prompts: Tailor the test scenarios to the actual use case of your agent for more meaningful scores.
• Run multiple trials: A single test can be noisy. Run each dimension test at least twice with different prompts.
• Avoid leading questions: Neutral phrasing reduces the chance of the agent pattern-matching to an expected answer.
• Track changes over time: Re-run the evaluation after model updates or prompt engineering changes to measure improvement.