Stage3 Sop Generation

Required

• extraction_output_dir: Directory with Stage 2 results (e.g., results/{company}/02_extraction)
• company: Company name
• sop_title: Title for the generated SOP set

Optional

• sop_type (default: "customer_support"): "customer_support", "troubleshooting", or "sales"

Steps

1. Load Extraction Results

Read all Stage 2 outputs:

1. patterns.json — patterns + sop_topic_map (authoritative topic plan)
2. faq.json — FAQ pairs by SOP topic
3. patterns_enriched.json — full conversation transcripts per cluster

Constraints:

• MUST follow sop_topic_map exactly — one SOP per topic, no additions or removals
• MUST read templates before generating: templates/HT_template.md and templates/TS_template.md
• MUST NOT redesign topic structure (already done in Stage 2)

2. Extract Concrete Details from Conversations

Before writing any SOP, mine patterns_enriched.json for actionable details per topic.

Extract from agent messages:

• Internal tool URLs/paths
• Step-by-step instructions agents actually gave (near-verbatim)
• UI navigation paths (e.g., "마이페이지 → 예약내역 → 하단 취소버튼")
• Conditional branches ("~인 경우", "~하셨다면")
• Warning messages ("주의", "중요", "꼭")

Extract from customer messages:

• Common expressions describing the problem (verbatim)
• Frequently asked follow-up questions

Extraction Guardrails:

• Distinguish standard procedures from exception measures (one-time workarounds should not become SOP steps)
• Don't over-generalize FAQ escalation phrases to all cases in the same topic
• Each Case in the SOP should cover ONE customer scenario, not mix multiple issues

3. Generate SOP Documents (per topic)

For each topic in sop_topic_map, generate one SOP file.

File naming: HT_{여정단계}_{주제}.sop.md or TS_{여정단계}_{주제}.sop.md

Template: Follow templates/HT_template.md for HT, templates/TS_template.md for TS. Do NOT use Overview/Parameters/Steps/Examples structure.

Each Case under 해결책 안내 MUST include:

1. Case 헤더 with 정량적 빈도 (e.g., "전체의 66.7%")
2. 주요 상황 서술 (customer scenario, not technical cause)
3. Step별 안내 멘트 in 블록 인용(>) — actual message-level detail, not "확인해 주세요"
4. 내부 도구 경로 or {placeholder} + <!-- TODO --> if not found
5. 조건 분기 and 완료 확인 ("~되셨나요?")

For TS SOPs: Follow non-destructive order (least → most invasive steps), with confirmation checkpoint after each step.

Constraints:

• MUST write SOP using LLM composition, NOT Python auto-generation
• MUST base 안내 멘트 on actual agent messages from enriched data
• MUST NOT invent URLs, tool paths, or contact info not found in data
• MUST save to results/{company}/03_sop/ (or 03_sop_v2/ if already exists)

4. Verify Each SOP Against Real Conversations (NEW)

This is the key quality improvement step. After generating each SOP, test it against 3-5 real conversations from that topic's enriched data.

Process per SOP:

1. Pick (Case 수 × 2) conversations from patterns_enriched.json that belong to this topic (prefer diverse scenarios). E.g., Case 4개 → 8건, Case 2개 → 4건.
2. Simulate: For each conversation, walk through the SOP as if you were the agent:
   • Does the SOP cover this customer's question?
   • Can you follow the steps to reach the same resolution the real agent reached?
   • Are there steps the real agent took that the SOP doesn't mention?
   • Did the real agent give information the SOP lacks?
3. Record gaps:

   Conversation ID: {id}
   Customer issue: "취소하고 싶은데 수수료가 얼마인가요?"
   SOP coverage: ⚠️ Partial — SOP mentions 취소 방법 but doesn't cover 수수료 안내
   Missing: 지역별 취소수수료 차이 설명, 바우처에서 확인하는 방법
   

4. Fix the SOP: Add missing Cases, steps, or details found during verification
5. Re-save the improved SOP

Constraints:

• MUST verify every generated SOP (not optional)
• MUST use conversations NOT used during initial extraction if possible (to test generalization)
• MUST document verification results in extraction_summary.md
• If >50% of test conversations expose gaps → substantially rewrite the SOP before moving on

Expected Output per SOP:

✅ SOP 검증 완료: HT_예약전_차량검색.sop.md
  테스트 대화: 5건
  완전 커버: 3건 (60%)
  부분 커버: 2건 (40%) → Case 3에 수수료 안내 추가, Step 2에 앱 경로 보강
  미커버: 0건

5. Generate Summary and Metadata

After all SOPs are generated and verified:

1. Save metadata.json with version, source files, statistics, verification results
2. Update extraction_summary.md with SOP list and verification coverage

Output Summary:

✅ Stage 3 완료: {N}개 SOP 생성 + 검증
저장 위치: results/{company}/03_sop/

검증 결과:
  총 테스트 대화: {M}건
  평균 커버율: {X}%
  보강된 SOP: {Y}개

Troubleshooting

Notes

• Verification loop is where quality comes from — generation alone produces 70-80% quality; verification + fix reaches 90%+
• Process SOPs sequentially in main agent for stability
• Templates (HT_template.md, TS_template.md) define structure — this SKILL focuses on content quality and verification
• Total time: ~10-15 min (generation ~7 min + verification ~5 min), but better quality per token spent