Langextract Extraction

2. Send to LLM and parse JSON response

3. Align extractions to source text (find character positions)

That's it. No complex prompts needed.

Extraction Format

Standard Structure

{
  "extractions": [
    {
      "类别": "提取的文本",
      "类别_attributes": {
        "key": "value"
      }
    }
  ]
}

Entity Types

Common entity types (customize as needed):

• 人物 (Person): Names, roles, descriptions
• 地点 (Location): Physical/virtual locations, addresses
• 事件 (Event): Actions, occurrences, milestones
• 时间 (Time): Dates, time periods, durations
• 组织 (Organization): Companies, teams, groups
• 情感 (Emotion): Feelings, sentiments, reactions
• 关系 (Relationship): Connections between entities

Adding Attributes

Attributes provide structured metadata:

{
  "人物": "张三",
  "人物_attributes": {
    "role": "工程师",
    "department": "研发部"
  }
}

Relationships

Model relationships between entities:

{
  "关系": "张三是李四的经理",
  "关系_attributes": {
    "type": "管理关系",
    "source": "张三",
    "target": "李四",
    "direction": "manages"
  }
}

Workflow

Step 1: Understand Requirements

Ask clarifying questions:

• What entity types to extract?
• What attributes are important?
• Any relationships to capture?
• Domain-specific requirements?

Step 2: Create Example

Build ONE high-quality example showing:

• All entity types
• Attribute structure
• Relationship format (if needed)

Example text should be similar to target text (same domain, style, length).

Step 3: Build Prompt

[Optional: Brief instruction]

Examples
Q: [Example text]
A: [Example JSON with extractions]

Q: [Target text - can be chunked if long]
A:

Key principles:

• Keep instruction minimal or omit entirely
• One example is usually enough
• Example quality > quantity
• Match example domain to target domain

Step 4: Process Response

1. Parse JSON: Extract the extractions array
2. Align to source: Find each extraction in original text
3. Record positions: Store character offsets (start, end)
4. Handle chunks: If text was split, merge results

Step 5: Present Results

Format options:

• Grouped by type: List all people, then all events, etc.
• With attributes: Show entity + its metadata
• With positions: Link back to source text
• Visualization: Generate HTML with highlights

Advanced Patterns

Long Documents

For text > 3000 chars, split into chunks:

1. Chunk with overlap: Keep 200-300 chars overlap between chunks
2. Add context: Include previous chunk tail in prompt
3. Process in parallel: Send multiple chunks simultaneously
4. Merge results: Deduplicate overlapping extractions

Context injection:

[Previous text]: ...end of previous chunk...

Examples
Q: [Example]
A: [Example JSON]

Q: [Current chunk]
A:

Multi-pass Extraction

For higher recall, run extraction multiple times:

1. First pass: Extract obvious entities
2. Second pass: Extract with different temperature/prompt
3. Merge: Keep non-overlapping extractions from both passes

Domain-Specific Extraction

Medical records:

{
  "症状": "头痛",
  "症状_attributes": {
    "severity": "中度",
    "duration": "3天",
    "frequency": "间歇性"
  }
}

Legal documents:

{
  "条款": "违约责任",
  "条款_attributes": {
    "section": "第5条",
    "type": "义务性条款",
    "parties": ["甲方", "乙方"]
  }
}

Social media:

{
  "话题": "#AI技术",
  "话题_attributes": {
    "sentiment": "正面",
    "engagement": "高"
  }
}

Alignment Algorithm

After extraction, map each entity back to source text:

Exact Match

# Find exact text in source
position = source_text.find(extraction_text)
if position >= 0:
    char_interval = (position, position + len(extraction_text))

Fuzzy Match

When exact match fails:

1. Tokenize both texts
2. Find longest common subsequence
3. Calculate overlap ratio
4. Accept if ratio > 0.75

Handle Failures

• Mark as "not aligned" if no match found
• Still include in results (extraction is valid even without position)

Common Patterns

Pattern 1: Simple Entity List

Use case: Extract names, places, dates

Example:

{
  "extractions": [
    {"人物": "Alice", "人物_attributes": {}},
    {"地点": "Paris", "地点_attributes": {}},
    {"时间": "2024-01-15", "时间_attributes": {}}
  ]
}

Pattern 2: Rich Attributes

Use case: Detailed entity metadata

Example:

{
  "extractions": [
    {
      "产品": "iPhone 15",
      "产品_attributes": {
        "category": "手机",
        "price": "5999",
        "features": ["A17芯片", "钛金属边框"]
      }
    }
  ]
}

Pattern 3: Knowledge Graph

Use case: Build entity relationships

Example:

{
  "extractions": [
    {"人物": "张三", "人物_attributes": {"role": "CEO"}},
    {"人物": "李四", "人物_attributes": {"role": "CTO"}},
    {
      "关系": "张三管理李四",
      "关系_attributes": {
        "type": "reports_to",
        "source": "李四",
        "target": "张三"
      }
    }
  ]
}

Tips

Prompt Design

✅ Do:

• Use one clear example
• Match example to target domain
• Keep instructions minimal
• Show exact JSON format

❌ Don't:

• Write long instructions
• Use multiple conflicting examples
• Over-explain the task
• Add unnecessary fields

Example Quality

Good example:

• Realistic text from target domain
• Shows all entity types
• Demonstrates attribute structure
• Similar length to target

Bad example:

• Generic/artificial text
• Missing entity types
• Inconsistent format
• Wrong domain

Attribute Design

Structured attributes (preferred):

"人物_attributes": {
  "age": "30",
  "occupation": "engineer"
}

Unstructured attributes (avoid):

"人物_attributes": {
  "info": "30 years old engineer"
}

Error Handling

• Invalid JSON: Retry with clearer example
• Missing entities: Try multi-pass extraction
• Wrong format: Check example matches desired output
• Hallucinations: Verify extractions exist in source text

Output Formats

Console Display

提取结果:
============================================================

【人物】(3 个)
------------------------------------------------------------
  • 张三
    描述: 项目经理
    属性: {"department": "研发部"}

【事件】(2 个)
------------------------------------------------------------
  • 项目启动会
    描述: 讨论Q1目标

JSON Export

{
  "document_id": "doc_001",
  "text": "原始文本...",
  "extractions": [
    {
      "extraction_class": "人物",
      "extraction_text": "张三",
      "char_interval": {"start": 10, "end": 12},
      "description": "项目经理",
      "attributes": {"department": "研发部"}
    }
  ]
}

HTML Visualization

Generate interactive HTML with:

• Highlighted entities (color-coded by type)
• Hover tooltips showing attributes
• Click to see details
• Filter by entity type

Reference

See EXAMPLES.md for complete working examples across different domains.