Unstructured Medical Text Miner

pandas>=1.3.0
spacy>=3.4.0
scispacy>=0.5.1
radlex (for radiology terminology)
negspacy (for negation detection)

Example Usage

See ## Usage above for related details.

cd "20260318/scientific-skills/Evidence Insight/unstructured-medical-text-miner"
python -m py_compile scripts/main.py
python scripts/main.py --help

Example run plan:

1. Confirm the user input, output path, and any required config values.
2. Edit the in-file CONFIG block or documented parameters if the script uses fixed settings.
3. Run python scripts/main.py with the validated inputs.
4. Review the generated output and return the final artifact with any assumptions called out.

Implementation Details

See ## Workflow above for related details.

• Execution model: validate the request, choose the packaged workflow, and produce a bounded deliverable.
• Input controls: confirm the source files, scope limits, output format, and acceptance criteria before running any script.
• Primary implementation surface: scripts/__init__.py with additional helper scripts under scripts/.
• Reference guidance: references/ contains supporting rules, prompts, or checklists.
• Parameters to clarify first: input path, output path, scope filters, thresholds, and any domain-specific constraints.
• Output discipline: keep results reproducible, identify assumptions explicitly, and avoid undocumented side effects.

Quick Check

Use this command to verify that the packaged script entry point can be parsed before deeper execution.

python -m py_compile scripts/main.py

Audit-Ready Commands

Use these concrete commands for validation. They are intentionally self-contained and avoid placeholder paths.

python -m py_compile scripts/main.py
python scripts/main.py --help
python scripts/main.py -h

Workflow

1. Confirm the user objective, required inputs, and non-negotiable constraints before doing detailed work.
2. Validate that the request matches the documented scope and stop early if the task would require unsupported assumptions.
3. Use the packaged script path or the documented reasoning path with only the inputs that are actually available.
4. Return a structured result that separates assumptions, deliverables, risks, and unresolved items.
5. If execution fails or inputs are incomplete, switch to the fallback path and state exactly what blocked full completion.

Overview

Mine "text data" that has been long overlooked in MIMIC-IV, extracting unstructured diagnostic logic, order details, and progress notes.

Purpose

The MIMIC-IV database contains large amounts of structured data (vital signs, laboratory results, etc.), but its true clinical value is often hidden in unstructured text:

• Diagnostic reasoning chains in discharge summaries
• Subtle finding descriptions in imaging reports
• Treatment decision logic in progress notes
• Personalized medication considerations in orders

This Skill provides a complete text mining toolchain to transform raw medical text into analyzable structured insights.

Features

1. Text Extraction

• NOTEEVENTS: Extract clinical notes from MIMIC-IV NOTE module
• Radiology Reports: Extract imaging diagnostic text
• ECG Reports: Parse ECG interpretation text
• Discharge Summaries: Extract complete diagnostic and treatment course

2. Information Extraction

• Entity Recognition: Diseases, symptoms, medications, procedures, anatomical sites
• Relation Extraction: Medication-disease treatment relationships, symptom-disease diagnostic relationships
• Timeline Extraction: Event occurrence times, disease progression sequence
• Negation Detection: Identify negated clinical findings (e.g., "no fever")

3. Clinical Logic Parsing

• Diagnostic Reasoning Chain: Reasoning path from symptoms → examination → diagnosis
• Treatment Decision Tree: Clinical basis for medication selection and dosage adjustment
• Disease Progression: Disease progression and outcome descriptions

4. Structured Output

• FHIR-compatible clinical document format
• Knowledge graph-friendly triple format
• Temporal event sequences

Usage

from skills.unstructured_medical_text_miner.scripts.main import MedicalTextMiner

# Initialize miner
miner = MedicalTextMiner()

# Load MIMIC-IV note data
miner.load_notes(notes_path="path/to/noteevents.csv")

# Extract all text records for a specific patient
patient_texts = miner.get_patient_texts(subject_id=10000032)

# Execute complete information extraction
insights = miner.extract_insights(
    text=patient_texts,
    extract_entities=True,
    extract_relations=True,
    extract_timeline=True
)

Input

Data Sources

• MIMIC-IV NOTEEVENTS table (csv/parquet format)
• Discharge summary files
• Imaging report files
• Custom medical text

Field Requirements

Output

Entity Extraction Results

{
  "entities": [
    {
      "text": "acute myocardial infarction",
      "type": "DISEASE",
      "start": 156,
      "end": 183,
      "confidence": 0.94
    },
    {
      "text": "aspirin 81mg",
      "type": "MEDICATION",
      "start": 245,
      "end": 257,
      "attributes": {
        "dose": "81mg",
        "frequency": "daily"
      }
    }
  ]
}

Clinical Logic Graph

{
  "clinical_logic": {
    "presenting_complaint": "chest pain",
    "differential_diagnoses": ["ACS", "PE", "aortic dissection"],
    "workup": ["ECG", "troponin", "CTA chest"],
    "final_diagnosis": "STEMI",
    "treatment_plan": ["PCI", "dual antiplatelet"]
  }
}

Temporal Events

{
  "timeline": [
    {
      "time": "2020-03-15 08:30",
      "event": "admission",
      "description": "presented with chest pain"
    },
    {
      "time": "2020-03-15 09:15",
      "event": "ECG",
      "description": "ST elevation in V1-V4"
    }
  ]
}

Configuration

# config.yaml