Data Processing

When to Trigger

Activate this skill when user input contains:

• Keywords: data processing, data cleaning, data transformation, data analysis, dataframe, CSV, Excel, JSON, ETL, pipeline, wrangling
• Contexts: Loading datasets, cleaning messy data, transforming columns, aggregations, joins, merging data, filtering, sorting, memory optimization
• Frameworks: Pandas, Polars, NumPy, PyArrow, Dask, Vaex

Review Workflow

1. Identify the data source format and size.
2. Choose the right library (Pandas for small/medium, Polars for large, NumPy for numerical).
3. Apply vectorized operations and dtype optimization.
4. Handle missing data appropriately.
5. Validate results with assertions or sanity checks.

Scope

• Pandas (small/medium datasets).
• Polars (large datasets/performance).
• NumPy (numerical operations).

Best Practices

• Vectorization over loops.
• Memory efficiency (dtype optimization).
• Handle missing data appropriately.

Checklists

Data Loading Checklist

• Chosen appropriate library for dataset size
• Specified column dtypes to reduce memory usage
• Parsed dates at load time
• Verified row count and schema after loading

Data Cleaning Checklist

• Checked for null/missing values
• Identified and handled duplicates
• Validated data types per column
• Standardized string formats (case, whitespace)
• Handled outliers appropriately

Data Transformation Checklist

• Used vectorized operations (no loops)
• Optimized dtypes (int32 vs int64, category for low-cardinality strings)
• Used method chaining for readability
• Verified output shape and content

Performance Checklist

• Profiled memory usage with df.info(memory_usage='deep')
• Used chunking for large files
• Considered Polars for datasets > 1GB
• Avoided unnecessary copies

OpenCode Usage Examples

Clean a messy CSV:

Clean this CSV file: data/raw/sales.csv — remove duplicates, fill missing values in the 'revenue' column with the median, convert 'date' to datetime, and save to data/clean/sales.csv. Use Pandas.

Convert Pandas to Polars for performance:

Convert this Pandas data processing script to Polars for better performance on large datasets: scripts/transform.py

Optimize memory usage:

Analyze and optimize memory usage of the dataframe in scripts/load_data.py — downcast integers, use category dtype for low-cardinality strings.

Create a data pipeline:

Create a Python data pipeline that reads all CSV files from data/input/, applies these transformations: remove nulls, normalize numeric columns, add a 'processed_at' timestamp column, and writes to data/output/ as Parquet.

NumPy numerical analysis:

Write a NumPy function that computes the cosine similarity matrix between all pairs of vectors in a 2D array of shape (n, d). Use vectorized operations only.

Following Agent Skills specification for progressive disclosure and best practices.