R Package Refactor

1. Analyze existing code structure
2. Convert scripts to clean, testable functions
3. Add package infrastructure (following CEDA structure)
4. Add interactive mode (Shiny app)
5. Polish with documentation and tests

This approach is safer than "big bang" refactoring - we ensure code works before adding package complexity.

Key Principles:

• Functions First, Package Second: Ensure code works before adding package complexity
• Incremental progress: One phase at a time with user approval
• Test continuously: Verify each change works before proceeding
• Follow CEDA standards: Package structure, tidyverse style, interactive mode (Shiny), data conventions

See standards/principles.md and standards/r-style.md for detailed CEDA guidelines.

Configuration

Before starting, ask the user about:

1. Package name (if not already chosen)
2. Repository type (Ingestion or Analysis - see CEDA Project Structure)
   • Ingestion: Transforms raw data sources into clean, research-ready formats
   • Analysis: Produces enriched data, predictions, reports, visualizations
3. Documentation language (Dutch for README, English for code/comments per CEDA standards)
4. Current pain points (what motivated this refactoring?)
5. Interactive mode needed? (Shiny app for user-friendly interface - recommended per CEDA standards)

Phase 1: Project Discovery & Analysis

Goal: Understand the current codebase structure and design the transformation plan

Step 1: Analyze Directory Structure

# Discover project layout
rtk ls -R . | grep "\\.R$"

Identify:

• Script directories: Are scripts organized by stage (download, prepare, analyze)?
• Utility functions: Where are shared functions located?
• Data/config: Where is metadata, config, or reference data stored?
• Tests: Are there existing test scripts?
• Documentation: README, vignettes, or markdown docs?

Step 2: Catalog Scripts and Dependencies

For each R script:

1. Purpose: What does it do?
2. Dependencies: What packages/functions does it use?
3. Inputs/Outputs: Data files? Objects? Side effects?
4. Execution order: Which scripts run first?

Create a visual map:

scripts/download/ → scripts/process/ → scripts/export/
         ↓                 ↓                 ↓
      utils/helpers.R (used by all)

Step 3: Identify Function Extraction Opportunities

Look for:

• Repeated code patterns across scripts
• Scripts that are really functions (take input, return output, no side effects)
• Utility functions already defined in utils/ or helper files
• Configuration loading that should be packageized

Step 4: Design Package Architecture

Based on analysis, propose:

Package Name:

• Follow R conventions: lowercase, letters/numbers/periods only
• Short (5-10 chars), memorable, descriptive
• Check availability with available::available("pkgname")

File Organization (R/ directory):

See standards/project-structure.md for complete CEDA project structure guidelines.

For Ingestion repos:

• R/ingest_source.R - Read raw files
• R/decode_fields.R - Parse fixed-width, apply metadata
• R/validate_data.R - Quality checks
• R/export_data.R - Write Parquet + CSV output
• R/run_app.R - Launch Shiny app function
• R/package.R - Package-level documentation

For Analysis repos:

• R/prepare_data.R - Load and merge input data
• R/transform_data.R - Feature engineering, enrichment
• R/run_analysis.R - Core analysis/modeling
• R/create_plots.R - Visualization functions
• R/render_report.R - Report generation
• R/run_app.R - Launch Shiny app function
• R/package.R - Package-level documentation

Export Strategy:

• Exported (@export): Main pipeline functions, utilities users need, run_app()
• Internal (@keywords internal): Helper functions, low-level details

Directory Structure (CEDA standard):

• inst/app/ - Shiny app (app.R, config.yml)
• inst/metadata/ - Mapping tables, reference data, data_dictionary.csv
• inst/templates/ - Report templates (Quarto .qmd files)
• data/ - Example datasets for demos
  • data/01-raw/demo/ - Sample raw data (committed to git)
  • data/02-prepared/demo/ - Sample processed data
  • data/03-output/demo/ - Sample output
• man/ - Auto-generated roxygen2 docs
• tests/testthat/ - Unit tests
• vignettes/ - Usage documentation
• main.R - Pipeline orchestration script (NOT part of package)

Step 5: Present Analysis and Plan

Create a structured plan document showing:

1. Current structure (directory tree with file purposes)
2. Proposed package structure (R/ file organization)
3. Script → Function mapping (which scripts become which functions)
4. Export plan (what users will see)
5. Migration strategy (what stays, what moves, what changes)

Wait for user approval before proceeding to Phase 2

Phase 2: Create Package Infrastructure

Goal: Set up basic package structure without breaking existing code

Step 2.1: Initialize Package Structure

# Create DESCRIPTION file