Sagemaker Patterns

Rules

• entry_point must be a file inside source_dir; do not pass absolute paths.
• All hyperparameters must be JSON-serializable. No Python objects, no enums — serialize enums to their .value before passing.
• base_job_name follows the pattern queryforge-<component>.
• Always tag jobs with {"Key": "project", "Value": "queryforge"}.

2. Processing Step — ScriptProcessor

# src/queryforge/pipeline/steps/datagen_step.py
from sagemaker.processing import ScriptProcessor, ProcessingInput, ProcessingOutput
from shared.schemas.config import PipelineConfig

def build_datagen_processor(config: PipelineConfig) -> ScriptProcessor:
    """Build a processor for the dataset generation step."""
    return ScriptProcessor(
        role=config.execution_role_arn,
        image_uri=config.processing_image_uri,   # versioned URI from config
        instance_type=config.processing_instance_type,
        instance_count=1,
        base_job_name="queryforge-datagen",
        tags=[{"Key": "project", "Value": "queryforge"}],
    )

ProcessingInput / ProcessingOutput conventions

3. SageMaker Pipeline — Structure

# src/queryforge/pipeline/finetune_pipeline.py
import sagemaker
from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.parameters import ParameterString, ParameterFloat

# --- Pipeline parameters are declared at the top, never inside step definitions ---
schema_name   = ParameterString(name="SchemaName",   default_value="orders")
schema_version = ParameterString(name="SchemaVersion", default_value="v1")
run_id        = ParameterString(name="RunId",         default_value="")
accuracy_threshold = ParameterFloat(name="AccuracyThreshold", default_value=0.75)

# ... build steps using parameters above ...

pipeline = Pipeline(
    name="QueryForgeFinetuning",
    parameters=[schema_name, schema_version, run_id, accuracy_threshold],
    steps=[datagen_step, train_step, eval_step, condition_step],
    sagemaker_session=sagemaker.Session(),
)

Step ordering rules

1. ProcessingStep — datagen: generates JSONL dataset
2. TrainingStep — train: runs QLoRA fine-tuning
3. ProcessingStep — evaluate: measures Execution Accuracy
4. ConditionStep — gate: compares accuracy against accuracy_threshold
5. RegisterModel — register: only runs if condition is met

4. ConditionStep — Model Registration Gate

from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.functions import JsonGet

accuracy_condition = ConditionGreaterThanOrEqualTo(
    left=JsonGet(
        step_name=eval_step.name,
        property_file=eval_output,
        json_path="execution_accuracy",
    ),
    right=accuracy_threshold,
)

gate_step = ConditionStep(
    name="AccuracyGate",
    conditions=[accuracy_condition],
    if_steps=[register_step],
    else_steps=[],
)

5. Model Registry — Registration

# src/queryforge/registry/register.py
from shared.schemas.registry import ModelRegistryMetadata

def register_model(metadata: ModelRegistryMetadata, model_data_url: str, config) -> None:
    """Register a model version with the SageMaker Model Registry.

    Args:
        metadata: Validated model metadata including schema name, version, and metrics.
        model_data_url: S3 URI of the model artifact (tar.gz).
        config: Validated pipeline configuration.
    """
    model_package_group_name = f"queryforge-{metadata.schema_name}"
    # ... registration code using boto3 or sagemaker SDK ...

Metadata requirements

Every create_model_package call must include all fields from ModelRegistryMetadata as CustomerMetadataProperties. See shared/schemas/registry.py for the full contract.

Useful commands

# Upsert a pipeline definition (create or update)
python scripts/run_pipeline.py --config config/pipeline.yaml --action upsert

# Start a pipeline execution
python scripts/run_pipeline.py --config config/pipeline.yaml --action start \
    --schema-name orders --schema-version v1

# Describe a pipeline execution
python scripts/run_pipeline.py --action describe --execution-arn <arn>