S3 Compatible Storage Patterns

For non-AWS providers, always require endpoint_url / endpoint override — the SDK defaults to s3.amazonaws.com and will silently fail.

Step 2 — Client Configuration Checklist

Go (aws-sdk-go-v2)

cfg, _ := config.LoadDefaultConfig(ctx,
    config.WithRegion(os.Getenv("STORAGE_REGION")),           // e.g. "auto" for R2
    config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
        os.Getenv("STORAGE_ACCESS_KEY_ID"),
        os.Getenv("STORAGE_SECRET_ACCESS_KEY"),
        "",
    )),
    config.WithEndpointResolverWithOptions(
        aws.EndpointResolverWithOptionsFunc(func(service, region string, opts ...interface{}) (aws.Endpoint, error) {
            return aws.Endpoint{URL: os.Getenv("STORAGE_ENDPOINT_URL"), SigningRegion: region}, nil
        }),
    ),
)
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
    o.UsePathStyle = true  // Required for MinIO and Wasabi
})

Python (boto3)

import boto3
client = boto3.client(
    "s3",
    region_name=os.environ["STORAGE_REGION"],
    endpoint_url=os.environ.get("STORAGE_ENDPOINT_URL"),   # None = default AWS
    aws_access_key_id=os.environ["STORAGE_ACCESS_KEY_ID"],
    aws_secret_access_key=os.environ["STORAGE_SECRET_ACCESS_KEY"],
)

Node.js (@aws-sdk/client-s3 v3)

import { S3Client } from "@aws-sdk/client-s3";
const s3 = new S3Client({
  region: process.env.STORAGE_REGION,
  endpoint: process.env.STORAGE_ENDPOINT_URL,   // omit for AWS
  credentials: {
    accessKeyId: process.env.STORAGE_ACCESS_KEY_ID!,
    secretAccessKey: process.env.STORAGE_SECRET_ACCESS_KEY!,
  },
  forcePathStyle: true,  // Required for MinIO and Wasabi
});

Checklist:

• Credentials loaded from environment variables — never hardcoded
• endpoint_url / endpoint configurable per environment
• use_path_style / forcePathStyle set to true for MinIO and Wasabi
• Region set to "auto" for R2, actual AWS region for S3
• Single client instance created at startup and reused (not per-request)

Step 3 — Multipart Upload Pattern

Use multipart upload for files >5 MB. Never buffer the entire file in memory.

Minimum part size: 5 MB (except the last part)
Maximum parts:     10,000
Maximum object:    5 TB

Go — use the Transfer Manager:

uploader := manager.NewUploader(client, func(u *manager.Uploader) {
    u.PartSize = 10 * 1024 * 1024  // 10 MB parts
    u.Concurrency = 3
})
_, err := uploader.Upload(ctx, &s3.PutObjectInput{
    Bucket:      aws.String(bucket),
    Key:         aws.String(key),
    Body:        reader,
    ContentType: aws.String(contentType),
})

• Use SDK-managed multipart upload — do not implement the 5-step multipart protocol manually
• Stream from source (disk/network) — do not io.ReadAll into memory
• Pass ContentType on every upload — do not rely on auto-detection

Step 4 — Object Key Design

Rules:

• Object key is always server-generated — never derived from the client-supplied filename
• Use URL-safe characters only ([a-z0-9_\-/.])
• No leading slashes (breaks some SDKs)
• Max key length: 1024 bytes

Step 5 — CORS Configuration

Required for browser direct-to-storage uploads (presigned PUT).

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://app.yourproduct.com"],
      "AllowedMethods": ["PUT", "GET", "HEAD"],
      "AllowedHeaders": ["Content-Type", "Content-Length", "x-amz-*"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 3600
    }
  ]
}

• AllowedOrigins is an explicit allowlist — never "*" on a private bucket
• Only PUT and GET allowed — never DELETE from the browser
• ETag exposed so the client can confirm multipart assembly
• Apply via IaC (Terraform / Pulumi) — not via console ad-hoc

Step 6 — Provider-Specific Gotchas

Quality Checks

• Client is a singleton — one instance per application lifecycle
• Credentials come from environment — not hardcoded or committed
• forcePathStyle set correctly per provider
• Multipart upload used for files >5 MB
• Object keys are UUID-based and server-generated
• CORS config applied via IaC with explicit origin allowlist
• Provider-specific endpoint and region configured correctly
• Client configuration is environment-specific (prod/staging/dev each have own bucket + creds)

After Completion

Produce:

1. A ready-to-use storage client module for the chosen language/provider
2. A CORS configuration JSON/YAML snippet
3. An environment variable reference table (STORAGE_ENDPOINT_URL, STORAGE_ACCESS_KEY_ID, etc.)
4. A multipart upload helper function if files >5 MB are expected