Redpanda Migrator

• Redpanda Connect deployment type: Redpanda Cloud, Redpanda Serverless, Local, or Custom?
  • If Redpanda Cloud or Redpanda Serverless: Use environment variable secrets for credentials
  • If Local or Custom: Use direct credentials in configuration

Important: This determines whether credentials should be:

• Cloud/Serverless: Stored as secrets (${SECRET_NAME} format)
• Local/Custom: Embedded directly in YAML

Source and Destination Clusters

Required Information:

• Source cluster type: Apache Kafka, Redpanda, Confluent Cloud, AWS MSK, Azure Event Hubs, or other
• Destination cluster type: same options as source
• Bootstrap servers for both clusters (format: host:port)

Authentication (ask if not provided):

• Authentication method: SASL/PLAIN, SASL/SCRAM-SHA-256, SASL/SCRAM-SHA-512, AWS_MSK_IAM, or none
• TLS/SSL enabled: yes/no
• If authentication required: credentials will be needed (username/password or IAM role)

Migration Scope

Ask the user:

• Data migration: yes/no
• Schema migration: yes/no (requires Schema Registry URLs)
• Consumer offset translation: yes/no
• ACL migration: yes/no

Topic and Schema Filtering

Ask the user:

• Migrate all topics or filter specific topics?
  • If filtering: provide topic patterns (regex) or specific topic names
  • Common pattern: ^[^_] (excludes internal topics starting with _)
• Migrate all schemas or filter specific subjects?
  • If filtering: provide subject inclusion/exclusion patterns

Replication Settings

Ask the user:

• Keep source cluster replication factor or override?
  • If override: specify new replication factor

Additional Configuration

Optional questions (ask if relevant):

• Consumer group name for the migrator (default: migrator_bundle)
• Start from oldest messages or latest? (default: oldest)
• Include soft-deleted schemas? (default: true)
• Max in-flight messages for ordering preservation? (default: 1)

Configuration Generation Process

After gathering requirements:

1. Read the complete config specification:

   file_read: references/config-spec.md
   

2. Generate the YAML using the template structure from config-spec.md

3. Include these essential sections:

   • input.redpanda_migrator_bundle.redpanda_migrator - source cluster config
   • input.redpanda_migrator_bundle.schema_registry - source schema registry (if schema migration enabled)
   • output.redpanda_migrator_bundle.redpanda_migrator - destination cluster config
   • output.redpanda_migrator_bundle.schema_registry - destination schema registry (if schema migration enabled)
   • metrics.prometheus - monitoring configuration

4. Apply user requirements:

   • Set seed_brokers for source and destination
   • Configure authentication (SASL, TLS)
   • Set topic filters
   • Configure schema registry if needed
   • Enable/disable consumer offset translation
   • Enable/disable ACL migration
   • Set replication factor override

5. Add comments explaining critical settings like:

   • Topic filtering patterns
   • Authentication configuration
   • Max in-flight for ordering
   • Consumer group offset translation behavior

Common Migration Scenarios

Reference these patterns for typical requests:

Kafka to Redpanda Cloud:

• Source: Apache Kafka with standard authentication
• Destination: Redpanda Cloud with SASL/SCRAM-SHA-256
• Usually includes: data + schemas + consumer offsets + ACLs

AWS MSK to Redpanda Cloud:

• Source: AWS MSK with SCRAM-SHA-512 or IAM authentication
• Destination: Redpanda Cloud Dedicated with SASL/SCRAM-SHA-256
• Usually includes: data + consumer offsets, optional schema migration
• May need topic_replication_factor override
• Use enable_renegotiation: true for TLS if needed

Confluent Cloud to Redpanda Cloud:

• Source: Confluent Cloud with PLAIN authentication (API keys)
• Destination: Redpanda Cloud with SASL/SCRAM-SHA-256
• Requires enable_renegotiation: true for Confluent Cloud TLS
• Usually data-only, no schema migration
• Fast consumer offset sync (interval: 10s)

Redpanda Dedicated to Redpanda Serverless:

• Source: Redpanda Cloud Dedicated (.fmc.prd.cloud.redpanda.com)
• Destination: Redpanda Cloud Serverless (.mpx.prd.cloud.redpanda.com)
• CRITICAL: Set serverless: true in output
• Use consumer_groups.exclude to filter internal consumer groups
• Both data-only and data+schema migrations supported
• Full TLS and authentication for both clusters

Redpanda to Redpanda (Data-Only):

• Source: Redpanda cluster (local or BYOC)
• Destination: Another Redpanda cluster
• Data only with consumer offset translation
• Use metadata_max_age: 30s for faster regex topic discovery
• Configure consumer_groups.interval for offset sync frequency
• Disable schema registry with enabled: false

Redpanda to Redpanda (with Schemas):

• Source: Redpanda cluster with Schema Registry
• Destination: Another Redpanda cluster with Schema Registry
• One-time schema sync: Set enabled: true (no interval)
• Periodic schema sync: Set interval: 10s or desired frequency
• Use versions: latest for faster migration or versions: all for full history
• Filter schemas with include or exclude patterns

Data-only migration:

• Minimal config with just data streaming
• No schema registry configuration (or explicitly disabled)
• Consumer offset translation optional
• ACL migration disabled

Important Notes

• Message Ordering: The Redpanda Migrator automatically preserves message ordering at the partition level (max_in_flight=1 is hardcoded)
• Secrets Management: When deploying to Redpanda Cloud or Serverless, always use environment variable secrets for credentials
  • Format: ${SECRET_NAME} instead of plain text values
  • Required for: seed_brokers, username, password, schema_registry URLs
  • Documentation: https://docs.redpanda.com/redpanda-cloud/develop/connect/configuration/secret-management/
• Schema Registry Mode: Destination Schema Registry must be in READWRITE or IMPORT mode
• Consumer Group Offsets: Require identical partition counts between source and destination
• ACL Safety: ALLOW WRITE permissions are excluded, ALLOW ALL is downgraded to ALLOW READ
• Topic Creation: Destination topics are created automatically if they don't exist
• Monitoring: The input_redpanda_migrator_lag metric tracks migration progress

Secrets Management for Cloud Deployments

When Redpanda Connect deployment type is Redpanda Cloud or Redpanda Serverless, use environment variable secrets for all sensitive credentials.

Standard Secret Names by Platform

Redpanda Cloud/Serverless:

• ${REDPANDA_BROKERS} - Seed brokers
• ${REDPANDA_USER} - Username
• ${REDPANDA_USER_PWD} - Password

Confluent Cloud:

• ${CC_BROKERS} - Bootstrap brokers
• ${CC_USER} - API key (username)
• ${CC_USER_PWD} - API secret (password)

AWS MSK / Custom Kafka:

• ${KAFKA_BROKERS} - Bootstrap brokers
• ${KAFKA_USER} - Username
• ${KAFKA_USER_PWD} - Password

Example: Redpanda Cloud to Redpanda Cloud