This skill should be used when users need to create or fix Redpanda Connect pipeline configurations. Trigger when users mention "config", "pipeline", "YAML", "create a config", "fix my config", "validate my pipeline", or describe a streaming pipeline need like "read from Kafka and write to S3".
Create working, validated Redpanda Connect configurations from scratch or repair existing configurations that have issues.
This skill REQUIRES skills: component-search, bloblang-authoring.
Deliver a complete, valid YAML configuration that passes validation and meets the user's requirements. Whether starting from a description or fixing a broken config, the result must be production-ready with properly secured credentials.
Handle Two Scenarios: Creation - User provides description like "Read from Kafka on localhost:9092 topic 'events' to stdout" Repair - User provides config file path and optional error context
This skill focuses ONLY on pipeline configuration orchestration and validation.
Skill Delegation:
NEVER directly use component-search or bloblang-authoring tools.
component-search skill when it is unclear which components to use OR when you need component configuration detailsbloblang-authoring skill when creating or fixing Bloblang transformations and NEVER write Bloblang yourselfThis skill requires: rpk, rpk connect.
See the SETUP for installation instructions.
Generates YAML configuration template from component expression. Useful for quickly creating first pipeline draft.
# Usage:
rpk connect create [--small] <input>,...[/<processor>,...]/<output>,...
# Examples:
rpk connect create stdin/bloblang,awk/nats
rpk connect create file,http_server/protobuf/http_client # Multiple inputs
rpk connect create kafka_franz/stdout # Only input and output, no processors
rpk connect create --small stdin/bloblang/stdout # Minimal config, omit advanced fields
inputs/processors/outputs separated by /,--small flag omits advanced fieldsUse the component-search skill's Online Component Documentation tool to look up detailed configuration information for any Redpanda Connect component containing usage examples, field descriptions, and best practices.
Validates Redpanda Connect pipeline configurations.
# Usage:
rpk connect lint [--env-file <.env>] <pipeline.yaml>
# Examples:
rpk connect lint --env-file ./.env ./pipeline.yaml
rpk connect lint pipeline-without-secrets.yaml
pipeline.yaml)--env-file flag provides .env file for environment variable substitution0 indicates success, non-zero indicates validation failuresExecutes Redpanda Connect pipeline to test end-to-end functionality.
# Usage:
rpk connect run [--log.level DEBUG] --env-file <.env> <pipeline.yaml>
# Examples:
rpk connect run pipeline-without-secrets.yaml
rpk connect run --env-file ./.env ./pipeline.yaml # With secrets
rpk connect run --log.level DEBUG --env-file ./.env ./pipeline.yaml # With debug logging
pipeline.yaml)--env-file flag provides dotenv file for environment variable substitution--log.level DEBUG enables detailed logging for troubleshooting connection and processing issuesTest pipeline logic with stdin/stdout before connecting to real systems.
Especially useful for validating routing logic, error handling, and transformations.
Example: Content-based routing