Create a custom Dagster Component with demo mode support, realistic asset structure, and optional custom scaffolder using the dg CLI. Use this skill if there is no Component included in an existing integration or if Dagster does not have the integration.
This skill automates the creation and validation of a new custom Dagster component using the dg CLI tool with uv as a package manager. It incorporates demo mode functionality for creating realistic demonstrations that can run locally without external dependencies. The documentation for creating good components can be found here https://docs.dagster.io/guides/build/components/creating-new-components/creating-and-registering-a-component and here https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/components/dbt_project/component.py for a complex example of a component.
When invoked, this skill will:
dg scaffold component ComponentNamebuild_defs() function with both real and demo mode implementationsdemo_mode boolean flag in the component YAML for toggling between real and local demo implementationsdg scaffold defs my_module.components.ComponentName my_component commanddg check defs and dg list defs to ensure that the expected component instances are all loaded.Before running this skill, ensure:
uv is installed (check with uv --version)Ask the user for:
MyDagsterComponent. Validate that:
Use dg to create the component
uv run dg scaffold component <ComponentName>
This will:
defs/componentscomponent_name.py fileFill in the build_defs() function in the component file. The component should:
demo_mode parameter in the component params (default: False)defs/ folder in a resources.py file, using dg scaffold defs dagster.resource resources.pyassets field in the YAML that describes what assets are used in the underlying component. See https://dagster.io/blog/dsls-to-the-rescue for best practices in how to design a good DSL. Refer to https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-dbt/dagster_dbt/components/dbt_project/component.py and https://github.com/dagster-io/dagster/blob/master/python_modules/libraries/dagster-fivetran/dagster_fivetran/components/workspace_component/component.py for two reference architectures for good component design with mutli-assets.kinds argument to indicate technologies in useExample asset structure:
CRITICAL: When creating a custom component, consider what will consume your component's assets. The asset keys you generate should align with downstream component expectations to avoid requiring per-asset configuration.
Your component (upstream) should generate asset keys in a structure that downstream components naturally reference. This eliminates the need for meta.dagster.asset_key or complex translation configuration.
If dbt will consume your assets:
["<source_name>", "<table_name>"]["fivetran_raw", "customers"] or ["api_raw", "users"]source('fivetran_raw', 'customers')If custom Dagster assets will consume them:
deps["category", "name"])["system", "subsystem", "type", "name"] unless necessaryIf another integration component will consume them:
If your assets are intermediate and consumed by your own component:
["raw", "table"] → ["processed", "table"] → ["enriched", "table"]import dagster as dg
class APIIngestionComponent(dg.Component, dg.Model, dg.Resolvable):
"""Ingests data from REST APIs."""
api_endpoint: str
tables: list[str]
demo_mode: bool = False
def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:
assets = []
for table in self.tables:
# Design key for dbt consumption: ["api_raw", "table_name"]
# NOT: ["api", "ingestion", "raw", "table_name"]
@dg.asset(
key=dg.AssetKey(["api_raw", table]), # ← Flattened for easy downstream reference
kinds={"api", "python"},
)
def ingest_table(context: dg.AssetExecutionContext):
if self.demo_mode:
context.log.info(f"Demo mode: Mocking API call for {table}")
return {"status": "demo", "rows": 100}
else:
# Real API call
pass
assets.append(ingest_table)
return dg.Definitions(assets=assets)
Result: dbt can reference these assets naturally:
# sources.yml