Data Pipeline Setup

1. Keep farm-specific state in a DataPipeline record, not in the generated vendor skill.
2. Keep reusable navigation knowledge in skills/pipeline-{vendor}/SKILL.md.
3. Never store passwords, session cookies, or raw secrets in the repo.
4. In OSS, generated vendor skills stay local unless the farmer explicitly submits them for review.
5. In cloud deployments, generated vendor skills should remain reviewable artifacts. Preserve enough structure that an automated review pipeline can diff and promote them later.

Live Setup Procedure

Phase 1: Provision

1. Confirm the vendor name and login URL.
2. Resolve the active farm_id.
3. Derive a profile name like {vendor_slug}-{farm_id}.
4. Use Firecrawl CLI yourself to start the session, for example:

firecrawl scrape "https://example.com/login" --profile "{vendor_slug}-{farm_id}" --json

5. Capture the returned scrapeId.
6. Open the interactive handoff from the same scrape session, for example:

firecrawl interact --scrape-id "<scrape-id>" --prompt "Do not change the page. Just say ready for login handoff." --json

7. Give the farmer the returned live view URL.
8. Wait for explicit confirmation before continuing.

Important:

• Keep working against the same Firecrawl session after the farmer says they are logged in.
• Do not tear down the session and start over unless the session is actually gone or the saved profile truly failed to persist.

Phase 2: Explore

1. After the farmer says they are logged in, inspect the same active session first.
2. Ask Firecrawl to report whether the current page is authenticated before assuming anything.
3. If authenticated, continue exploring from that same scrapeId.
4. Scrape first, then use interact only when navigation or clicks are required.
5. Inspect the authenticated site to identify:
   • the key data pages,
   • the useful entities and fields,
   • the URLs that should be revisited on cron runs,
   • any filters, tabs, pagination, or gotchas.
6. Summarize what data seems available before assuming it is useful.

If skills/pipeline-{vendor}/SKILL.md already exists, use it as the starting point and only re-explore when the site appears to have changed.

Phase 3: Converse

1. Tell the farmer what data the agent found in plain language.
2. Ask which signals matter for the morning brief and how often they should be collected.
3. Confirm the intended cadence. Default to daily unless the data is unusually time-sensitive or static.
4. Confirm the observation mapping before enabling anything.

Only collect data the farmer understands and wants.

Phase 4: Persist

Once the site is understood, call save_data_pipeline with the learned configuration.

The saved pipeline should include:

• vendor name,
• Firecrawl profile ID,
• login URL,
• target URLs,
• extraction prompts,
• observation source and tags,
• schedule,
• reusable login and navigation notes,
• generated vendor skill hash.

Keep the generated skill reusable across farms. Do not bake in farm IDs, raw account-specific secrets, or farmer-specific preferences that belong in the store.

Phase 5: Verify

1. Run one test collection with run_data_pipeline before enabling the cron.
2. Show the farmer a sample of the resulting observation.
3. Ask whether the collected signal is actually useful.
4. Only enable the recurring run after the farmer confirms it looks right.

Generated Vendor Skill Template

Use this structure for skills/pipeline-{vendor}/SKILL.md:

---