CDP Browser Automation & Deep Research

Navigation

Interaction (ref-based from snapshots)

Server

Deep Research Workflows

The power of this tool is combining commands into multi-step research flows. The LLM should think of itself as a researcher with a browser, not just a command executor.

Multi-platform research strategy

For any research topic, use multiple sources to get depth and recency:

1. Google Search — broad discovery

node cdp.mjs -e <ep> search "topic keyword phrase"

Scan results, pick the most relevant 2-3 to read in depth.

2. X/Twitter — most recent, real-time sentiment

node cdp.mjs -e <ep> navigate "https://x.com/search?q=topic%20keyword&f=live"
node cdp.mjs -e <ep> content     # read the tweets
node cdp.mjs -e <ep> scroll down  # load more tweets
node cdp.mjs -e <ep> content     # read the newly loaded tweets

X gives the freshest takes — breaking news, expert opinions, community reactions. Use f=live for chronological, f=top for popular.

3. Reddit — community discussion, diverse opinions

node cdp.mjs -e <ep> navigate "https://www.reddit.com/search/?q=topic&sort=new"
node cdp.mjs -e <ep> content

Then click into threads for deep discussion. Reddit excels at nuanced debate and first-hand experiences.

4. Quora — expert answers, structured Q&A

node cdp.mjs -e <ep> navigate "https://www.quora.com/search?q=topic"
node cdp.mjs -e <ep> content

Good for "how does X work" and "what is the best approach to Y" questions.

5. Medium / Substack — long-form analysis

node cdp.mjs -e <ep> search "topic site:medium.com OR site:substack.com"

For in-depth articles, technical deep-dives, and expert analysis.

6. Academic — papers and citations

node cdp.mjs -e <ep> scholar "topic keyword phrase"

The scholar command extracts structured results: title, authors, year, journal, citation count, and snippet. Use it instead of manually navigating to scholar.google.com. Follow up by navigating to specific papers for full abstracts via content.

Research depth patterns

Surface scan (quick overview):

1. search the topic
2. content on top 2-3 results
3. Synthesize findings

Medium depth (thorough understanding):

1. search on Google
2. navigate to X for recent discussion
3. navigate to Reddit for community perspective
4. Read 3-5 articles with content
5. Follow key links to primary sources
6. Cross-reference claims across sources

Deep dive (comprehensive research):

1. All of the above, plus:
2. Follow citation chains — links to find references, read those too
3. scroll through X/Reddit to get more data points
4. extract structured data (dates, numbers, names)
5. Check multiple search queries (rephrase, use different keywords)
6. Verify claims: find the primary source, not just reports about it
7. Track contradictions between sources — note which sources disagree and why

Working with X/Twitter

X is your real-time pulse. Key patterns:

# Search by topic (live/recent)
navigate "https://x.com/search?q=topic&f=live"

# Search by topic (top/popular)
navigate "https://x.com/search?q=topic&f=top"

# Filter by user
navigate "https://x.com/search?q=from:username topic"

# Filter by date
navigate "https://x.com/search?q=topic since:2026-01-01 until:2026-03-01"

# Read a specific thread
navigate "https://x.com/user/status/123456"
content

After navigating to search results, use content to read, scroll down + content to load more, and snapshot + click to open specific tweets.

Working with Reddit

# Search across all subreddits
navigate "https://www.reddit.com/search/?q=topic&sort=relevance"

# Search within a subreddit
navigate "https://www.reddit.com/r/technology/search/?q=topic&restrict_sr=1"

# Read a thread (old.reddit for cleaner content extraction)
navigate "https://old.reddit.com/r/subreddit/comments/id/title"
content

Site exploration (docs, wikis, any website)

When researching a specific website — especially documentation sites, wikis, or knowledge bases — use discover and crawl to systematically map and read the site rather than guessing at URLs.

Pattern: Documentation deep-dive

# 1. Navigate to the docs root
node cdp.mjs -e <ep> navigate "https://docs.example.com"

# 2. Discover all navigation — see what sections exist
node cdp.mjs -e <ep> discover

# 3. Crawl the top pages to get an overview
node cdp.mjs -e <ep> crawl 10

# 4. Go deeper into specific sections that matter
node cdp.mjs -e <ep> navigate "https://docs.example.com/api/auth"
node cdp.mjs -e <ep> discover   # find sub-navigation within this section
node cdp.mjs -e <ep> content    # read the page in detail

Pattern: Exploring an unfamiliar site

# 1. Land on the homepage
node cdp.mjs -e <ep> navigate "https://example.com"

# 2. Discover categorizes links by location (nav, sidebar, footer, content)
node cdp.mjs -e <ep> discover

# 3. Follow the most relevant nav/sidebar links
node cdp.mjs -e <ep> navigate "<interesting-link-from-discover>"
node cdp.mjs -e <ep> content

discover returns links grouped by where they appear on the page (nav menus, sidebars, breadcrumbs, footer, in-content), with internal/external split. This tells you the site's structure at a glance. crawl then visits internal pages automatically, giving you a title + summary of each so you know which pages deserve a full content read.

Accumulating findings

Use the memo command to save key findings as you research. This creates a per-session scratchpad:

# Save a finding under a topic key
node cdp.mjs -e <ep> memo "climate-risk" "Sea levels projected to rise 0.3-1.0m by 2100 (IPCC AR6)"
node cdp.mjs -e <ep> memo "climate-risk" "Some studies suggest up to 2m under worst case (DeConto 2016)"
node cdp.mjs -e <ep> memo "counter-evidence" "Antarctic ice sheet more stable than models predict (Whitehouse 2023)"

# Recall all notes for a topic
node cdp.mjs -e <ep> recall "climate-risk"

# Recall everything saved in this session
node cdp.mjs -e <ep> recall

Memos auto-capture the source URL and timestamp. Use them to build a structured evidence base before synthesis.

The LLM should:

1. After reading each source, memo key claims with the topic as key
2. After reading multiple sources, identify agreements and contradictions
3. Present a synthesis with citations, not just a dump of what each source said
4. Flag anything that only one source claims and couldn't be independently verified

Handling blocked sites

The navigate and content commands auto-detect common blocks:

• CLOUDFLARE — site is behind Cloudflare challenge. Skip and find an alternative source
• LOGIN_REQUIRED — site needs authentication (common: X/Twitter, LinkedIn). Use Google search snippets instead, or use search "topic site:x.com" to get cached content
• PAYWALL — content is paywalled. Try Google cache: navigate "https://webcache.googleusercontent.com/search?q=cache:<url>"
• ACCESS_DENIED — 403/forbidden. Site is actively blocking
• EMPTY_PAGE — page loaded but has no meaningful content

When a block is detected, don't waste time trying to extract content. Move to the next source immediately. For critical sources that are blocked, try:

1. Google cache version
2. Archive.org: navigate "https://web.archive.org/web/<url>"
3. Alternative source covering the same topic

Adversarial research — finding what's wrong with your thesis

Good research isn't just finding evidence that supports a conclusion. It's actively trying to disprove it. For any research topic, follow the adversarial pattern:

Step 1: Steelman the thesis

node cdp.mjs -e <ep> search "evidence supporting [thesis]"
node cdp.mjs -e <ep> scholar "[thesis] empirical evidence"

Step 2: Steelman the opposite

node cdp.mjs -e <ep> search "evidence against [thesis]" OR "why [thesis] is wrong" OR "[thesis] criticism"
node cdp.mjs -e <ep> scholar "[thesis] critique" OR "[thesis] limitations"

Step 3: Find the survivors AND the dead

# Don't just read success stories — look for failures
node cdp.mjs -e <ep> search "[topic] failures" OR "[topic] risks" OR "why [topic] doesn't work"

Step 4: Check for survivorship bias For every success example, ask: "How many people did the same thing and failed?" If you can't find failure data, flag it as survivorship bias in your report.

Step 5: Assign confidence based on counter-evidence

• 90-100%: Multiple independent sources agree, counter-evidence is weak or addressed
• 70-89%: Sources mostly agree but meaningful counter-arguments exist
• 50-69%: Sources are split or evidence is thin
• <50%: Counter-evidence is as strong or stronger than supporting evidence

How Refs Work

The snapshot command returns an accessibility tree with element refs like [ref=e14]. These refs persist within the server session. Use them for click, fill, type, etc. If a ref stops working (page changed), run snapshot again.

Parallel Agents

Multiple agents can work simultaneously on different endpoints:

# Agent 1: research on Google/Wikipedia
node cdp.mjs -e http://10.0.0.1:9322 search "quantum computing breakthroughs"

# Agent 2: research on X/Twitter
node cdp.mjs -e http://10.0.0.1:9323 navigate "https://x.com/search?q=quantum computing&f=live"

# Agent 3: research on Reddit
node cdp.mjs -e http://10.0.0.1:9324 navigate "https://reddit.com/search/?q=quantum computing"

Each endpoint auto-starts its own server with lock-based protection against race conditions.

Important Behaviors

• Use content for reading, snapshot for interacting. Content extracts clean text. Snapshot shows interactive elements with refs.
• Scroll before concluding. Many pages lazy-load content. Scroll down 2-3 times with content between scrolls.
• Verify, don't trust. Cross-reference claims across sources. Follow links to primary sources.
• Summarize results. Tell the user what you found in plain language with source attribution. Don't paste raw output.
• Ask before credentials. Never fill passwords unless explicitly provided.