PayPerCrawl Development Skill

Key Development Workflows

Adding a New Bot Signature

1. Edit includes/class-bot-detector.php
2. Add entry to $known_bots array: 'BotName' => ['company' => 'Company', 'type' => 'ai_crawler']
3. If bot has special pricing, update get_bot_multiplier() in backend/production-worker.js

Modifying Bot Detection Heuristics

Edit includes/class-bot-detector.php → analyze_heuristically(). Scoring:

• Each signal adds points (0–40 range)
• Threshold: ≥70 = classified as bot
• Signals: missing headers, suspicious UA patterns, datacenter IPs, scraping service headers

Modifying Monetization Pricing

1. Base price: sites.pricing_per_request in PostgreSQL (default $0.001)
2. Bot multipliers: analyzeBotRequest() in backend/production-worker.js
3. Content length bonus: >5000 chars = +50%, >2000 chars = +25%
4. Revenue split: Stripe 2.9%+$0.30 / Platform 20% / Creator remainder

Adding Admin Pages

1. Add add_submenu_page() call in class-admin.php → add_admin_menu()
2. Create render method following existing pattern (e.g., live_sync_page())
3. For AJAX endpoints, add hook in __construct() and handler method
4. Enqueue scripts/styles in enqueue_admin_scripts() for the new page slug

Adding WooCommerce Sync Events

1. Register new hook in class-woocommerce-connector.php → register_hooks()
2. Create handler method following on_product_updated() pattern
3. Build event payload with build_product_event() structure
4. Events auto-send via $this->api_client->send_live_event()

Adding API Endpoints

1. Add route in backend/production-worker.js using router.get/post()
2. Implement handler with validateApiKey() middleware
3. Update docs/API_REFERENCE.md
4. Add corresponding method in includes/class-api-client.php

Database Schema Changes

1. Modify database/schema.sql for PostgreSQL changes
2. For WordPress tables, update activate() in crawlguard-wp.php (uses dbDelta())
3. Four local WP tables: crawlguard_logs, crawlguard_rate_limits, crawlguard_signing_keys, crawlguard_fingerprints

Critical Implementation Details

Request Processing Order (every page load)

init hook → process_request()
  → Skip admin/REST/AJAX/cron
  → Rate Limiter (if enabled)
  → HTTP Signatures (if enabled)
  → Bot Detector → $bot_info
  → IP Intelligence (if enabled)
  → JS Challenge (if bot + no cookie)
  → API Client → monetization decision
  → Log to crawlguard_logs
  → Serve content or paywall

Feature Flags (disabled by default)

All optional features are gated behind crawlguard_options['feature_flags']:

• enable_rate_limiting → Rate Limiter
• enable_http_signatures → HTTP Signatures
• enable_ip_intel → IP Intelligence

JS Challenge Cookie

• Name: crawlguard_verified
• Value: sha256(IP + SECURE_AUTH_SALT + date('Ymd'))
• Duration: 1 hour
• HttpOnly + Secure (auto-detect SSL)

Environment Variables

Required: DATABASE_URL, STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, JWT_SECRET, CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_API_TOKEN, CLOUDFLARE_ZONE_ID

WordPress Options Key

All settings stored in wp_options as serialized array under key crawlguard_options.

Reference Files

• Full PRD: See references/prd.md for the complete Product Requirements Document including detailed architecture diagrams, database ER diagrams, all API specs, subscription tiers, and known gaps analysis
• API Reference: See docs/API_REFERENCE.md in the plugin directory for endpoint documentation
• Security: See docs/SECURITY.md in the plugin directory for security considerations

Known Issues

Build Commands

npm run build      # Webpack production build
npm run dev        # Webpack watch mode
npm run lint       # ESLint check
npm run package    # Create distribution ZIP