UK Planning Intelligence Scraper and CRM

The Buckinghamshire Council Idox planning portal renders content with JavaScript and uses JSESSIONID session-based pagination. Standard HTTP scraping returned empty pages because the portal requires an active session with valid CSRF tokens extracted from hidden form fields. The portal rate-limits aggressively, returning 429 responses that block the IP for extended periods. The HTML structure changes without notice, so the scraper would silently produce corrupt data when table layouts or CSS class names shifted. Duplicate records were a constant issue because the same planning application appears across multiple search result pages with slightly different formatting. The client needed every application within the Chiltern Beechwoods SAC Zone of Influence captured and scored, but the zone boundary is a geographic radius, not a simple postcode prefix list. A postcode-based filter missed applications at zone edges where postcodes straddle the boundary.

The scraper engine has five modules totaling 1,107 lines. The HTTP client establishes a session by hitting the advanced search page to get a JSESSIONID cookie, then parses hidden form fields for CSRF tokens. It spoofs Chrome 145 on macOS with full Sec-Fetch and sec-ch-ua headers. Twenty residential proxies from Webshare rotate every seven requests with startup randomization and five-minute per-IP cooldown tracking on 429 responses. When rotating, the system clears cookies, re-initializes the session on the new IP, and extracts fresh CSRF tokens. The HTML parser uses BeautifulSoup with lxml to extract application details, agent contacts, postcodes via regex, and dwelling counts through three regex strategies in priority order. The catchment filter runs four steps on every application: geocode the postcode via postcodes.io, calculate Haversine distance from the SANG site and SAC boundary, check dwelling count thresholds by zone, filter by application type whitelist, and scan against seventy-six noise keywords. The lead scorer assigns points based on dwelling count, zone proximity, keyword matches, and application stage. Qualified applications are upserted into Supabase PostgreSQL keyed on reference number, preserving existing status and dwelling data on re-scrape. Supabase Edge Functions written in TypeScript on the Deno runtime handle confirmation and follow-up emails by posting to the FastAPI backend SMTP relay because Deno cannot open raw TCP sockets. HTML sanity checks validate page structure before parsing and upload raw snapshots to Supabase Storage on failure. The system deploys across three platforms: Vercel for the static React SPA frontend, Render for the FastAPI Docker container, and Supabase for database, auth, cron, edge functions, and storage.

Built a production planning application scraper and CRM for a UK conservation client. The system monitors Buckinghamshire Council Idox planning portal daily, runs a four-step catchment filter using Haversine distance calculations from the Chiltern Beechwoods SAC boundary, scores leads based on dwelling count and zone proximity, and automatically emails planning agents offering SANG credits. Twenty rotating residential proxies with per-IP cooldown tracking handle the portal rate limiting. Supabase provides PostgreSQL storage, Edge Functions for email automation, and pg_cron for daily scheduling. The validation confirmed zero missed applications across three consecutive runs.

The system replaced roughly eight hours per week of manual planning portal monitoring with a fully automated daily pipeline. Every qualified application within the five kilometer and twelve point six kilometer catchment zones is captured, scored, and emailed to the relevant planning agent within hours of appearing on the portal. The corpus audit across all scraped applications found zero missed applications after full deployment. Three consecutive identical clean runs confirmed no duplicates, no missed applications, and no false positives in the catchment filtering.