Job Description

Posted 3 weeks ago

Python Web Scrapping Engineer – Competetive salary – Remote

Overview

We are looking for a Senior Python Scraping Engineer to design, build, and operate large-scale, resilient web scraping systems, with a strong focus on Google and Google-like search environments.

This role sits at the intersection of data engineering, reverse engineering, and systems reliability, and is critical to delivering accurate, timely, and trustworthy data at scale.

You will work on complex scraping problems where naïve approaches do not work-dynamic rendering, sophisticated anti-bot defences, frequent markup changes, and high-volume data collection.

We are leaders in adopting AI-assisted and agentic coding workflows. This role is ideal for someone who believes in leveraging AI tools to accelerate development and wants to deepen their skills in this area.

What You'll Do

Design, build, and maintain large-scale Python scraping systems that operate reliably against Google and other highly defended websites

Develop scrapers for dynamic, JavaScript-heavy pages using browser automation and hybrid scraping approaches

Continuously adapt to changes in page structure, request flows, and anti-automation mechanisms

Engineer robust extraction pipelines that prioritise data accuracy, consistency, and observability

Implement and optimise proxy, fingerprinting, and request-routing strategies to maximise stability and throughput

Monitor scrape health, detect failures early, and debug production issues quickly and methodically

Optimise performance, cost, and latency across scraping infrastructure

Collaborate closely with data engineers, data scientists, and product teams to ensure scraped data is usable and trusted

Document scraping logic, architectural decisions, and operational runbooks for long-term maintainability

Required Skills and Experience

Strong professional experience with Python, including production-grade scraping systems

Deep, hands-on experience scraping Google or similarly protected platforms

Solid understanding of HTTP, TLS, cookies, headers, redirects, and browser networking behaviour

Experience with browser automation frameworks such as Playwright, Selenium, Puppeteer, or equivalents

Strong knowledge of HTML parsing, DOM traversal, and data extraction techniques

Proven experience handling rate limiting, CAPTCHAs, IP rotation, and bot detection systems

Comfortable designing and operating asynchronous and concurrent scraping architectures

Experience running scraping systems at scale in cloud environments

Excellent debugging skills and the ability to reason about complex, non-obvious failure modes

Hands-on experience with headless browser fingerprinting and anti-bot evasion techniques

To apply for this role please submit your CV or contact Dillon Blackburn on (phone number removed) or at (url removed).

Tenth Revolution Group are the go-to recruiter for Data & AI roles in the UK offering more opportunities across the country than any other recruitment agency. We're the proud sponsor and supporter of SQLBits, Power Platform World Tour, and the London Fabric User Group. We are the global leaders in Data & AI recruitment