Independent Developer Levels Cuts Monthly Scraping Costs to $1

Independent developer Pieter Levels has replaced paid scraping SaaS with his own crawler: Previously, he paid about $249 per month to ScrapingBee for Google search result scraping. With the help of AI and friends, he successfully ran a crawler with similar capabilities on his own VPS, achieving a success rate of about 90% at a monthly cost of approximately $1.

Levels operates products like Nomad List, Remote OK, Photo AI, Interior AI, and Hotelist, with scraping primarily used for programmatic SEO and hotel search pages. He publicly stated that after weeks of effort, he could not achieve stable Google SERP scraping, as headless browsers combined with residential proxies and CAPTCHA services were still easily blocked, leading him to continue paying. The new solution focuses on using real Chrome instead of Chromium, with a persistent configuration profile, making requests approximately every 30 seconds and caching results. Subsequently, the community has seen self-hosted alternatives based on this approach, providing search JSON and page scraping interfaces.

ScrapingBee was founded by Pierre de Wulf and Kevin Sahin, offering proxy rotation, headless browsing, and Google Search API, with the business plan at $249 including about 3 million credits and 200 concurrent requests. Levels had previously reduced total monthly expenses for non-AI sites to about $979, with a profit margin exceeding 99%, where scraping costs had dropped from $249 to $99; this time, he switched scraping from subscription to self-built. He listed the remaining paid items as domain names, email sending, object storage, servers, and large model interfaces.

Data center IPs are usually blocked by Google within hours, and the self-built solution requires residential proxies that match the country and search region, with CAPTCHAs still possibly needing manual intervention. The self-built solution trades off the engineering time needed to maintain anti-scraping and bans for the gross profit of middleware SaaS. Companies like DeepSeek continue to hire for infrastructure and training positions, indicating that computing power, storage, and model layers are still drawing engineering resources.

In market dynamics, the buyers are independent developers and small teams that originally purchased thinly packaged scraping APIs, while the sellers are middleware providers like ScrapingBee, SerpApi, and Bright Data; the driving force is code generation compressing "weeks of work" into a matter of days. Funds flow from monthly API bills to VPS, proxies, and model calls. Beneficiaries are individual developers who can accept about a 90% success rate and manage operations themselves, while those relying on anti-scraping packaged tools priced at $249 are under pressure, as underlying domains, storage, computing power, and model interfaces become harder to replace.

Source: Public Information