Independent Developer Pieter Levels Cuts Monthly Scraping Costs to $1 with Self-Built Crawler
Independent Developer Levels Cuts Monthly Scraping Costs to $1
Independent developer Pieter Levels has replaced paid scraping SaaS with his own crawler: Previously, he paid about $249 per month to ScrapingBee for Google search result scraping. With the help of AI and friends, he successfully ran a crawler with similar capabilities on his own VPS, achieving a success rate of about 90% at a monthly cost of approximately $1.
Levels operates products like Nomad List, Remote OK, Photo AI, Interior AI, and Hotelist, with scraping primarily used for programmatic SEO and hotel search pages. He publicly stated that after weeks of effort, he could not achieve stable Google SERP scraping, as headless browsers combined with residential proxies and CAPTCHA services were still easily blocked, leading him to continue paying. The new solution focuses on using real Chrome instead of Chromium, with a persistent configuration profile, making requests approximately every 30 seconds and caching results. Subsequently, the community has seen self-hosted alternatives based on this approach, providing search JSON and page scraping interfaces.
ScrapingBee was founded by Pierre de Wulf and Kevin Sahin, offering proxy rotation, headless browsing, and Google Search API, with the business plan at $249 including about 3 million credits and 200 concurrent requests. Levels had previously reduced total monthly expenses for non-AI sites to about $979, with a profit margin exceeding 99%, where scraping costs had dropped from $249 to $99; this time, he switched scraping from subscription to self-built. He listed the remaining paid items as domain names, email sending, object storage, servers, and large model interfaces.
Data center IPs are usually blocked by Google within hours, and the self-built solution requires residential proxies that match the country and search region, with CAPTCHAs still possibly needing manual intervention. The self-built solution trades off the engineering time needed to maintain anti-scraping and bans for the gross profit of middleware SaaS. Companies like DeepSeek continue to hire for infrastructure and training positions, indicating that computing power, storage, and model layers are still drawing engineering resources.
In market dynamics, the buyers are independent developers and small teams that originally purchased thinly packaged scraping APIs, while the sellers are middleware providers like ScrapingBee, SerpApi, and Bright Data; the driving force is code generation compressing "weeks of work" into a matter of days. Funds flow from monthly API bills to VPS, proxies, and model calls. Beneficiaries are individual developers who can accept about a 90% success rate and manage operations themselves, while those relying on anti-scraping packaged tools priced at $249 are under pressure, as underlying domains, storage, computing power, and model interfaces become harder to replace.
Source: Public Information
ABAB AI Insight
Levels has been running million-dollar businesses since 2014 using a single PHP machine, SQLite, and a VPS, achieving nearly 99% gross margins with products like Photo AI. His approach to SaaS is to closely monitor bank notifications and cut costs wherever possible: backups changed from unlimited versions to fixed quotas, cloud hosts downgraded, and map and tweet APIs swapped out when possible. Scraping is the last bill he couldn't cut, as Google treats headless fingerprints and request frequencies as bot signals. AI has not changed the physical laws of anti-scraping but rather the time cost of a person writing Playwright scripts.
The capital path is to convert middleware subscriptions into fixed costs: replacing a $249 API with a $1 VPS saves the pricing power of the packaging layer, leaving costs for Cloudflare domains and emails, R2 and B2 storage, Linode-like hosts, and model bills. The motivation is that programmatic SEO requires stable SERP pulling, and the strategy is to only purchase layers that cannot be stabilized in a few weeks and have network effects. Companies like ScrapingBee rely on proxy pools and CAPTCHA networks to charge scale premiums, and once leading clients demonstrate that a "90% success rate is acceptable," the pricing for business plans will loosen first.
Similar cases involve developers using open-source solutions to replace monitoring, forms, screenshots, and internal backend SaaS, as well as Zapier-like orchestration being broken down into scripts by model proxies. The industry phase sees thin tools being replaced, while infrastructure becomes centralized: scraping, reporting, and internal CRUD are delegated to personal repositories, while domains, object storage, GPUs, and foundational models become harder charging points. SerpApi and Oxylabs still serve enterprises requiring SLA and global proxies, while the personal webmaster market is rewriting the boundary between manufacturing or purchasing.
Structurally, this is a technical substitution driving pricing power downward: generative programming lowers the fixed costs of "making another internal tool," making the $249 monthly fee packaging lose its monopoly over individual buyers. The mechanism is that anti-scraping remains expensive, but the cost lies in proxies and fingerprints, not in that layer of JSON packaging; whoever sells the packaging will be the first to be eaten by code generation, while those selling irreplaceable addresses, electricity, and weight will leave behind bills.
ABAB News · Cognitive Laws
AI first eats thin and expensive middleware, not water and electricity
Manufacturing costs drop below subscription prices, pushing the purchasing boundary back
The remaining SaaS mostly grows on physical and protocol layers