Meta Scrapes Web Pages Frequently to Build AI Search Index
Meta is scraping web pages on a large scale to build its own AI search index. Independent developer Pieter Levels reported that several of his websites experienced high-frequency access this week, triggering VPS load alarms.
DataDome data shows that in the second quarter of 2026, Meta's two AI crawlers made a total of 9.1 billion requests, with the Meta-WebIndexer growing by 163% in a single quarter, and the request volume in June surpassing that of the training crawler for the first time; this move aims to reduce reliance on Google and Bing.
Web traffic and computing power are being consumed by large-scale AI indexing, putting pressure on content sites driven by events, while platforms with self-built search capabilities benefit, and traditional search-dependent parties and small website operators face rising costs.
Source: Public Information
ABAB AI Insight
Since 2024, Meta has been laying out its own search capabilities to support Meta AI in reducing reliance on external engines, shifting crawlers from model training to real-time indexing, and extending the path from content consumption to independent retrieval infrastructure.
Resource investment is concentrated on high-frequency, low-return web scraping, motivated by the desire to control the source of answers and data loops, avoiding the leakage of search requests that competitors could use for training, while also enhancing the timeliness and controllability of AI responses.
This is similar to the early expansion of Google's indexing or crawlers from platforms like OpenAI; it is currently in a phase of expanding from reliance on external searches to building vertical indexes independently, with the AI search industry shifting from open web pages to closed data moats.
Essentially, this represents a restructuring of the industry chain: web content is transformed from a user traffic entry point into raw materials for AI training and indexing, with the mechanism being that high-intensity scraping brings almost no return traffic, leading to further imbalance between costs and revenues for content providers.
ABAB News · Cognitive Law
The expansion of AI indexing comes at the expense of content providers' computing power.
Building a self-owned search is a strategic choice to cut off data leakage.
High-frequency scraping with zero return visits is a new value extraction model.