OpenAI Trains Cutting-Edge Models with Over 100,000 GPUs
OpenAI has publicly stated that its largest NVIDIA GB200 supercomputer has been used to train cutting-edge models, with the cluster connecting over 100,000 GPUs into a single training network through a self-developed multi-path reliable connection protocol.
The company's engineering documentation notes that a switch capable of connecting 64 800Gb/s ports was modified to have 512 100Gb/s ports, allowing approximately 131,000 GPUs to be fully connected using only two layers of switches; traditional 800Gb/s networking would require three to four layers. This two-layer structure reduces power consumption, points of failure, and network costs, while providing redundancy across multi-plane networks, allowing training jobs to continue running despite multiple link jitters per minute.
This protocol has been deployed on OpenAI's largest GB200 cluster, located in Abilene, Texas, and at the Stargate site co-built with Oracle Cloud, as well as Microsoft's Fairwater supercomputer. The hardware combination comes from NVIDIA and Broadcom, and the protocol has been open-sourced in collaboration with Supermicro, Intel, and Microsoft. During the training of recent cutting-edge models used for ChatGPT and Codex, operations had to restart four first-layer switches, which would have directly interrupted training under the old architecture.
The product line is tied to the cluster. GPT-5.5 has completed training on the GB200 system at the flagship site in Abilene and Oracle Cloud. The company has also separated the inference side: the self-developed Jalapeño chip has released the first batch of InferenceX data, and a super-fast mode for GPT-5.6 Sol has been launched in collaboration with Cerebras. The computing power combination has expanded from a single Microsoft and NVIDIA setup to include Amazon Cloud, Supermicro, Broadcom, Cerebras, CoreWeave, Oracle, SoftBank, and energy providers.
Disclosed power contracts specify demand in gigawatts. Third-party statistics indicate that OpenAI has contracted approximately 26 gigawatts of computing power, with about 10 gigawatts from NVIDIA and about 16 gigawatts from non-NVIDIA sources, including approximately 6 gigawatts from Supermicro MI450 and about 10 gigawatts from Broadcom's custom silicon; the non-NVIDIA portion is now greater than the NVIDIA portion and exceeds the total disclosed combination of Anthropic. xAI's Colossus has deployed 100,000 cards internally with a plan to double to 200,000, compared to installation speed; OpenAI uses a 100,000-card Ethernet structure that maintains training compared to available computing power.
Buyers are model labs and cloud providers, competing for synchronized whole network GPU hours over extended periods, rather than single card pricing. Sellers include switch, optical module, power, and cabinet supply chains. The driving force behind the event is whether the cluster can continue training without interruption during failures. Beneficiaries are networks and clouds that can turn 100,000-card level training into stable operations, while pressured suppliers can only provide fragmented card rentals that drop connections halfway through training.
Source: Public Information
ABAB AI Insight
OpenAI started by renting Microsoft GPUs, making training scale a core part of its company strategy. Sam Altman plans to celebrate "over 1 million GPUs online" by 2025, and aims to scale up by two more orders of magnitude for AGI; the tools are not just slogans, but include the Abilene Stargate, Fairwater, and the 100,000-card plan in Norway, as well as open-sourcing the network protocol to Supermicro, Broadcom, and Intel. The historical path of exclusive binding with Microsoft is loosening: computing power is shifting from a single cloud to multi-cloud with self-developed inference chips, avoiding the next generation of pre-training cards being tied to a single landlord's delivery rhythm.
Money flows to three areas. First is the whole network-level GB200 data center, used to conduct pre-training and reinforcement learning training within the same fault domain; second is the non-NVIDIA backup, with Supermicro cabinets and Broadcom custom silicon hedging supply and pricing; third is inference ASICs like Jalapeño, which output tokens on the server using lower watt-hours from the parameters stacked during training. The motivation is that training still stacks intelligence by card count, while inference competes for profit based on throughput per watt. Resource delivery is through long-term power, cabinet, and protocol packages, rather than outright purchasing all chip ownership—labs still primarily rent assets from large-scale clouds.
In contrast to xAI's Colossus proving "how fast 100,000 cards can be plugged in", Meta and Google keep training within their own clusters, while Anthropic exchanges gigawatt contracts with Amazon and Google for diversified supply. OpenAI is at a stage of overlapping expansion and control: externally, it needs more cards, while internally, it aims to turn cards into a continuously training system, while extracting inference profits from NVIDIA's rented power. The industry has shifted from competing for peak FLOPS to competing for sustained usable training hours.
Structurally, this belongs to capital concentration overlaying technological substitution. The mechanism is that cutting-edge models must synchronize gradients on the same network of 100,000 cards, with entry points shifting from algorithm papers to whether power, optical modules, and second-layer switches can sustain the load; whoever can stabilize a structure of 130,000 cards will determine the release timing of the next generation of foundational models. Card count is the ticket, while network and power consumption dictate pricing power.
ABAB News · Cognitive Laws
Card count is the ticket; continuous training is the capacity.
A single network of 100,000 cards is more expensive than having an additional 100,000 scattered cards.
Training competes on scale, while inference competes on profit per watt.