Back to Flash News

Codex Head Tibo: OpenAI Tokens Are Not Standardized Units

Tibo, head of OpenAI Codex, pointed out that tokens from different models are not standardized units, and pricing per million tokens cannot be directly compared to actual costs, as the number of tokens consumed for the same text varies significantly across different models.

He illustrated this with an example of two identical pizzas: one cut into 8 slices at $2 each, and the other cut into 16 slices at $1.25 each. The latter seems cheaper per slice, but the total price of the whole pizza increases from $16 to $20, as a person's stomach does not care about the number of slices.

In a small comparison covering English, technology, multilingual, and digital texts, the tokenizer used by GPT-5.6 Sol consumed only 766 tokens, while Claude Opus 5 was estimated to require 1170 tokens, which is about 34.5% more for the latter.

The same exact text may incur additional token costs due to differences in tokenizers, and simply looking at the price per token does not reflect the true bill.

Even if tokenizer differences are adjusted, the key factor is the price per successful task completion; benchmark tests can serve as a starting point, but actual measurements in specific use cases are necessary.

From a market mechanism perspective, when buyers of AI inference services choose models based on token pricing, they tend to prefer providers with higher token efficiency to reduce total expenditure; sellers attract volume by optimizing tokenizers and output efficiency, driving funds from merely low-priced token models to solutions with lower actual outcome costs. Efficient tokenizer models benefit, while token inflation models face pressure.

Supplementary data shows that this comparison directly quantifies the actual impact of cross-model token density differences on bills.

Source: Public Information

ABAB AI Insight

Tibo, as a core member of OpenAI Codex and ChatGPT, has previously emphasized model efficiency and actual deployment costs. OpenAI has continuously optimized tokenizers and context handling in the iterations of the GPT series, historically consolidating its API market share by reducing effective token consumption, and has faced developer complaints over early token pricing transparency issues.

On the capital path, OpenAI focuses resources on self-developed tokenizers and model compression technologies, motivated by improving output per unit of computing power and reducing actual bills for customers to secure long-term API usage. Strategically, it aims to combat competitors' pricing competition through efficiency advantages rather than merely lowering prices.

Similar cases can be seen with Anthropic continuously adjusting caching and output pricing structures in the Claude series, and Google enhancing context caching in the Gemini long-context model. The current AI inference industry is transitioning from "pricing competition" to "effective cost control."

The structural judgment indicates a transfer of pricing power: because tokenizer efficiency directly determines the actual token consumption for the same task, model providers gain real cost discourse power through technical optimization, shifting customer focus from nominal prices to completion rates and total costs, thereby forcing the industry from the illusion of standardized pricing to differentiated competition centered on outcomes.

ABAB News · Cognitive Laws

  1. Cheap units ≠ low total cost

  2. Poor efficiency determines the real bill

  3. Pricing is superficial; outcomes determine pricing power