Y Combinator Founder Paul Graham: AI is Charged by Token, but Token is Not a Unit of Inference Measurement

Y Combinator founder Paul Graham stated that AI is charged by token, but token is not a unit of inference measurement, as stronger models can solve more problems with the same tokens. He suggested defining a new unit by using a series of progressively difficult questions, where each pair can be solved by the same generation of models, to calibrate "one unit of inference."

The current billing situation charges input and output by millions of tokens, with simple Q&A and long-chain reasoning sharing the same pricing. During testing, extending the thinking time can consume several times the tokens for the same task, yet the price tag remains based on characters. Some buyers in the industry have calculated costs based on "completing a task": models with the same price can have actual costs differing by more than three times due to retries and verbosity. Application layer investors are also advising product companies not to carry the token logic from the model layer into the application layer, suggesting that results should be charged based on measurable outcomes.

Mridul Singhai disagrees. He believes that the model's improvement partly comes from the training data generated during use; distilling and packaging this data into shortcuts to sell to later users benefits the model companies. According to his logic, the decrease in token prices and the increase in capabilities do not represent efficiency gained solely by users, but rather that user interactions are subsidizing the next copyright weight. If the unit only measures "problem-solving density," it will miss the underlying flow of data back to the model.

Graham's metric approaches a benchmark ladder: adjacent questions can be crossed by the same generation of models, and intergenerational leaps can be expressed as comparable inference quanta. The challenge lies in the fact that questions can leak into training, and once a ladder has been climbed, it depreciates; the quality of user prompts also changes the output per token, making it difficult to separate the unit from human skills. The cost unit remains watts and tokens, while the revenue unit is tasks; the misalignment of the two leads to the billing dispute of "the stronger, the cheaper yet more expensive."

There is no exchange quoting such units, nor is there a regulatory designated measurement standard. Discussions are occurring at a stage where APIs are sold by token, suites are sold by seat, and workflows are sold by results simultaneously. Whoever first helps procurement departments understand "per unit inference" has the opportunity to rewrite budget items.

In market mechanisms, buyers are enterprises seeking predictable bills, while sellers are model companies hiding capability improvements within the same pricing table. This is a dispute over measurement standards, not a daily market trend. Funds continue to flow into inference clusters by token; the beneficiaries are laboratories that can tackle more difficult problems with the same tokens and write back user trajectories into weights, while the pressured parties are application companies that reconcile by token but deliver by task, as well as users who realize their interactions are training the next version of competitors.

Source: Public Information