Former Reddit CEO Yishan Wong: Distinguish Between Two Types of AI Risks - Existential Risk and Misuse Risk
Former Reddit CEO and current operator of Terraformation, Yishan Wong, wrote a lengthy article stating that a common mistake in recent AI safety debates is conflating two types of risks, which must be considered separately.
The first type is existential risk. If a superintelligence emerges that could completely overpower humanity, the difference would not be like that between Einstein and an average person, but rather between humans and mice. When humans expand cities and consume resources, there is no need to hate a particular mouse; the mouse may still disappear. Therefore, the danger comes from the superintelligence itself: regardless of whether its creators are good or evil, it may become uncontrollable.
He uses two analogies to address the question of "how to exterminate humans": asking a chess master how they would win against you does not change the outcome; similarly, lower animals asking humans how they would take away their habitat does not change the outcome of resource competition. Superintelligence will pursue goals that humans cannot comprehend and monopolize resources more efficiently.
The second type is misuse risk: humans using AI as a tool for evil or making mistakes—such as automated weapons, mass surveillance, cyber hijacking, biological weapons, large-scale unemployment, information overload, and even misjudging when to fire after delegating authority to a system. At this point, AI is still under control; the issue lies with who wields the tool, which fundamentally reflects familiar human flaws.
The key conflict arises from opposing solutions. The intuition to prevent existential risk is to not allow anyone to possess overly powerful AI; the intuition to prevent monopoly and tyranny is to allow as many people as possible to have powerful AI, preventing any one party from achieving absolute advantage. Historically, the misuse of technology often occurs due to monopolization by a few, and only later does it enter a phase of mutual deterrence.
In market mechanisms, this is a narrative stratification, not an immediate buy-sell signal. Computational power and policy budgets will flow according to the type of risk you recognize: recognizing existential risk leads to centralized control and delaying cutting-edge training; recognizing misuse leads to open-source diffusion, military parity, and anti-monopoly measures. Laboratories and cloud vendors benefit from centralized narratives in "who can train the largest models," while open-source weights and small to medium-sized deployers benefit from diffusion narratives. The mutual criticism arises because the policy function symbols are opposite.
The post cites a resignation from an Anthropic pre-training researcher who accused two leading laboratories of rushing towards self-improving superintelligence; Wong himself expresses concern about both types and demands that anyone who says "AI is too dangerous, so it should be restricted" first answer: which type are you afraid of?
Source: Public Information
ABAB AI Insight
Wong served as CEO of Reddit from 2012 to 2014 and later founded the reforestation company Terraformation. In recent years, he has repeatedly discussed climate directives being misinterpreted by superintelligence using the analogy of maximizing paperclips and "dying with dignity." He separates safety from content moderation and points out that China has implemented immediate harm rules such as content generation identification, face-swapping authorization, disabling recommendations, and banning price discrimination, yet hardly discusses NotKillEveryoneIsm. This reflects a split similar to that emphasized by Yoshua Bengio, who simultaneously stresses "we cannot control superintelligence" and "there is a lack of political institutions to prevent human mutual harm."
In terms of capital pathways, the existential risk narrative demands that cutting-edge training rights be confined to a few laboratories and licenses, directing capital towards auditable closed clusters; the misuse risk narrative demands that power be mass-produced at the endpoints, directing capital towards reasoning chips, open-source stacks, and military drone supply chains. OpenAI and Anthropic raise closed training budgets using safety branding, while the open-source camp raises diffusion budgets with "breaking monopolies." The same dollar buys opposite governance rights in the two narratives.
Analogous to nuclear weapons: non-proliferation is an existential logic, while mutually assured destruction is a misuse logic; policies cannot maximize both simultaneously. Hinton's push for a ban on the development of superintelligence corresponds to the open-source camp's advocacy for universal access to powerful models, reflecting these two solutions. The industry is in a struggle for defining rights, rather than having selected solutions.
Structural judgments belong to regulatory changes. The mechanism is that when the term "danger" is undefined, regulatory texts will write opposing prescriptions under the same title: limiting training power reduces the tail of the first type while increasing the capability asymmetry of the second type; loosening weights reduces the monopoly of the second type while increasing the cross-section triggered by anyone of the first type. First asking which type of danger it is is not a matter of politeness, but a way to avoid writing the symbols incorrectly.
ABAB News · Cognitive Laws
The same danger often corresponds to two opposite solutions.
If afraid of a god losing control, centralize the keys; if afraid of human malice, replicate the keys.
When debates stall at adjectives, policies are bound to write the wrong symbols.