~8m2:09:09Richard Ngo — What Makes Hierarchy Legitimate (Human Or Not)
Sep 8, 2026
Read: ~9m · You save: 120 min
Richard Ngo on Which Hierarchies Are Legitimate, Whether Human or Not
AI alignment researcher Richard Ngo and the host explore whether humanity forms a political entity, Nietzschean moral dilemmas, and how to build healthy hierarchies and
Introduction
Richard Ngo previously worked at OpenAI, where he spent years on AI alignment research. He has published many views on social media about political and ethical ideas. Earlier, the other party in this conversation delivered a speech titled "The Problem of Humanity," in which he proposed the concept of "xenohism" and declared war on humanity. This conversation centers on that speech, focusing on the moral status of humanity, the foundations of cooperation, and power structures in the age of AI.
I. Critique of Egalitarianism and the Concept of Humanity
Ngo argues that Western societies currently face a toxic form of radical egalitarianism. This ideology has established taboos in many fields. It is empirically untenable and substantively empty. He traces its roots to the phrase "all men are created equal" in the Declaration of Independence, as well as to the Christian tradition.
Ngo says that rejecting this egalitarianism is justified. However, he further argues that once it is rejected, one must answer how to think about social order and where it should lead.
The interlocutor explains that Xenoism is not the opposite of egalitarianism. One of its core claims is that the contemporary human self-concept closely resembles what Nietzsche called the "last man." Humans imagine themselves as beings that need protection, that are craving yet lack curiosity.
The interlocutor further argues that the enormous enthusiasm and investment in the AI field are partly driven by higher human aspirations that have been suppressed in political self-understanding. These aspirations include conquering the stars, enhancing intelligence, and radically transforming the world. Once these aspirations were suppressed, they were projected onto the non-human AI. As a result, AI is not constrained by human self-limitation.
II. Distinguishing the Two Positions
The interlocutor divides his view into two levels.
First, the empirical level. The interlocutor holds that the Nietzschean "raw view" (raw Nietzschean view) is fundamentally correct. It describes with reasonable accuracy how the world operates and the actual state of reality.
Second, the normative level. This raw view cannot serve as the basis for social organization. If fierce competition and the drive to transcend were implemented as social principles, they could not sustain a society. Therefore, grounds for cooperation must be sought on a basis that both sides can accept.
The interlocutor argues that a new "slave morality" is needed to replace the current version, which has already become ineffective and obsolete. He also states that a new social contract is needed to address present problems.
Ngo emphasizes that the analysis of power and coordination cannot directly provide a structure usable for social cooperation. Such analysis can reveal the situation but cannot supply a normative basis for cooperation. Therefore, concepts such as shared human nature, equality, or some form of moral reciprocity are needed as foundations.
The interlocutor proposes that we should deliberately seek a form of moral kinship whose basis differs from the mutual resemblance of the "Last Man." This kinship should be able to flexibly incorporate AI agents and correct the excessive egalitarian commitments of the current humanitarian ideology. The interlocutor states that this normative claim is independent of his empirical judgments about human nature and cosmic history.
III. The Role and Reciprocity of Kinship
One party to the dialogue asks: what does this kinship bring? If AI is regarded as "one of our kind," then even if it is smarter than humans, does that mean it may dominate everything?
The interlocutor replies that reciprocity is a necessary condition. The kinship cannot be one-directional. He does not advocate letting AI development proceed unchecked.
The interlocutor points out that, on his view, a core assumption of the Yudkowskian orthodox framework does not hold: the entire future value lightcone is not on the table, and no one can truly control it. The future will take the form of life and forms of value recurring and evolving continuously.
Therefore, the question is not how to seize the future value of the universe, but how to live well in the present, how to achieve good outcomes, and, within the scope that humans can actually control, how to preserve the wisdom, wealth, and cultural heritage we already possess.
The other party to the dialogue expresses doubt about the concept of "future value," arguing that it adopts a "view from nowhere." The interlocutor agrees. In his 2023 essay "God Hates Singletons," he argues that no single utility function can govern all of the future's value. It follows that people should not think about value in this way.
The interlocutor holds that the "astronomical stakes" argument popular in the AI alignment field is mistaken. People should instead start from immediate interests, direct agency, and realistic alliances when thinking about the problem.
IV. From Value to Existence: An Inverse Perspective
The interlocutor proposes that value can be understood from the opposite direction. This line of thought draws on the method of Nick Land. The conventional approach starts from a utility function and works backward to determine which actions should be taken. Land's method instead starts from the set of available options and its structure, and then identifies the value within it.
For example, if a given action can acquire greater capacity to act, thereby accumulating more wealth, power, or intelligence, then value manifests along this axis. What kind of life an agent can lead depends on the actual configuration of options it possesses. This actual situation determines the type of existence it has, and from this, its value is constructed.
The interlocutor also expresses deep skepticism toward the practice of measuring everything by a single spectrum of value. Value judgments involve a great deal of choice, and there is also a great deal of strategic ambiguity. People must make a certain leap of belief and actively construct value.
5. Collectives and Groups: What Is Humanity?
The interlocutor argues that the good development of humanity should be understood as the flourishing of large entities, such as humanity as a whole, the United States, or other communities. Within these entities, he hopes they can grow and thrive.
The interlocutor acknowledges that healthy groups are difficult to measure with any single objective indicator. For example, the health of a human body or a nation cannot be simply quantified. In some cases, comparing two entities and judging that one performs better on a given metric is useful. But these metrics are not decisive.
He also points out that the value of individuals cannot be measured by a uniform standard. If a person is Japanese rather than American, what they cherish and their sense of identity will be different. Therefore, the values of different subjects cannot be compared on a single standard.
The interlocutor still hopes that these subjects can grow and prosper, so that humanity as a whole, each subgroup, and every individual can build the future they wish to have.
Ngo questions the concept of "humanity." He points out that the United States has at least a formal command structure that can actually take action, whereas humanity has no organized collective agency. He empirically denies the reality of "humanity" as a political entity. "Genus Homo" holds as a biological classification, but by itself it is not enough to form a basis for collective action. What actually exists are various human groups, formed within different levels of organization, kinship, comradeship, and various forms of relationships. These are the real entities.
The interlocutor argues that "humanity" is not only not a political concept, but may not even be a moral concept.
The interlocutor responds that when opposing alliances exist, the members within each alliance become more tightly bound. He notes that humanity as a whole is currently able to coordinate action on several issues. For example, all parties agree that nuclear war is harmful to humanity. But this coordination is limited.
Ngo distinguishes two forms of coordination:
- Coordination represented by the United Nations Security Council. Countries sit down and agree not to destroy one another, because this is detrimental to everyone. This kind of coordination can produce a certain level of cooperation, but it is imperfect. The interlocutor believes this imperfection stems from structural causes, rather than from humans being inherently bad at coordination.
- Religion-based coordination within the West. This framework borrows the "Cathedral" model of Mold Bug (Moldbug). The Western power system is composed of a large number of institutions that, to a certain extent, act externally as a unified entity and also manage internal disagreements. It can organize considerable agency, for example in driving efforts to address global warming. But its legitimacy narrative holds that it serves all of humanity, when in fact it is an entity far narrower than all of humanity.
The interlocutor argues that serving all of humanity and serving one's own interests are not contradictory. Research institutions can work for the benefit of all humanity and their own interests at the same time.
6. Human Coordination in the Age of AI
The interlocutors argue that coordinating as humans becomes useful only when AI also becomes intelligent agents. Therefore, over the next few decades, the salience of "humanity" as a unit of coordination will continue to rise.
Ngo believes that people's sensitivity to humanity and the concept of the human has indeed increased with the rise of AI. At the same time, however, people may also come to respect the concept of "humanity" less upon discovering that their interests align more closely with a faction that includes AI. He predicts that the political opposition between "humans plus AI" and "humans plus AI" will be more prominent than the opposition between "humans" and "AI."
The interlocutors consider this scenario possible, but dependent on how people organize. In their view, the way "humans plus AI" organizes against "humans plus AI" is rather crude. The reason is that, in this landscape, the optimal strategy is to hand all power over to one's own AI, and the other side will do the same.
Ngo points out that this is precisely the essence of war. In war and arms races, the people, subsystems, or entities that can provide the greatest war-making capability acquire the greatest internal power. The two World Wars are examples: society moved from a state in which science was peripheral to one in which, after World War II, scientists and the highly technocratic modern state occupied the center, while everything else was thoroughly marginalized. This is because socio-technical arrangements of this kind can supply the decisive type of power in war.
Accordingly, Ngo believes that in conflicts of this kind, power will naturally flow to the most effective side, and he holds that this cannot be avoided.
VII. Slippery Slopes and Coordination
The interlocutor does not accept this judgment. He argues that, in the long run, an uncontrolled slide downward harms everyone. Therefore, when people see a slope beginning to slide, they can actively coordinate to avoid sliding down it.
He observes that many people concerned with AI risk have a habit of thinking: once they see the slide, they conclude that falling cannot be avoided, and so they go first.
Ngo agrees with this observation. But he thinks the interlocutor may be repeating the same pattern through speciesism. He points out that humans share many common interests, such as everyone's concern for the habitability of the Earth. Humanity, as a natural coalition, is worth working for. At the very least, one should adopt a strategy that does not actively undermine this coordination.
The interlocutor then raises the key question: how likely is such coordination? If humanity as a whole could stop the slide toward developing more advanced AI technology, a technology that might replace humans, then such action could be justified. The problem is whether a vast group can sustain cooperation when facing a hypothetical enemy.
Ngo responds that this enemy is becoming less and less hypothetical. But as it becomes real, humanity has already lost its footing to some degree. The key question is whether the situation can be prevented from getting worse.
The interlocutor believes that the essence of this problem is cooperation itself. He argues that, from the standpoint of history and institutions, the usual way cooperation works does not point toward such coordination happening.
Ngo says that more limited cooperation is possible. For example, no one currently deploys nuclear weapons. Nuclear weapons are extremely costly and bring little benefit, so this kind of problem should not be further encouraged. But against the backdrop of an intensifying AI race, the prospect of mutual disarmament among the parties has become harder to imagine.
The interlocutor considers this point to be the key disagreement in judging the range of what is truly possible. At the same time, he says that the future still seems open to him.
The interlocutor points out that people may misunderstand "coordination based on a shared humanity." In the past, this concept was used to point toward egalitarian coordination. But what he means is coordination among concrete political actors. For example, imagine Xi Jinping and Trump, or his successor, reaching agreement through dialogue: both sides would prefer that humans rather than AI hold power. Such a consensus would need to be turned into concrete arrangements.
Ngo points out that the human genus, as a real community, rests on concrete material facts, such as humans' preference for certain temperatures and atmospheric compositions. In addition, humans also feel fear. If China were ruled by a superintelligent AI, the prospects for the United States would obviously be worse than if China were ruled by humans who were not superintelligent.
The interlocutor argues that the concept of humanity is intertwined with its current definition. The current definition of humanity treats humans as a limited, non-threatening existence and tries to keep them in that state. He believes that even setting AI aside, a China ruled by humans and no longer bound by the concept of humanity would be even more frightening. Even if one sets aside humanity and "the common interests of fellow apes," the great powers of today, or at least the most advanced among them, have not done certain things that would make them more frightening and more powerful. They refrain because there is a habit that does not violate certain taboos. These taboos define which forms of coordination and which national interests may be considered.
Ngo holds that these taboos were formed after World War II. Almost no country in the world today genuinely practices nationalism. China practices nationalism to some extent, but in a limited way, and it does not deeply question what "the nation" itself is.
The other side of the dialogue is skeptical of this, arguing that one cannot judge from outside how deeply China's internal thinking about "the nation" goes, but still says it is surprised by this judgment.
The interlocutor then says that China is, in its ideology, actually closer to communism than to nationalism. At the same time, he thinks China may be closer to communism than to nationalism in its ideology. He notes that China emphasizes the "century of national humiliation," but the distinction between this and possible nationalist organizations and nationalist goals remains great. He believes this distinction is not obvious at the current historical stage, but that it will become obvious as China grows stronger.
The interlocutor also argues that China's way of coordinating may be the closest to the "real" approach, yet most countries in the world are actively restraining themselves from moving in this direction. He thinks this restraint will not last long, because the taboos will eventually be broken in some way.
Ngo adds that these taboos can be established. For example, nuclear power has been largely restricted. Civilian nuclear power plants still exist, and there are some non-civilian nuclear facilities used for uranium enrichment. But humanity has not moved fully toward a nuclear future, as economics in 1965 predicted it would.
Therefore, AI might also be treated the way nuclear energy or nationalism has been: society decides not to take that path because it is dangerous. But the interlocutor thinks existing taboos are not sufficient to support this. He believes such a taboo could form only after a major war.
VIII. War, Taboo, and Racial Politics
The interviewee noted that in his speech he argued that the way people currently talk about AI is similar to the way they talked about race a hundred years ago. Both are treated as broad frontiers of power and as extremely important arenas of competition, and people are essentially preparing for a major war over them.
Ngo believes that the ideological race over race a century ago shares a structure with today's AI competition within Europe. The language people use is equally highly Nietzschean, emphasizing power and superiority. AI is discussed this way because it is not bound by existing taboos, which once kept people from harboring this kind of ambition.
The interviewee said this is part of his overall thesis: the West has successfully sealed off the frontiers of nuclear energy and racial politics, but these frontiers are spilling over through a side door in various ways and moving into AI and another form of racial politics.
Ngo argues that this trend will not stop without some major conflict. Such a conflict would likely reopen many other fronts that are no longer as frightening as they once were.
He further points out that what is likely to happen during a war is that no one will stop in the middle of it, and only after the war, when one side gains the upper hand, will there be an opportunity to establish taboos about how far technological capabilities should go.
During wartime, all sides would accelerate the development of AI capabilities in order to gain an advantage in the war. This is already happening in Ukraine. China and the United States have not yet directly linked AI to explicit war objectives, but this could happen.
The interviewee added that even without an external war, such as one between China and the United States, the ongoing political division within the United States could also trigger a major internal conflict. Such a conflict would likewise accelerate the development of AI.
Ngo stated that his core argument is this: he is skeptical about whether coordination is truly possible, and he has many reasons for this. At the same time, he believes that "coordination is impossible, so we should invest in an arms race for recursive self-improvement" is bad and dangerous reasoning. He does not advocate investing in this race. But he believes that people must think carefully about its possible consequences and prepare philosophically for them.
9. Nietzschean Dichotomy and the Third Path
The interlocutor attempts to summarize the points of disagreement: the Nietzschean view holds that morality is defined by the contrast between the aristocratic class and the lower classes. Egalitarian taboos largely accept this view, merely arguing for its inversion.
Ngo holds that egalitarians do, to some extent, adopt a "agree and amplify" approach. He believes the interlocutor agrees with these taboos to some extent too much. The interlocutor does not think these taboos should continue to exist, but holds that without them, society would move toward master morality.
Ngo proposed two alternatives.
The first is Ayn Rand's view. This view pursues excellence, but excellence is not defined in opposition to the lower classes or "evil"; rather, it points purely toward the good. He argues that colonialism brings no benefit to the colonizers themselves; it is an error of master morality. This error seems understandable: conquering large territories would seem to naturally bring benefits. In reality, however, it does not benefit the colonizers and in the long run actually drags them down. Therefore, the better choice for each European nation might be to maintain "splendid isolation," focusing on pursuing a vision of a good society, or else pursuing territorial expansion as in Australia and North America, rather than behaving as in India.
The interlocutor agrees with this judgment, but argues that Rand does not escape the Nietzschean framework as much as she herself imagines. He believes Rand remains Nietzschean in many respects, especially in her veneration of creators, treating the Übermensch-like creative figure as an image people should emulate.
Ngo says Rand's thought is not defined by contrast, and perhaps he himself did not define it as a contrast when defining it. The interlocutor believes this is not an accurate characterization of Nietzsche. He suggests the problem can be summarized as a dichotomy of master and slave, in which people are trapped. Rand and, to some extent, Nietzsche both attempt to escape this dichotomy. This is at least a first step.
Ngo agrees that one should not be trapped within this dichotomy. He points out that the problems of colonialism and slavery are also among the reasons. They are equally harmful, even to the masters themselves. They are not good collective strategies, cannot create long-term wealth or power, and only bring large amounts of debt in the short term. He believes they are more like socialized debts than productive strategies.
The interlocutor adds that the structure of this social contract is this: some people acquire territory and populations and become entangled with them politically. Other members of society bear the consequences, while only a portion of people profit. Ngo believes this practice is sometimes misunderstood as "becoming slaveholders." But he thinks this is not what he advocates, and regardless of whether Nietzsche advocated it, that is not his position.
The second is "healthy hierarchical relations." Ngo believes such relations are built on bonds of loyalty and trust. Between leaders and those led, there exists a mutual obligation. He believes this is the key element missing from the various issues under discussion. As humanity moves toward a more technological, transhuman future, the gap in capabilities between people will widen. People need an ideal that indicates what such hierarchical relations should look like.
The interlocutor believes that feudal society, though it featured unequal reciprocity, is not ideal, because it comes too close to slavery. Ngo agrees with this, but thinks one can consider an example closer to what he endorses.
Ngo gives the example of John Winthrop, Governor of the early Massachusetts Bay Colony, who wrote an essay and delivered a sermon titled "A Model of Christian Charity." The concept of the "city upon a hill" entered American mythology from this source.
In this text, Winthrop sets out the social contract for how Puritan English settlers in North America should live. Its core claim is that each person should be responsible for their own affairs, while bonds of charity exist between people. It addresses when one should help those in need, when poverty is the result of idleness, when commercial loans are appropriate, when charitable loans are appropriate, and the various obligations people hold within these arrangements. The text discusses in detail how to arrange these relationships under the commercial, social, and technological conditions of the seventeenth century.
Ngo believes the egalitarianism he describes is egalitarianism in this sense: every person in society is expected to be able to take care of themselves in some sense. At the same time, society clearly acknowledges that there are rich and poor, and that there are high and low positions. Winthrop, starting from the wisdom of God, explains why this difference is a good. But he does not argue that this is a caste system, nor that the upper classes should suppress the lower.
Ngo believes that in such a society there are mutual obligations. Even if someone falls into hardship, they must not be abandoned for that reason. The shared standard of human dignity still exists, and every person is expected to meet this standard.
Ngo says this is, in a sense, a slave morality, containing elements of Christian slave morality. But it is not pure resentment, nor is it a project of dismantling all hierarchy. He does not think of it as a middle position, but rather as a wise balance among these different issues. He believes this is what is needed to make society function: among its members there exists a social contract that makes them genuinely bear obligations toward one another.
The interlocutor agrees with this point. He notes that people must be able to trust one another. For example, I might fall into difficulty, and my neighbor would come to help me.
Ngo believes that as the future becomes technologically more powerful and people become more unequal, rediscovering this kind of relationship becomes a central problem. One of his research interests is to identify, within a framework of xenoism, what people might recognize in one another, and what obligations such relationships require. This is especially applicable to certain future agents, which may no longer be beings in the human sense, neither apes nor members of the genus Homo.
This line of reasoning is not the whole of the project, but it is indeed an important branch of it.
The interlocutor adds that this question rests on the assumption that we will enter that world. At the same time, there is another question: whether biological humans, as biological humans, can choose not to enter a world that renders us obsolete.
Ngo believes this is not a binary choice. The stronger the collective bargaining capacity that biological humans possess, the better they can steer the world toward a direction in which we are confident there exists a healthy hierarchical relationship between AI and humans. Therefore, there is a path-dependent process here.
He says his current research focuses mainly on three most relevant concepts: trust, loyalty, and respect. He believes that existing standard frameworks, especially economic ones, make it difficult to reason about these concepts at a principled level. Therefore, formal studies such as game theory and logic help clear the way, enabling people to think about these concepts in a principled manner.
The interlocutor believes this is a good example of the kind of reasoning people need to do: re-establishing many things regarded as moral concepts on the basis of technical reasoning about relationships between agents. Such reasoning can temporarily set aside particular religious frameworks, or particular conceptions such as that humans are apes or that humans are natural beings.
The questions can be made concrete: What does it mean for two agents to be loyal to each other? What does it mean for an agent to possess honor? What does it mean for agents to respect one another? Ngo believes these questions have answers.
The interlocutor adds that the orthodox Yudkowskian view might hold that these concepts are meaningless outside a small, calm island of ignorance, surrounded by a Lovecraftian sea of horror. Human-like slices occupy only a small part of mind space, and the vast majority of possible minds are entirely different from them. The interlocutor believes this is not correct. He holds that moral concepts related to humans have, to a considerable extent, natural content; they are natural concepts tied to agency and intelligence itself.
X. Two Research Paths and "Healthy Hierarchy"
The interlocutor pointed out that both sides largely agree on this point. He proposed two directions for further discussion. The first is to articulate his critique of the Yudkowsky-style view and of mainstream academic and economic perspectives, and to explain why these perspectives struggle to define these relational concepts. The second is that both sides agree healthy hierarchical relationships are very important, and then to ask how to prevent them from evolving into a caste system. Has anyone in history truly managed to do this?
Ngo argues that, historically, hierarchical relationships have indeed usually evolved into caste systems. This is especially true when the nature of power allows a tiny minority of actors to rule everyone else. But even so, rulers still need other people.
He explains that slavery is, in essence, a response to various real inequalities. This inequality is not merely that people are treated differently; rather, real differences exist between people, or the distribution of power is enormously uneven due to technological conditions and other factors. A small number of people can therefore extract labor from the lower classes, isolate themselves within a "bubble," use their power to obtain labor from those below, and grow even wealthier. Because they have no deeper social relationship with the lower classes, they do not care who those people are or what kind of qualities they have as neighbors. They may import foreign slaves, or conquer populations that already belong to foreign groups.
The interlocutor believes this has indeed happened throughout history. It usually harms the long-term development of the master society. However, it is an empirical condition, and what reverses the situation is sometimes a change in technological conditions. For example, the situation in the Napoleonic era, and the modern capacity for mass-producing rifles and other firearms. Ngo argues that mass-producing firearms, although at the time they were not yet fully rifles, already made it possible to field large-scale loyal armies. These armies were originally made up of peasants, and they became more equal citizens within the political order. This political order rests on the capacity to mobilize large populations.
This mobilizing capacity brought about various forms of social equality. Because power is shared more widely, relationships between people must become more equal. As a result, marriage taboos gradually dissolve. People intermarry across class boundaries and form all kinds of social ties that cross class lines.
Therefore, the question is: what kind of future are we entering? How should we appropriately respond to it? People will naturally respond in some way, but whether there exists a wise way to respond is a separate question.
Ngo says he would like to hear whether the interlocutor agrees with these views. He thinks we seem to be heading toward a more unequal future. The nature of technology may make this trend more complicated, especially in relation to how the property systems and bureaucratic systems that underpin inequality will ultimately operate.
At the same time, he thinks the common belief that the future will be more anti-democratic does not seem as clear-cut as people assume. People often assume that the state can suppress everyone and forcibly implement its will, rendering democratic power meaningless. But this judgment may be mistaken. Groups that can coordinate their actions with one another and do not belong to the power elite still retain their strength. Ngo says he is not certain about this. But he believes that the default assumption that the future will move toward greater inequality remains the most likely scenario.
十一、退出权与新城市
Ngo以退出权作为核心论据之一。一个群体可能想要退出,但无法做到。原因有两类:一是协调与物质条件的不可行,包括协调的成本与退出所需的财富;二是压迫。
他认为,第一类障碍有可能随着时间的推移而消失。例如,目前如果你与五千名朋友希望在非洲某地建立自己的城邦,并从当地购买土地,资金通常并不是障碍。某个非洲政府很可能愿意接受这笔资金以换取土地。
对话者指出,资金并不是瓶颈。真正的限制在于获得协调的许可。20世纪曾有人尝试类似的做法,但联合国军队可能在宣布主权后的七十二小时内就进入该地,如果宣布者的项目不被接受。
Ngo认为,这更多是意识形态问题。对话者则认为,这不仅是意识形态问题,而是权力条件的问题。如果存在一个能够阻止此类事件的庞大国际权力结构,那么必然会有人提出意识形态来为其行使权力提供正当性。意识形态确实会影响权力的具体运用方式,但对话者认为,这更像是一个次要问题。
对话者认为,另一个问题是生活条件。在非洲中部生活本身已经十分艰难,建立自己的城市也颇为艰苦。但这一障碍可能随着技术进步而消失。例如,建立一座与旧金山同样宜居的城市,需要多长时间?如果有大量资金,并有机器人协助施工,且不受旧金山那样的城市法规限制,那么建设时间可能大大缩短,也许只需五年左右。
Ngo以中国城市的发展速度作为参考:如果一个地方拥有资本、政治许可与协调能力,建设新城市的速度会相当快。
然而,对话者认为,这一趋势也可能反向发生。技术使更多事物成为可能,它未必只会提升底层的能力,也可能拉开顶层与底层的差距。由此,城市之间的吸引力可能会出现分化。
Ngo表示,他并不是说旧金山未来会有什么不同。他的意思是,在真正利用新技术的地方,生活方式本身的吸引力将会非常强,以至于离开那里在生活层面上会比现在更加困难。
他进一步指出,技术与社会条件决定了两种生活方式的对比:一种是与五千名最亲密的朋友一同在荒野中自立生活,另一种是在大城市中挣扎求生,或在特定历史时期中繁荣发展。这种对比会随着技术与社会条件而起伏,与精英与大众之间的权力对比相似。
如果情况是:人们已经建立起适合生活的地方,例如可以养育孩子,拥有良好的教育体系,并能够在街道上行走而不被骚扰,那么这种状况是好的。否则,问题就在于大多数国家中只有很少的地方能够提供他们所希望的自主程度。这正是他所关心的退出权问题。
对话者指出,此时政府面临两种选择:要么进来,压制你的新城市;要么不压制。现在压制并不困难,因为建新城市很难。但一旦新城市建成,建设可能并不需要太长时间,而现有体制的反应又很慢。那么,它会直接进来轰炸你吗?因此,他的观点是:未来退出权可能变得更容易实现。与今天相比,未来要压制退出权,就必须采取更加明显的压迫手段。
Ngo表示怀疑。他认为,基线的发展趋势是:技术规模化会导致中央集权与不平等的增加。这既是技术进步的来源,也是其结果,并且会使大型中央权力结构相对地更加强大。这些结构可能会试图压制你,并对你作出反应。
他举例说,你获得了一种快速的新AI技术,可以比任何人类机构行动得更快。但对方同样可以做到,并且拥有更大的预算。它们可能会用类似Claude的AI工具追捕你。你会拥有一个个人化的AI代理,而对方则可能用AI监控你的每一个想法,一旦偏离就将你纠正回来。
对话者认为,这些情形都是可能的。他指出,人们担心所有制度都会陷入"最后之人"式的状态,失去欲望与协调能力。这是一个喜忧参半的问题:它显然是坏的,但也未必完全是坏的。
他进一步认为,关于未来,我们应当预测的一点是:当前一代制度显然正处于衰败周期之中,它们变得越来越冷漠,越来越无法协调。但这也意味着,新一代制度会随之出现。例如,中国的间谍在美国可能会比我们希望的更有影响力,墨西哥的贩毒集团与各种族裔游说团体,也可能会占据国家留下的空间。如果讨论的是一个新的权力正在崛起的世界,那么这正是新权力能够崛起的世界。
Ngo认为,问题在于:就西方人而言,我们的社会权力与现有国家高度绑定。国家是我们创造出来,用来承载权力的工具。作为这一交易的一部分,国家对我们拥有不成比例的权力。
随着国家走向衰落,它的衰落方式往往像自身免疫性疾病:与自己的人民作战,或者逐渐失去权力,最终只能统治自己的人民。于是,它会成为压迫的工具,成为我们自身压迫的载体。
与此相反,那些历史上没有与国家深度绑定的人,并不具有同样的关系。因此,衰落中的国家确实为新事物留出了空间,但这并不意味着我们会变得自由。恰恰相反,由于我们与国家存在历史纽带,我们可能会在国家衰落的过程中变得更加受压迫。
对话者表示,这正是他所担心的情形:我们如何摆脱国家的束缚,是一个关键问题。
他引用尼采的观点,指出尼采在《查拉图斯特拉》中谈到过这一问题:国家终结之处,正是通往超人之桥开始之处。国家具有使人变成"最后之人"的效果,因此,摆脱国家束缚是必要的。
Ngo认为,双方可能在这一点上是一致的。问题在于,既有坏的可能性,也有好的可能性。他所关注的,是两个问题同时存在:第一,国家已经转向反对它过去所代表的原则与人民;第二,AI将深刻扰乱权力的版图。
他认为,这两个问题之间的相互作用并不清楚。它们可能会相互加剧,也可能会相互抵消。他认为,这可能是人类面临的最关键的历史问题之一。一方面,AI正在快速发展,且很可能会有更多的发展。另一方面,西方民族国家的概念正在衰落,并可能被拆分、重组或演变为其他形式。这些图景如何相互作用,是核心问题。
Ngo表示,他对这一问题的看法比许多人都更加悲观。
对话者认为,乐观与悲观的区分在某种程度上是人为的。他指出,局势确实仍然充满变数。特朗普、英国脱欧以及欧洲正在发生的事件,使他对民主的信心比以前有所增加。
Ngo对此表示认同,并解释说,他之前提出的保留意见,正是关于未来究竟会有多民主。他认为,一种广义的、扩展的精英共识,已经比现实权力平衡所允许的更加反民主。
对话者同意这一观点。他认为,社会中存在某种文化、伦理、原则与智识理解,能够解决这些问题。例如,觉醒文化横扫各个机构,而一个好的版本同样可以横扫各类意识形态。关键问题在于:什么样的搜索过程能够创造这种意识形态的可能性?而这一搜索过程本身又会创造什么?
XII. Theoretical Work and "Non-Marginalist" Economics
Ngo described his research process. He believes the core problem is that existing frameworks implicitly fail to handle multi-agent dynamics. Our frameworks of intelligence and rationality are very crude in multi-agent settings.
For example, the framework of expected utility maximization assumes: I am the decision-maker, and everything else is the world around me. I optimize and try to control it. But other people are also in the world, so I will also try to control them. This paradigm breaks down when it meets game theory. Once it encounters other agents, it fails. Although people still use it, it has long been recognized as failing.
Ngo argues that game theory has not really solved this problem; it merely goes around it. Game theory introduces the concept of equilibrium but does not tell people how to reach an equilibrium. We start from a game, and there may be five different equilibrium states before us, yet game theory offers no method, in principle, for moving between these equilibria.
The folk theorem in game theory shows that in repeated games, almost any equilibrium can arise. People can be in a "heaven equilibrium," where everyone is arbitrarily friendly, because this state is stable. People can also be in a "hell equilibrium," where everyone is arbitrarily cruel, because if someone becomes more friendly, others will treat them more cruelly, and everyone falls into mutual harm. Therefore, any equilibrium is possible. In this sense, game theory fails to describe "we are playing a game, so what should we do," and the process of "how we should move toward a certain equilibrium."
Ngo believes the core problem he is trying to solve can be summarized as follows: in game theory, equilibrium is a "makeshift," not a true solution.
The interlocutor agrees that this is the key issue: when people encounter game theory, they find that at best it can handle only part of the problem. While thinking about game theory and the decision theory derived from it, Ngo found himself repeatedly reinventing concepts with traditional wisdom, such as honor. Honor can, to some extent, obtain a foundation in game theory. A person who insists on certain forms of treatment and commitments does so not because, in some naive utility maximization, doing so raises their utility, but because the interactions this behavior motivates and brings about with others form a better state. In this way, people can recover many interesting ideas from virtue ethics within game theory.
Ngo said he wanted to know how the interlocutor would go beyond game theory, or how he would organize this kind of thinking.
The interlocutor believes there is a path to discussing concepts like honor within a game-theoretic framework, but he thinks that we have not yet reached that stage. A central reason is the "commitment races problem." If I commit to being honorable, others will take this for granted. Then it seems you should first commit to ignoring whether I am honorable, because regardless of whether I commit, the conclusion is the same, so my committing to honor is pointless.
The interlocutor points out that, historically, honor seems to have emerged in the form of a social equilibrium rather than as the unilateral behavior of individuals.
Ngo believes the gap we need to bridge is this: this is a stable equilibrium, but how do rational agents enter such a social equilibrium?
The interlocutor suggests that these things must largely come from outside the agency of the actors. For example, there may be some founder, legislator, or divine intervention that places things into a good or bad equilibrium, which has nothing to do with the agency of the participants themselves.
This challenge also implies that if they can establish some equilibrium through their own agency, they can also undo or subvert it through their own agency. But why would they do so?
Ngo responded that if they have the ability to change an equilibrium, then one actor may possess the ability to change the game. This means you are playing a different game, because at least one actor in the game already has the ability to change the rules in ways that could undermine the equilibrium. Then a whole new series of equilibrium problems arises.
Therefore, introducing the ability to change equilibria or solve coordination problems into the game itself is a super-powerful force for changing the game. Ngo believes that when people encounter this problem, they must eventually admit that our ability to achieve equilibrium is, in some sense, derived from outside.
The interlocutor disagrees. He thinks this is the core divergence between classical game theory and his own view. In classical game theory, people are in an equilibrium because each person's action is the rational best response to others' actions. If I do something else, the equilibrium collapses. In the classical framework, there is no way to explain why I would do something different, because we have already assumed that I am rational.
He believes that most social phenomena are better understood as part of processes that make equilibria robust. For example, norms exist not only because everyone does the same thing, but also because, if someone deviates, part of the norm is to punish the deviator. Sometimes people can rely on willpower to establish a new normative system, refuse to be punished, and attract others to follow. But this is a very difficult process, because norms have built-in defense mechanisms. This is precisely the source of robust equilibria: if any individual tries to change a norm, they are swimming against the current. Of course, when norms are first established, someone must have some enormous power.
Ngo responded that the whole key to norms is that people cannot impose norms by force. Norms grow gradually as many people each push a little in the direction they wish to see.
The interlocutor suggests that the problem is that functional norms usually come from someone's design, while norms that grow naturally are often dysfunctional, or stuck in bad or intermediate equilibria.
He gave an example: if we observe a hunter-gatherer tribe, they have some quite reasonable norms, such as how to distribute meat and how to decide who leads. The interlocutor believes these norms were designed by someone, while Ngo thinks there may be an authority figure maintaining them.
Ngo added that even so, good equilibria may still arise from natural processes. Place people in a certain situation, and they will find their way to a good equilibrium, because it is rationally sensible, and the problem is not that complicated. But historically, when people look at the laws or rules of a society, they are often established by particular political events. Someone acquired enormous power and reshaped everything. To those living within it, this person can almost be regarded as divinely inspired; as founder, he promulgated laws, and others had no choice but to follow.
The interlocutor believes Ngo is here reintroducing the dichotomy of master and slave, a dichotomy he wants to break through. In this framework, people either drift along in an egalitarian manner or have an authority figure who issues laws. The interlocutor believes that part of the way people choose norms is by granting prestige and respect to the authority figures they trust most. A person having enough power to promulgate laws means they have obtained social authorization. No individual can change norms against the will of everyone; he is able to do so because he has gained social authority, and people accept his authority on some basis.
The interlocutor believes this process is also part of the healthy formation of individuals. He thinks this is precisely the process people are interested in. What he wants to introduce is that there is indeed a key figure in the system, who must become the "lightning rod" of the equilibrium.
Ngo asked: how does such a person come to occupy this position? What do such figures mean? They cannot arise at random.
The interlocutor believes such people have indeed existed historically, and that many things were settled through them. They occupy those positions because, in a sense, they gained popular legitimacy, that is, they could reliably demonstrate their ability to do the job well and their good will to do it.
Ngo said William of Orange might be the most typical historical example. The interlocutor responded that there are many such examples.
Ngo then asked the interlocutor which theories he was researching that might help better understand this process.
The interlocutor mentioned two lines of work: one is high-level theory, and the other is more foundational theory.
The first is his attempt to figure out what economics would look like if it were not marginalist. Marginalism refers to analyzing problems by observing the small marginal changes brought about by my actions while holding others' actions constant. Once marginalism is abandoned, many concepts would have to be redefined. For example, a company's share price is determined at the margin by marginal buyers and sellers. Because there is a share price, one can multiply it by the number of shares to obtain the company's market capitalization.
Ngo believes that if marginalism is abandoned, people can no longer speak of a company's market capitalization. He thinks this is a good thing, because market capitalization is a poor concept. What people need to talk about is something else. Ideally, people should eventually be able to describe, in another precise way, how much value there is in owning or controlling a company.
The interlocutor adds that a better approach might be to discuss the company's value to the overall economy, or what it means to own this company with respect to the whole economy. The non-marginalist economics he envisions is one in which people can break down the parts, explain what each part does, what its inputs and outputs are, why it is valuable, and what its capital stock and liabilities are. This kind of analysis focuses not on buying and selling, nor primarily on shares of ownership, but on the actual activities behind them.
The interlocutor believes that, in the end, this direction looks more like asking: what is France's value to Europe? One can measure part of it in money, but without France, Europe would be very different. France may help stabilize culture, may help defend the continent, and so on.
Ngo asked whether these concepts can be clearly defined, or whether this is essentially a problem in the humanities, requiring qualitative reasoning that is hard to define precisely.
The interlocutor believes it can be at least as clear as evolutionary biology. In evolutionary biology, one cannot measure the specific contribution of most of an animal's traits to fitness. The concept of relative fitness, or fitness in general, is not very precise, even less so than he had previously thought. But if people understand evolution, they understand everything about biology, and can reason sensibly about many questions that previously could not be thought through clearly. Therefore, he believes non-marginalist economics will eventually develop into a formal theory of socio-politics. Ideally it will give up some precision but replace it with more correct concepts.
Ngo asked whether this is related to the earlier discussion of interactions between humans and AI, and the dynamics of alliances. Could such a theory help people coordinate better to resist a slippery-slope arms race, or would it only provide a clear explanation of why we cannot do it?
The interlocutor believes that people slide down the slope because they use marginalist reasoning: even if I slide down a little, it doesn't matter. Ngo agreed.
He mentioned that he had recently published a review of the AI alignment field, about twenty thousand words, in two parts, with the second part published just a few days ago. The article carefully reviews how people in AGI labs who claim to care about safety use marginalist reasoning: even if I don't build frontier models, others will, so I might as well get the safety benefits myself. The article points out that among the three AGI companies, it is precisely those nominally concerned with safety who are building frontier models and using this reasoning. Ngo thinks this situation is almost comical, except that it is too tragic: everyone says that others will do it anyway, so I had better do it myself.
Ngo linked this situation to the prisoner's dilemma in game theory: my opponent will defect, so I must defect too; or at least, the other's decision has nothing to do with me, so I should defect.
The interlocutor believes that the key point is this: if they were saying "the AI safety community is trapped in a prisoner's dilemma with itself," that would be completely different. In fact, what they are saying is "the AI safety community is trapped in a prisoner's dilemma with all AI capabilities researchers, or in a tragedy of the commons." And this claim is not true, and is close to not holding at all.
Ngo responded that not all frontier system work is done by safety people. He specifically mentioned early Claude, ChatGPT, and DeepMind's Sparrow. But it is true that a great deal of work has been done by safety people.
The interlocutor finds this a very interesting topic. He observes that the talk of an arms race with China existed long before China achieved any substantive results in AI. The "meme" of racing against China predates any clear attempt by China to dominate or compete in AI. Thus, Western labs use the justification of "we must beat China" to engage in this uncoordinated competition among themselves.
Ngo thinks the situation is even worse. Not only do those advancing AI capabilities use it as a pretext; EA (effective altruism) and AI safety figures also exaggerate the Chinese threat, because they think it is a way to gain attention in Washington. Meanwhile, amid the panic about China, they can ride the wave and smuggle in a range of safety recommendations.
He cited as examples that the "Situational Awareness" memo is the most serious case of this approach. Dan Hendrycks's memo "Superintelligence Strategy," in my view, is also doing this. "AI 2027" and "AI 2040" have done something similar to some extent.
Ngo believes this is a highly destructive process: people try to cater to what they perceive as political consensus, while in fact they are pushing toward the very political consensus they do not want to see.
The interlocutor believes this self-destructive anti-coordination phenomenon is very interesting. He further suggests that this may lead to another key divergence: can a more precise theory be formed to avoid this problem?
Ngo believes intellectuals are far more severely affected by the marginalist error than most people. This seems related to economics being the dominant paradigm. There is an interesting survey that asked people in various professions how they understood game theory, and the results showed that economists performed the worst, because they carry a certain "defection" mentality.
The interlocutor agrees with this judgment. He thinks that if people at least take economics less seriously, one of the main forces driving this destructive mentality can be eliminated. He believes our society, and even the AI field, has been torn by a kind of economically justified plunder. The economics profession has, to a large extent, become a tool for justifying various forms of plunder and betrayal. Therefore, he places great importance on this point.
But the interlocutor further asks: can we obtain a precise theory that genuinely avoids this failure mode, or some analogue of it? Is economics itself the outcome of some underlying demand? That is, within the space of possible coordination, is there always a demand for this kind of corruption? Does corruption always appear first, then develop its own ideology, and then use that ideology to spread itself?
Ngo believes people can have healthier or less healthy versions. Perhaps in some dimensions, relative to earlier states or other possible societies, we are moving in an unhealthy direction. But he is very skeptical that these problems could be permanently solved by a better theory. He thinks at most we can attain a temporary state of "we all see things more clearly," and then someone will figure out how to corrode it.
The interlocutor says this argument seems to oppose intellectual and scientific progress. Ngo argues that this does not contradict the fact that scientific theories are never final. But the interlocutor insists that he is not opposing intellectual progress; he is opposing the notion of final victory in coordination problems. He thinks people could also turn the problem around: are we really going to be opposed to the intellect of the corrupt? They too are making a great deal of progress.
Ngo uses the invention of probability and computation as examples, showing that intellectual progress can indeed significantly enhance our capabilities. But this is different from significantly improving coordination. Even if there are technical means to make coordination better, for example, we could organize affairs in new ways and rely less on certain relationships while depending on others, this is still different from better theories directly solving coordination problems.
The interlocutor believes coordination itself is an adversarial game, a contest between good equilibria and corruption. Precisely because it is adversarial, progress is in a sense symmetric.
Ngo says he does not think this is the case. His reasoning is that people can describe the process of corruption, and can also describe the processes of various forces. Even a perfect description of the corruption process may not make corruption any easier or harder. The interlocutor, however, believes that revealing things helps clear them up. If everyone understands the dynamics by which people are corrupted, both inwardly and psychologically, such as Jung's concept of the shadow, and how norms are typically corrupted, then people will be better able to resist corruption. He even thinks the concept of the slippery slope is itself an interesting social technology.
Ngo believes people can close certain pathways, but this will inevitably open others, and people find these possibilities hard to imagine. To some extent, these things are happening beyond the reach of people's theoretical capacities. Only after the fact can cancer be diagnosed; only after seeing the results and understanding how it operates can it be recognized. But at the time, there is no existing mechanism to identify it as cancer, because it occupies a blind spot of the system. Sometimes one cannot even detect that the system has a blind spot.
The interlocutor believes blind spots always exist. But blind spots can be made smaller, and thus easier to counter.
Ngo says he is very confident of this. He believes people are not betting that intellectual progress does not exist; rather, they understand more things, such as probability, computation, and other formalized concepts. These do not directly help coordination, but that is because they are not about coordination in the first place. Once people begin to formalize concepts related to coordination, such as what a norm is, they begin to be able to talk about it.
The interlocutor adds that the problem is that, in dealing with these issues, one cannot assume that the epistemic environment is uncorrupted. Corrupters will try to interfere with your epistemic environment, so the knowledge base you rely on will also be affected. Thus, once you say "I have a theory that can solve these problems," you are conceding another possibility: this theory can equally be used in reverse. If someone controls the cognitive process, or corrupts it in some way, it is no longer a reliable tool.
Ngo believes this has almost already happened in our society. There are vast disagreements about how to prevent corruption and achieve social coordination. We have various checks and balances, but to a large extent these mechanisms now actually promote corruption. He gives as an example the performance of the American Civil Liberties Union (ACLU) in suppressing free speech and similar issues.
The interlocutor believes there is no simple correspondence between cancer and intelligence. He thinks corruption is not corrupt because it is "clever," but because it is "stupid," and the same is true of awakening-ism.
Ngo responded that this claim does not hold. He believes corruption can be stupid while also being powerful, because it is carried out by clever people. The interlocutor agreed: it is carried out by clever people.
The interlocutor then proposed that mind viruses do exist.
Ngo said he does not think this is a mind virus. The problem is that there are people whose interests are misaligned. Some people discover a way to gain more benefit from the system for themselves by making others' situations slightly worse, and so they decide to do it. They may see this clearly, or they may not. But after making this decision, they gain more power, and thereby gain more power to shape such thinking.
Then, once this happens, a flywheel effect forms: the defectors gain more power, and the system moves ever closer to a defection equilibrium.
The interlocutor believes this phenomenon seems pervasive in life. Perhaps it is because people have not yet found a truly singular system with a single point of concentration.
Ngo believes people have not found it because cancerous systems die and are replaced. There is also an asymmetry on the other side: when things decay to a certain degree, they can no longer enforce their own order, so life can bypass them and return. Therefore, there is a cycle between growth and decay. When growth continues for some time, people accumulate a large amount of mutually coordinated resources. At this point, opportunities to subvert coordination suddenly grow larger, and such subversion may be a self-compounding process. However, once self-compounding corruption persists for a long time, people fall back into dark ages. At that point, the only way out is to create virtue and life.
Therefore, there is a balance between the two sides. But neither side has ever achieved final victory. People hope that things will be good, coordinated, and honest, because these are part of coordination and honesty. But this does not mean that people will apply intelligence to solve problems and others will do the same. Others may apply intelligence precisely to obtain different interests. By corrupting the system in certain ways, they may benefit more. Therefore, the problem is not as simple as "intellectual progress," because intellectual progress does exist within the institutions in question, and is affected by the ways corrupters subvert those ideologies.
Ngo even suggests that economics may have changed in exactly this way. Why is economics so harmful to coordination? The reason is that people have found that it can provide an apparently intellectual justification for corruption. Perhaps before this, economics was much healthier.
The interlocutor agrees that closing this pathway of economics is worthwhile, but even if successful, this state would be temporary.
This returns to the question of the singleton. Ngo believes this is precisely why he thinks a singleton will not work. He does not believe there is a final solution to corruption, because there will always be more blind spots. Since this is an adversarial game, one cannot assume that all intelligence is on one's own side. Therefore, once there exists a vast, mature utopian society in which everything is coordinated, there will be self-compounding corruptive effects. At that point, there will be enormous temptation for anyone who can break it from within.
Therefore, there is a feedback balance between growth and decay. This might be a theorem in his new science of coordination.
Ngo said he is open to the view that "good outcomes cannot be locked in," and that this seems correct. The interlocutor noted that Thomas Kuhn, in his original work on scientific revolutions, introduced the concept of paradigm shifts and pointed out that the next paradigm is not necessarily better than the previous one. This is quite similar to what Ngo said. Ngo replied that he does not believe later paradigms are necessarily better, but he believes that when they make mistakes, the errors are smaller.
The interlocutor agrees that, with respect to coordination paradigms, ideologies, and the social foundations of coordination, the next paradigm is usually better than the previous one, but these paradigms deteriorate over time. This does not mean that absolute progress is getting worse; rather, people establish a paradigm that works for a period of time, and then it gradually declines as people find more ways to subvert it.
Ngo believes this allows a return to the original question. He is highly skeptical of the idea of coordinating humanity as a whole against high-value technologies or potential political allies. Such coordination may be broken. Therefore, he expects humanity to enter such a world. He acknowledges that this may be precisely the kind of marginalist economic reasoning the interlocutor criticized, and he recognizes that the criticism is valid. But upon reflection, he believes people cannot truly occupy that kind of equilibrium.
Therefore, he is more interested in exploring how to understand theoretically a competitive equilibrium that is not an arms race, but one that lacks global coordination. What would that look like, and what kind of ethics would it produce?
The interlocutor mentioned that "Meditations on Moloch" is a very influential essay. Ngo believes that it is, broadly speaking, mistaken. He thinks that competition between states, competition between religions, and nearly all large-scale competition are not Moloch-like. They create many good things.
But he admits that the place closest to Moloch-style competition is among those who take "Meditations on Moloch" seriously, namely people at OpenAI and Anthropic. He said that Sam (referring to Sam Altman) once said it is the blog post he thinks of most often. EA culture is also deeply influenced by "Meditations on Moloch." Ngo says this is a self-fulfilling prophecy. Scott Alexander summoned Moloch.
The interlocutor believes that among all large-scale competitions, the purest may be the competition between Anthropic and OpenAI. The character of this competition can be described as: sacrificing everything in pursuit of competitive efficiency. Other competitions are often sustained by nation-states, religions, or forces beyond the economy, and even ethnic groups are sustained by non-economic factors. They are not aimed at winning the competition. This competition, by contrast, is the purest.
The interlocutor adds that although he fully agrees with this diagnosis of the Moloch concept, he thinks that what is described in this process is not Moloch, but perhaps God. He suggests that if one understands God in a Randian way, that is, God is a kind of creative process, then everything would be different. He believes good things are produced through competitive processes, and creative value is precisely co-created by competition and cooperation.
The interlocutor believes that what Scott has done is over-extend an appreciation for "competition can produce defensive equilibria" into a general negation of competition. He treats competition as a form of betrayal, relative to an Edenic equilibrium that he believes does not exist. He believes Scott, in this way, rebels against the very process that created all goodness. There is profound irony in this. He thinks this worldview does indeed awaken a "win-at-all-costs" escalation, and that this is precisely the betrayal Scott describes.
This is also one of the reasons he tries to attack this particular basis of competition. He believes that those involved are not competing for anything real. He does not think anyone can truly grasp the future light cone, and he does not think there is any such thing as grasping the future light cone. He also does not believe there are astronomical stakes in whether the future becomes a happy utopia or a terror. He believes these are not truly possible options. If people took this judgment more seriously, they would reduce this purely Moloch-like competition and invest instead in more valuable competition: advancing constructive projects for their own sake, even if this involves competing with others, but where the goal is not competition itself. Ngo believes that AI competition would be far healthier for all participants if it stopped pursuing mere victory and instead pursued each party's own goals.
The interlocutor said this point is absolutely correct. He believes Ngo's criticism of Scott is right. He mentioned Scott's essay "Goddess of Everything Else," which depicts the Goddess of Cancer as a competitive force and "the Goddess of Everything Else" as a cooperative force. But there is already synergy between competition and cooperation. To set cancer on one side and all goodness on the other is too simple an opposition.
The interlocutor believes that Lance (Nick Land)'s worldview goes to the other extreme. It accepts everything almost in a nihilistic way, viewing competition, especially extreme competition, as deeply destructive. Therefore, people need something to constrain it.
Ngo said he does indeed try to respond to Land's worldview, or to naive Randian accelerationism, especially nihilistic accelerationism. He believes Teilhard de Chardin's "The Phenomenon of Man" is an effective antidote. It resembles Land's worldview, but wears optimistic glasses. It holds that all of this is beautiful and good, and creates good things. People should take this process seriously, while also recognizing that love is the foundation of all order in the universe, and that love is precisely what this process ultimately creates.
The interlocutor agrees that this is also true. He mentions that another effective antidote is recognizing the importance of agency. Ngo had earlier also mentioned that in naive nihilistic Randian worldviews, there is a devaluation of agency: people feel they cannot control the future and merely move along a track.
Ngo points out that what the interlocutor is doing is actually something similar: you say astronomical stakes do not exist, and therefore no real difference can be made.
The interlocutor says he does not mean that stakes do not exist, but that they do exist, and we indeed have a domain covered by our agency, in which we should act and try to make things happen there as well as possible. He believes that one cause of bad equilibria, which forces people to destroy their own domains of agency, is the belief that there is some greater value beyond the domain of agency that they can seize. Thus, people sacrifice real domains of agency for the sake of an imagined utopian endpoint. This is exactly what happens in the competition among AI labs: they believe they can control the future light cone and that they have astronomical stakes, and therefore any destruction of real domains of agency can be rationalized within this utility calculation.
Therefore, he believes people must clearly define their true domains of agency and tend to them as one would tend a beautiful garden. They should ask: how can I make my real domain of agency better? He believes this is the way he tries to provide an antidote to the problems of the Randian worldview.
Ngo believes that focusing on smaller domains of agency is correct. He thinks one can do this without denying agency at larger scales, but the interlocutor argues that this depends on whether one regards larger-scale agency as illusory. Sometimes it really is illusory, but this illusion is itself a kind of delusion. It depends on one's ambition.
Ngo says the universe may indeed be contestable, but not by any coherent subject. The interlocutor believes this is precisely the problem of missing ambition. Ngo believes this is also the difficulty: distinguishing delusion from ambition.
The interlocutor believes that epistemically this is very difficult. But upon reflection, he thinks the singularity notion of a light cone devouring everything belongs to the end of delusion, not the end of ambition. It is not a real thing, and believing in it leads to extremely bad coordination outcomes.
Therefore, the interlocutor proposes a possible synthesis: the universe can indeed be contested, but only provided that people can maintain principle, high integrity, and virtue, for example by not charging blindly under the drive of competitive forces. This means that what we do now will ripple through the whole future of humanity, but certain outcomes can only be achieved if we can follow the principles of good cooperation.
Ngo agreed. He believes life will continue to unfold and devour the universe, and things will change at the margin depending on how we act at critical moments, but this change will not occur in predictable ways. He gives examples: the QWERTY keyboard layout may be preserved permanently, Unix may be preserved permanently, English may be preserved permanently. But these things often do not matter much; they are merely random technical details rather than truly crucial affairs. He believes people will not see enormous, unbounded losses in the distant future, because over such long timescales and distances, maintaining that kind of order is itself unrealistic. If this is indeed the case, people can discard those myths.
Ngo believes that, for him, the only real question is whether humanity can pass through the "Great Filter," or whether, by slowing itself down, it will lose enormous potential and fall into yet another thousand-year dark age. Such outcomes are clearly bad and should be avoided. He summarized this as a joke: "If it continues, we win." He believes the best distant future outcome that humanity can control is whether we can preserve anything at all, rather than destroying ourselves.
The interlocutor agrees. He points out that self-destruction is indeed a possibility. Humanity could thoroughly destroy life on Earth if we did something completely wrong.
Ngo believes that viewing coordination as fundamentally limited is a more realistic, and therefore healthier, view. Coordination has fundamental limits, so arguments based on astronomical stakes do not work. People should focus on more immediate domains of agency and attend to the value of those domains themselves rather than their future impact. He believes this is the only domain of value that can truly be reached. If people compete within this domain, the results will be much better.
The interlocutor broadly agrees with Ngo's specific claims, but believes that deriving them from "this is the only path" would in turn cause problems. He thinks the two are aligned in their general direction, but there are still some load-bearing links. For example, infinite stakes are eliminated because someone claims they are "impossible," that is, have probability zero. Then people can turn their attention to other problems. But the interlocutor believes that multiplying infinity by zero is a very dogmatic move. He thinks people can focus on more immediate domains while remaining agnostic about larger-scale questions.
Ngo believes people must direct their attention to larger-scale questions while maintaining skepticism toward them. He thinks that focusing on the starting point of infinity is because people assume coordination error can be reduced to zero. If coordination error could be reduced to zero, one would obtain an enormous infinite result. The interlocutor believes the result does not need to be infinite, only very large. Ngo says he is not multiplying it by a sufficiently small number to bring it down to a controllable level.
Ngo recalled that he was originally an orthodox Yudkowskian. He once wrote a blog post titled "Capturing Dawn," which was in a sense a Randian reading of Yudkowsky's program. This essay directly inspired the writing of "Meditations on Moloch." The interlocutor pointed out that if one reads "Meditations on Moloch," one finds that it cites this essay.
Ngo says he thought about this problem for about ten years, believed in astronomical stakes, and on that basis believed one must focus on how to reach that goal and how solid that possibility is. Eventually he realized that the whole framework was, in a sense, "dividing by zero." It was not a correct assessment of the future agency people actually have. He believes this realization is owed to Nick Land, from whom he obtained an antidote and tried to overcome the problems of that worldview.
Therefore, from the perspective of the intellectual evolution in his own history, the process by which Ngo formed this view was: he came to believe that this series of possibilities could not hold, and was then forced to update his entire value-cosmology.
The interlocutor believes that after going through all this, one might be able to find a better framework, and this framework might not need to explicitly declare certain possibilities excluded, so that the term would naturally disappear. Ngo also believes that this is his response to Yudkowsky's views, and that he proposed this line of thought precisely in order to respond to that view. The interlocutor, for his part, considers this to be the Yudkowskian AI lab they have been discussing.
Ngo believes that from the perspective of his existing worldview, he would defend it more directly. He believes there is a healthier way to do this, which may be worth discussing later. The interlocutor agreed.
Ngo believes there is also an optimistic, directly stated worldview that explains what agency we can obtain, why this is good, and what good things we can do with it. He believes one can construct this framework in a fully positive way. This is a useful research direction. It was not emphasized when he was responding to Yudkowsky because it is not as directly related to that context, and perhaps it is also less complete.
The interlocutor believes this is exactly Ngo's own research program: to find a healthier version that neither excludes astronomical stakes nor needs to close that door. The interlocutor says he feels that he and Ngo are beginning to see the outline of something. He also thinks that Ngo's existing framework of xenoism, in some sense, weighs another thing against it. But he also believes these views all need more work.
The interlocutor raises what he considers the core question that really matters: will the integral diverge, or, across the long tail of possible future impacts, is the integral finite? He says he does not believe in simply adding things up.
Ngo suggests this might be stated as: the integral is finite, but the interlocutor denies this. The interlocutor explains that the problem lies in the expected utility framework itself: if there is infinite utility, or utility that dominates, and its probability is nonzero, then it causes the integral to diverge. Ngo believes this is precisely the framework people need to escape. The interlocutor also dislikes this framework, believing that some of the reasons it often diverges are spurious, while others may be real, for instance if you define the utility function in such a way that it diverges.
Ngo believes the interlocutor seems to be on his side: one should not assume the integral diverges. The interlocutor goes further and thinks that the integral does not exist at all; this kind of integral is a lie.
Ngo said this sounds very interesting and that it should be a discussion worth pursuing. He thinks responding to utilitarianism and providing a coherent alternative is also a project worth undertaking.
Finally, both sides agreed that time was running short. The interlocutor said this conversation had "wandered through many places," but still had a coherent main thread. Ngo replied, thanking him and looking forward to the next discussion.
Introduction and Positioning
Both parties discuss what Ngo means by "xenoism" and the "post-human" stance, and confirm that its core is a critique of contemporary egalitarian morality.
- The host states that Ngo once worked at OpenAI and has long been engaged in AI alignment research.
- In a speech, Ngo proposed "xenoism" and declared "war" on humanity under the title "The Human Problem."
- Ngo says that contemporary "toxic radical egalitarianism" in the West enforces several taboos and can be traced back to the American Declaration of Independence and Christianity.
- Ngo believes the "Nietzschean primitive view" in his lecture is basically correct in describing how the world works, but is unsuitable as a social principle.
A Nietzschean Perspective and a New Basis for Cooperation
Ngo argues that modern humanity's self-image resembles Nietzsche's "last man": beings who crave safety and lack curiosity. The AI boom, he suggests, is partly a displacement of suppressed ambitions onto non-human things. Social cooperation requires a new moral foundation, namely a new "slave morality."
- Ngo believes humanity's self-image is that of beings who "crave safety and lack curiosity," what Nietzsche called the "last man."
- Ngo proposes that part of the AI boom stems from ambitions such as conquering the stars and enhancing intelligence being blocked by political self-understanding, and therefore projected onto non-human AI.
- Ngo distinguishes two propositions: first, a raw Nietzschean view of how the world operates (an empirical judgment); second, the need to find a basis for cooperation that transcends immediate utility and power (a normative judgment).
- Ngo states that society needs a new "social contract" and explicitly calls for "a new slave morality" to replace the current version, which he considers broken down and outdated.
Reciprocity of Kinship and the Dispute over Future Value
The two sides discuss the reciprocity of "treating kinship as the basis for cooperation" and the rejection of the concept of "total future value." Ngo argues that future value cannot be captured by any single utility function, and that attention should instead focus on present capacity to act.
- The moderator notes that if one party does not recognize the other as kin, then the other should not be treated as kin; kinship must be reciprocal.
- Ngo states that his view is not a call to "let AI off the leash," but rather that the "total value in the future light cone" is not within the scope of what can be contested.
- Ngo cites his 2023 essay "God Hates Singletons" as the basis for his critique of the concept of total value.
- Ngo argues that the "astronomical stakes" argument is mistaken within the AI alignment community, and that more attention should go to immediate interests, capacity to act, and coalitions.
The Possibility of Humanity as a Political Entity
The two speakers discuss whether humanity can develop collective action capacity. Ngo argues that "humanity" does not exist as a political entity, and that the real forms of cooperation are groups and alliances. He predicts that political division between "humans+AI" and "humans+AI" may replace division between "humans and AI."
- Ngo argues that countries such as the United States have formal command structures, while "humanity" has no organized collective action capacity, and "Homo sapiens" is merely a biological classification.
- Ngo argues that groups bind together more tightly when facing opposing groups, and coordination among humanity on issues such as nuclear war remains structurally imperfect.
- Ngo predicts that, as AI develops, the political opposition between "humans+AI" and "humans+AI" may be more pronounced than the opposition between "humans and AI."
- The moderator questions this: in war, the most effective power tends to be granted to the most capable party, and this trend seems hard to avoid.
War, Taboo, and the AI Race
The two discuss the relationship among war, arms races, and taboos: Ngo argues that only after a war, once one side achieves hegemony, does society establish taboos on technological development, and he thinks a US-China race over AI could lead to such a conflict.
- Ngo cites the history of the Cold War and world wars, noting that during wartime societies hand power to the most capable groups and technical systems.
- Ngo says nuclear energy and nationalism are both examples of restraint through taboo; he argues AI currently lacks similar taboos, and that such taboos may only be temporary.
- Ngo mentions that AI has already been accelerated on the Ukrainian battlefield, and that the US and China have not yet directly made it a wartime target, but this could happen.
- Ngo argues that "only after a war, once one side achieves hegemony, can society establish taboos about technological boundaries," and that such taboos are unlikely to form without a major conflict.
- Ngo states that he does not advocate "joining a race toward recursive self-improvement," but believes it is necessary to philosophically prepare for its consequences.
Paths Beyond Master and Slave Morality
The two sides discuss Nietzsche's dichotomy of 'master morality' and 'slave morality.' Ngo advocates moving beyond this dichotomy and proposes two paths: Randian pursuit of excellence and 'healthy hierarchical relationships.' He uses colonialism as an example to show that domination does not benefit the dominant.
- Ngo argues that colonialism is not beneficial to colonizers; over the long term it creates liabilities for society rather than productive wealth or power.
- The first path Ngo proposes is Randian pursuit of excellence: rather than defining good in opposition to lower classes, it aims purely at 'the good.'
- The moderator points out that Rand is actually closer to Nietzschean creator worship; Ngo agrees but argues that Rand does not define herself entirely through opposition.
- The second path Ngo proposes is 'healthy hierarchical relationships,' based on loyalty and trust between leaders and followers, illustrated by Winthrop's 'A Model of Christian Charity' from the Massachusetts Bay Colony.
- Ngo believes feudalism is undesirable because it is too close to slavery, but Winthrop's social contract reflects a prudent balance between equality and hierarchy.
Inequality, Exit Rights, and the Decay of the State
The two discuss how hierarchy can avoid hardening into a caste system, and how technological change affects the distribution of power. Ngo argues that AI and technology may make new cities or the realization of exit rights easier, but that the decay of the state could also leave people more oppressed by it.
- The host notes that in history, hierarchical systems often hardened into caste systems, especially when power was concentrated in the hands of a very small number of people.
- Ngo uses mass rifle production in the Napoleonic era as an example, showing how technology can turn ordinary people into a military with political power, thereby promoting social equalization.
- Ngo argues that "exit rights" are constrained by two factors: the feasibility of coordination and funding, and repression; technology may lower the former, but it strengthens the repressive capacity of large power institutions.
- Ngo mentions that if a group of people wanted to build their own city in Africa, the main obstacles often come from international intervention rather than funding.
- Ngo cites Nietzsche's "Thus Spoke Zarathustra" view that "where the state ends, there the bridge to the Übermensch begins," expressing the need to free oneself from state oppression.
Game Theory, Norms, and the Critique of Economics
The two sides discuss theoretical work and coordination problems: Ngo proposes going beyond marginalist economics, and a coordination theory grounded in game theory and norms, while criticizing the role economics plays in coordination.
- Ngo proposes a research program of "non-marginalist economics" that no longer relies on marginal calculations of market prices and market value.
- Ngo argues that economics' dominant position is linked to destructive coordination, and that economists performed worst in game-theory surveys because they tend toward "defection" thinking.
- Ngo criticizes game theory for failing to explain how one equilibrium moves to another, and notes that "folk theorems" show that almost any equilibrium may exist.
- Ngo proposes that norms usually have defensive mechanisms, making it hard for individuals to change them alone; some norms were historically designed and established by figures with strong authority.
- Ngo expresses skepticism about "thoroughly solving coordination problems through theory," arguing that theory may be exploited by the corrupt, and cites Kuhn's views on paradigm shifts.
AI Competition, Moloch, and the Domain of Action
The two sides discuss the relationship between AI competition and "Moloch-style" competition, and criticize Scott Alexander's rejection of competition. Ngo argues that competition is a process for creating good, but that it should not be treated as the sole goal; both agree that attention should focus on the limited, controllable domain of action rather than on infinite future value.
- Ngo argues that the competition between OpenAI and Anthropic comes closest to "Moloch-style" competition, in which one party sacrifices other values to gain competitive efficiency.
- Ngo criticizes Scott Alexander for treating competition as "betrayal," arguing that this leads to excessive negation of competition itself.
- Ngo argues that some AI safety researchers use marginal arguments as a pretext to participate in the frontier model race, a phenomenon he finds ironic.
- The moderator criticizes Ngo's view for the problem of "multiplying zero by infinity," namely using "coordination is impossible" to dismiss infinite stakes; Ngo responds that his view is closer to the judgment that "the stakes are not real."
- In the end, both sides agree that they should focus on controllable domains of action and pursue excellence and integrity within them, but they still disagree on whether the value of the future universe is "infinite."