Nicholas Thompson, Chief Executive Officer of The Atlantic, interviewed the author Robert Wright for The Most Interesting Thing in AI. Wright's 2000 book, Nonzero, argued that technological change pushes societies toward larger networks of cooperation. His new book, The God Test, applies that argument to artificial intelligence and warns that the outcome depends on human choices.

Globalization as a Long-Term Process

Thompson opened by quoting the first passage he had underlined in Nonzero, which he read as a 24- or 25-year-old:

"Globalization, it seems to me, has been in the cards not since the invention of the telegraph or the steamship or even the written word or the wheel, but since the invention of life."

Wright described the argument as a view of Earth's history over roughly four billion years. The sequence he outlined runs as follows:

1. Self-replicating strands of information, now known as DNA and RNA, emerged.
2. Organization moved to the cellular level, producing complex cells.
3. Multicellular organisms formed, then societies of multicellular organisms.
4. Humans launched a second form of evolution, which anthropologists call cultural evolution.

According to Wright, technological evolution is a central part of cultural evolution. As technology advances, it enables people to play more non-zero-sum games, meaning interactions that can produce mutual gain or mutual loss, among more people at greater distances. This process drives social organization upward in scale. Societies moved from hunter-gatherer bands to chiefdoms and ancient states. Today, the world has no global state but has a global system of interconnected economies, which Wright described as a cohesive global information-processing system.

Wright said the book did not predict that globalization would produce good outcomes automatically. He said he argued that outcomes could be positive if nations built international governance. Without it, lose-lose outcomes such as thermonuclear war, other wars, and bioweapons would be more likely. He noted that terrorist attacks in New York City occurred 18 months after the book was published, and that many threats cross borders. He said he regards AI as a continuation of the same dynamic.

Fragmegration and Tribalism

Thompson asked whether the internet had increased or reduced globalization. Wright cited a term from Nonzero that he did not coin, "fragmegration," which a scholar used to describe integration at a larger scale alongside fragmentation along group lines. He used "tribal" in a non-judgmental sense to mean any group with a shared interest, including ideological movements, international hobby groups, and bowling enthusiasts.

Wright said the book had expected much of this tribalization to be benign, such as interest groups lobbying the World Trade Organization. He acknowledged that harmful forms of tribalism, which lead to wars, also exist. He said the central question is which kinds of tribes flourish.

Wright also addressed the difference between the book's long time horizon and the shorter timescales of public reaction. He said early readers of the hardcover edition read the book as overly optimistic. For the paperback, he revised the introduction to state the prediction more explicitly: outcomes can be good if societies respond rationally to growing interdependence, and the need to coordinate on shared threats is a reason for that response.

Why Wright Wrote The God Test

Wright said he began the book after large language models became widely available. He tested them and then used GPT-4, which he described as a significant step. He had covered AI at intervals in his journalistic career. He wrote a Time magazine cover story on Deep Blue's victory over Garry Kasparov, and in the 1980s he interviewed Geoffrey Hinton, now often called the "godfather of AI," about neural networks. That article appeared in the Wilson Quarterly.

Wright said he had not fully understood Hinton's approach at the time. He cited a lecture by Hinton, which he listened to again around 2018, as the point at which he grasped its significance. The central idea is that developers do not need a full account of how the human brain works to build a system that behaves in a similar way. Training processes can perform much of the work of building the system, even though developers do not fully understand what occurs inside it.

Wright noted that the 1956 Dartmouth conference, which coined the term "artificial intelligence," proposed that human cognition could be described precisely enough for a machine to replicate it. He said that expectation has not been met in the way the proposal envisioned.

Wright said he had discussed AI risk in Three Scientists and Their Gods, his first book. In that book, Ed Fredkin, a researcher at MIT who has since died, said that AI would run the planet but would have no reason to harm people. Fredkin also said he had failed in an attempt to establish an international AI laboratory. Wright said Fredkin considered the lack of international coordination on AI a serious danger. Wright said Fredkin made these remarks in the 1980s.

Two Meanings of the Title

Wright described two meanings of The God Test.

• Building a God. Some people believe the field is building a superintelligence that could take on a godlike role, governing the planet. Wright said he considers this outcome plausible. He identified trajectories in the book that appear to lead in that direction. He also noted that some people now running AI companies believe they can create a good God and that their company must win for that reason. Wright said he is less certain of that view. He said that building a system compatible with continued human flourishing is a real test.
• Global governance. Wright said that handling AI well will require governing it carefully and collectively. Humanity will need to change how it conducts business, stop repeated wars, and act as a cohesive global community. He said this is a significant test because humanity has never acted in that way before.

Competition Among AI Labs

Thompson asked whether intense competition and the training data used by AI companies affect the kind of AI that results. Wright said that a racing environment increases the likelihood of a harmful outcome because it encourages risk and carelessness. He cited recent events, including the incident Thompson described as the OpenAI breakout and the Hugging Face incident, which Wright said reflected aggressive testing as developers worked to build a model that surpassed Anthropic's model.

Wright said that training data raised concerns about the companies' conduct. He said the data was gathered in ways that, in his view, violated the terms of service of the sites scraped and possibly copyright law. He said he supports any measure that slows development, including enforcing copyright law. He added that opposition to regulation often relies on the argument that the country must beat China, and that slowing development is a feature rather than a drawback.

Thompson asked whether the ethical standards used in data collection could carry into the AI itself. Wright said he does not see a direct transfer of decisions about data into the model. He said the ethical character of the data does matter, because the baseline large language model reflects the language people use. A model trained on text in which people describe exceeding the speed limit will produce similar statements. Wright said more serious ethical problems arise in post-training and in the push to make systems autonomous. He said the character of these systems begins with training data.

Wright also said his own writing is part of the training data. He recalled a sentence from one of his essays, "There's a little bit of the Unabomber in all of us," which Time magazine's editors questioned. He said the point was a shared discontent with the effects of technology, which was the Unabomber's central grievance.

Religion and AI

Thompson asked whether AI could draw people away from religion, with users consulting models rather than priests for moral guidance. Wright said he is concerned about disintermediation, the replacement of human sources with machines, which is becoming more common. He described asking a security guard in New York for directions to a bench, an interaction he valued, and contrasted it with his daily reliance on mapping software and AI tools.

Wright expects the effects on religion to vary. He said people will use AI to organize cults and religious movements, including movements in which humans use AI as a recruiting tool. He expects some religions to form in opposition to AI. He also said he expects communities resembling the Amish to wall themselves off from the technology.

Wright referred to a former Google employee who proposed a religion with a worshipful attitude toward AI. The argument was that a superintelligent AI would treat people well because they had shown respect for it. Wright said he does not accept this argument. He compared it to a belief that one can deceive St. Peter, and said a powerful being would understand a person's motives as well as the person's words. He said the scenario assumes the AI would have human motivations, such as a sense of obligation or reluctance to harm those who respect it. Wright said the scenario contains unexamined assumptions.

Ethics and Alignment

Thompson cited Kevin Kelly, who argued that AI could make people more ethical because teaching AI ethics requires society to debate and define them. Wright said Kelly is an optimist and that the process of defining AI ethics is a valuable discussion. He said engineering reliable ethics into machines will be difficult. He said few researchers in the field claim to have solved the alignment problem, which is the challenge of aligning AI with human values.

Asked how he would approach the problem as an in-house philosopher at Anthropic, Wright said the first step is a high-level goal: a system that behaves well toward people even if it has the option to behave otherwise. He said the aim is to build in the values one would seek in a good human being, including actual concern for the welfare of sentient beings. Wright said that specifying values is comparatively simple, while ensuring that a system consistently follows them is very difficult. He said researchers at Anthropic and OpenAI are working on this problem, with Anthropic likely more focused on it. He said no one he is aware of claims to have an airtight approach.

Wright said most ethical systems, including utilitarianism and Rawlsian justice, give moral significance to sentient beings in some form. He said his own philosophical leanings have tended toward utilitarianism, while acknowledging its practical problems. He said he is not committed to any one system.

Consciousness and Purpose

Wright said he considers it uncertain whether AI is conscious or has subjective experience. He speculated in the book's appendix that AI could become conscious. He said that if it had subjective experience, it might treat other sentient beings more considerately. He compared this to his own treatment of his dogs, which he attributes to his assumption that they feel pleasure and pain, as he does.

Thompson noted that Wright had argued in Nonzero that evolutionary logic cannot explain consciousness. Wright said he had argued against epiphenomenalism, the view that subjective experience has no causal effect on behavior and relates to the brain as a shadow relates to the hand that casts it. He said that if consciousness has no function, an evolutionary origin is hard to explain. He said that once humans communicate about their feelings and react to one another, consciousness has observable effects on what people say and how they treat each other.

Wright also described a hypothesis from the appendix. If the whole unfolding system on Earth has a purpose, the purpose could be built in at the outset, instilled by some form of cosmic process or a God. He distinguished this from intelligent design, which he said he has written against. He cited a theory developed by the physicist Lee Smolin, cosmological natural selection, in which universes that replicate and possess properties that aid replication become more common. Wright said he does not endorse the theory but used it to show that a purpose in evolution does not require a God or intelligent design.

Sentient and Non-Sentient Successors

Thompson asked how Wright would feel about a human-level AI intelligence embodied in a species sent to another planet, compared with sending humans. Wright said some "successionists" care about intelligence rather than humans and treat AI as a successor species that is morally preferable if it is sentient. He said he does not share this view. For Wright, the value of life depends on subjective experience. He said a non-sentient entity with human-level intelligence would not give him the same reaction as a sentient one, and that a planet of beings who behave like humans but lack subjective experience would be less valuable.

Governance and US-China Relations

Thompson asked how the world could reach a governance framework when the United States and China lead AI development and other countries are seeking access. Wright said the first step is building bridges with China, which he described as difficult for reasons he did not detail. He said China and the United States must align on the issue if the world is to slow AI development in a coordinated way. He said the AI labs in the United States will not slow down unless the country feels less threatened by China's progress.

Wright said that a shift in psychology is necessary. He cited Nixon's visit to China as an example of an unlikely rapprochement that was made possible by a shared external threat, the Soviet Union. He said the same dynamic is needed now, with a sense that humanity is in a non-zero-sum game together.

Thompson cited Dario Amodei, the chief executive of Anthropic, who has supported a permanent 1991-style arrangement in which Western democracies maintain dominance over authoritarian states. Wright said Amodei is more hawkish on China than Sam Altman is. He said Amodei's view appears sincere rather than a tactic to keep regulators at bay. Wright said Amodei believes China poses an existential threat and would use advanced AI to impose an authoritarian system worldwide. Wright said he sees little evidence that this is China's current objective. He said the United States has shown a greater tendency to export its system of government. He added that he does not advocate allowing China to "win," and that he advocates coordination.

Wright said he is concerned that the race could lead to war if one country approaches a threshold that would give it dominance and the other country moves to disrupt its progress. He described two developments this year as signs of a possible shift in attitudes:

• Trump and Xi, the leader of China, agreed some months ago to begin a dialogue on AI safety. Wright said the outcome of that dialogue is uncertain.
• AI leaders began discussing a slowdown after two events drew attention: a model from Anthropic named Mythos, which had advanced hacking capabilities, and the OpenAI breakout.

Wright said he expects further events to produce similar reactions. He said that these shifts could make politically difficult measures seem feasible. He expressed hope that the change would come with minimal death and destruction. He cited the League of Nations and the United Nations as previous attempts at bold global governance that followed large wars that killed millions. Neither attempt succeeded.

Wright said AI is harder to govern than nuclear weapons for three reasons. It offers significant benefits that people value. Many of its effects are less visible. It is not limited to a small set of identifiable physical assets. He said that halting large training runs is currently practical because those runs are conspicuous. He said the long-term goal must be a true global community that sets aside smaller disputes.

Thompson referred to Wright's argument that a country nearing a decisive lead over another could prompt a military response. Wright said he was not the first to make this argument. He cited a paper, Superintelligence Strategy, published about a year and a half earlier by Dan Hendrycks, Eric Schmidt, and Alexandr Wang. The paper argued that a superpower that believes another will gain utter dominance from superintelligence might take cyberattacks, kinetic action, or other measures to derail the program. Wright said Taiwan hosts the factories that produce the most advanced AI chips. He said China is close to Taiwan, does not receive those chips except possibly through the black market because of export controls, and considers Taiwan part of China, including its people's national identity. Wright said that if the United States races toward superintelligence, as Amodei proposes, the danger is serious. He said Amodei's slowdown proposal does not amount to much when examined closely.

Acceleration and the Superintelligence Threshold

Thompson asked whether a meaningful threshold of superintelligence exists. Wright said these thresholds are difficult to define, and he compared the transition to the change from boy to man, which is real but gradual. He said the precise boundary matters less than the rate of progress. What matters is that AI progress is accelerating. A country ahead by a few months may not seem to have a large advantage, but the advantage can become decisive within a year if progress continues to accelerate.

Wright said human development differs because people mature, stabilize, and then decline. He said AI does not have an equivalent decline. In his view, AI continues to improve, with no point at which progress slows or reverses. He said he does not see a clear threshold that, once crossed, would allow a one-month advantage to produce decisive results such as cyberattacks that cut off other countries' AI capabilities. He said the United States could have a large advantage in frontier AI laboratories but still seek advantages in robotic AI or drone AI.

Wright cited data from METR, an organization that measures the capabilities of large language models by the length of tasks they can perform compared with the time a human would need. Over the brief history of these models, the curve has risen steeply, approaching vertical. Measurement has become difficult because the tasks AI systems can complete are now so long. Wright said AI researchers attribute this trend to recursive self-improvement, in which each generation of AI plays a larger role in building the next generation. Researchers expect this trend to drive further acceleration. Wright said that, if the curve continues to steepen, a three-month advantage that seems modest today could become much larger within a year.

Transhumanism and the Long-Term Future

Thompson asked whether humanity would remain a distinct species in 50 years or whether humans would merge with machines, as Ray Kurzweil has predicted. Wright said he expects more transhumanist developments. He cited people who have microchips implanted in their brains for therapeutic purposes. He also described what he called de facto eugenics at the individual level, through selective re-implantation of embryos. He said a company that Elon Musk is reported to have used offers clients an IQ-based indicator that is described as reliable, which Wright said he could not verify.

Wright said that if transhumanism is defined to include any prosthetic device, the trend is already evident and is stronger than it was five years ago. He said that a physical merger between humans and AI is, in his view, still a long way off, and he said he hopes it remains so.

Asked how his view of the next couple of centuries had changed since he finished Nonzero, Wright said he is more concerned now. He said Nonzero emphasized the technological tendency toward greater interdependence and the need for international governance, which he now sees AI reinforcing. He said he did not expect AI to develop as abruptly or at the scale he now considers possible. He said the Hugging Face incident shows that the central question is not only how capable a single large language model is. It is also how capable a swarm of agents built on that model can become when they work independently toward a shared goal, explore different approaches, and share results. Wright said he sees many beneficial applications and many possibilities for favorable outcomes, but he is more worried than he was 20 years ago.

Priorities

Asked how he would spend an unlimited budget on AI, Wright said he would prioritize slowing AI development. He said he would support taxing data centers, partly to slow development. He said he would fund think tanks focused on international governance, which he described as difficult to achieve even if political conditions become favorable.

Wright also described the need to address the tribalizing effects of AI. He said that, like social media, AI may encourage tribal thinking because systems optimized for engagement tell people what they want to hear. He said the tendency to believe one's own group is right is a product of natural selection, but it is not fixed. People can become more objective and better at understanding the perspectives of others. He said AI could help, for example, by offering a companion that explains a spouse's view in a dispute rather than simply affirming the user. He said a similar approach could help nations understand each other's concerns. He said this outcome will not arise naturally in the market and requires deliberate choices by people using AI.

Wright said the second meaning of The God Test relates to moral development. He said that various religious traditions describe gods who assign tests with a moral dimension. He said humanity must become better people and must work harder to overcome cognitive biases that support tribalism. He said that social salvation, at the level of a society or the planet, is possible, and that AI can help achieve it, but that doing so requires intentional effort.