Richard Ngo – A State-Space of Positive Posthuman Futures [Worthy Successor, Episode 8]

ByDan April 25, 2025April 25, 2025 April 25, 2025

This new installment of the Worthy Successor series is an interview with the brilliant Richard Ngo, an AGI researcher who recently worked for years imagining positive AGI futures for OpenAI (and spend his time before that at DeepMind and the Future of Humanity Institute at Oxford).

In this episode we go deep on Richard’s beliefs of what a “good” AGI would be – and some of the reasons he posits for why it might treat humanity well. He also explores some what what he wishes innovators would do in order to more robustly understand the drives and intelligences of AGI machines – moving beyond the current Skinnerian paradigm of analyzing AI minds.

Richard has a considerable optimism not only in some humans remaining recognizably human for millions of years, but also in AGI generally treating humans well and even considering us as moral standard-bearers for their own future actions. I save the bulk of my disagreements with these positions for the conclusion of the interview.

The interview is our eighth installment in The Trajectory’s second series Worthy Successor, where we explore the kinds of posthuman intelligences that deserve to steer the future beyond humanity. This series references the article: A Worthy Successor – The Purpose of AGI.

I hope you enjoy this unique conversation with Richard:

Below, we’ll explore the core take-aways from the interview with Richard, including his list of Worthy Successor criteria and his recommendations for innovators and regulators who want to achieve one.

Richard Ngo’s Worthy Successor Criteria

1. Respecting and building on human values

Like Nick Bostrom (in our 1st Worthy Successor episode), Richard wants human values to be expanded off of, not torn to pieces and replaced wholesale with entirely new and alien values.

He wants an AGI That can maintain a procedural approach that respects the origins and predecessors (humans), and demonstrate a willingness to recognize commonalities and maintain some connection to their ancestral origins. In the interview he states that he thinks its reasonable that far-future AGIs, millions of years into the future, might ask “what would humans do?” in order to make moral judgements, maintaining a continuity with humanity through eons.

2. Larger scales of cooperation and love

Richard believes that a worthy AGI would cooperate, love, and compete peacefully at increasingly larger scales. It wold have the ability to form larger-scale entities through potential mergers and interactions (like cells form tissues, or humans form organizations or nations – but on a vastly larger and more varied set of scales).

3. Capable of pursuing increasingly ambitious goals

I suspect this likely overlaps well with the idea of expanding potentia. When I asked Richard what “pursuing increasingly ambitious goals” might mean, he said that he suspects it’ll mean something like “Managing more complex and harmonious civilizations.”

Regulation / Innovation Considerations

1. We must move beyond behaviorist “AI psychology”

Richard’s concrete recommendations for innovators include:

• Shift from a behaviorist approach to a more scientific understanding of AI cognition
• Focus on understanding the internal representations and generalization processes of neural networks
• Move beyond just input-output mapping to study:
  • How concepts are arranged inside neural networks
  • How these concepts relate to each other
  • The deeper mechanisms of how neural networks actually work
• Prioritize interpretability research to better understand AI systems’ internal workings

2. Encourage some great power cooperation to avoid tar pits

Michael expresses concern about the arms race between the US and China in developing AGI, and the lack of a mature science of diverse intelligence to guide these efforts. He suggests that slowing down the current pace of technological development may not be an achievable goal, given the momentum behind these efforts.

While he does not think slowing down is feasible, he suggests the possibility of some coordination between entities like the US and China to “steer clear of the visible and invisible ‘tar pits’” – which are potentially harmful developments in the race to AGI.

3. Better governance frameworks must be developed

Richard is notably cautious about proposing specific governance frameworks. He suggests that the primary bottleneck is not people’s awareness of AI risks, but having a genuinely good plan to address those risks. He personally could aim for governance mechanisms that infringe as little as possible on individual freedoms.

He also hints that governance will likely evolve as AI capabilities accelerate, potentially making current slow coordination methods obsolete. The key is to have a plan that can adapt quickly and maintain trust across different stakeholders.

Richard does not provide a detailed governance blueprint, instead emphasizing the need for more research, understanding, and careful planning before implementing any governance strategy.

Concluding Notes

I found Richard’s light and non-dogmatic approach to big questions intellectually refreshing. He navigates complex themes – like values, psychology, and AGI – without clinging to rigid conclusions, and that openness makes his ideas both engaging and thought-provoking.

I especially appreciated his notion of “values” as interaction patterns between agents, untethered from human-centric definitions. His comparison between early psychology and today’s understanding of large models was also apt—we’re at risk of becoming behaviorists with LLMs, focusing too much on input-output while ignoring inner mechanics. But perhaps, given we built these systems, we stand a better chance of grasping their inner workings than we do with our own minds.

That said, I found myself diverging from Richard on several key intuitions. His implication that AGI might naturally pursue a “benevolent society” as its endgame struck me as unfounded. Evolution doesn’t move toward “nice,” and intelligence beyond humans may optimize for goals completely inaccessible to us – unfathomable even if they’re more ambitious.

Similarly, ideas like AGI showing eternal reverence to humanity, or “granting” us Earth out of a kind of parental gratitude, felt to me like misapplied anthropomorphism – like a theological remnant sneaking into futurism. Referencing human ideas of decision-theory (as justification for why AGI would care for us) feels implausible if these systems are truly beyond us. If rodents had a “decision theory”, surely we’d operate beyond it.

In fairness I didn’t get the sense Richard was deeply tied to these views (he admirably juggles ideas lightly), but even their suggestion seemed far outside what I consider likely or even coherent.

Finally, I raised questions about how Richard squares his skepticism about current AGI trajectories with what feels like an optimism underpinning his tone. If we’re not on the path to a worthy successor, that should catalyze urgency in governance.

He noted that today’s governance paradigms are insufficient – a point I agree with – and hinted that he’s working on frameworks of his own. I’d love to see those take shape, and I’ll surely continue following his work. Check out Richard’s Twitter account here.

…

Follow The Trajectory

• Sign Up for The Trajectory Newsletter
• Subscribe to The Trajectory on YouTube
• Subscribe to The Trajectory podcast
• Follow on LinkedIn
• Follow on X