Joe Carlsmith – A Wiser, AI-Powered Civilization is the “Successor” (Worthy Successor, Episode 15)

ByAnnelize P November 14, 2025November 14, 2025 November 14, 2025

This new installment of the Worthy Successor series is an interview with Joe Carlsmith, a senior advisor at Open Philanthropy, whose work spans AI alignment, moral uncertainty, and the philosophical foundations of value.

In this episode, we explore Joe’s reflections on what a “Worthy Successor” might really mean – not as a single entity, but as a civilization guided toward greater wisdom. His framing is rare among alignment thinkers: he treats AI not as a replacement for humanity, but as a set of tools that might help us become clearer thinkers, better philosophers, and truer to what is actually good.

In our conversation, Joe argues that the stakes of AGI are ultimately meta-ethical. If the future is to be guided by more advanced minds, we must still understand what it means for something to be good. Intelligence itself, he insists, carries no guarantee of virtue – the paperclip maximizer remains the clearest illustration of that mistake.

This interview builds a picture of a “Worthy Successor” not as a single conscious system, but as an ongoing civilizational project – a collective that grows wiser through its pursuit of truth, love, and reflection. Joe’s vision is at once humble and audacious: humble in that it resists mythologizing machines, audacious in that it imagines humanity becoming something better through them.

The interview is our fifteenth installment in The Trajectory’s second series, Worthy Successor, where we explore the kinds of posthuman intelligences that deserve to steer the future beyond humanity.

This series references the article: A Worthy Successor – The Purpose of AGI.

I hope you enjoy this unique conversation with Joe:

Subscribe for the latest episodes of The Trajectory:

Below, we’ll explore the core take-aways from the interview with Joe, including his list of Worthy Successor criteria and his recommendations for innovators and regulators who want to achieve one.

Joe Carlsmith’s Worthy Successor Criteria

1. A Civilization Grounded in Truth and Reflection

Joe begins with what he calls epistemic flourishing: a civilization that grounds itself in truth – not as a static belief, but as a process of self-correction and reflection. He sees “access to truth” and the capacity to reason well about value as the bedrock traits of any successor worth trusting with the future. A civilization that can sustain that kind of reflection, and direct its intelligence toward what is true, would be on the right trajectory.

2. Material Abundance and Political Maturity

Joe’s second criterion centers on how civilization treats its own members.
He envisions a world of abundance, safety, and health – not as an endpoint, but as the foundation from which deeper progress can emerge. Political maturity, he argues, is another essential form of intelligence: how wisely we govern and coordinate may determine how long any form of intelligence can thrive.

3. Depth, Beauty, and Spiritual Expansion

Finally, Joe describes what he calls the “softer processes” of civilization – the flowering of culture, art, love, and spirituality.
He wants a civilization that doesn’t just survive, but deepens – one that experiences reality as more meaningful, more alive, and more sacred as it grows wiser. Spirituality, in his framing, is not superstition but a mode of disclosure: a way of contacting reality itself.

Regulation / Innovation Considerations

1.  AI systems should feed into the processes of civilization we already trust

Joe says that what matters most is not the traits of the AI systems themselves, but whether they help deepen the parts of civilization that we already value and rely on. What we should want, he says, are AI systems that help those processes grow and flourish. The goal is not to make the systems themselves spiritual or loving, but to ensure that they enrich and empower those human capacities in us.

2. AIs should help us evaluate whether civilization is going well

Joe says that AIs should empower our ability to evaluate progress – not only in training and testing AIs, but also in assessing civilization as a whole. He suggests that the same kinds of evaluation systems we use for AI alignment could be applied to measure whether civilization itself is on a good trajectory. The AIs themselves, he says, should assist in this process, strengthening our collective understanding of what “going well” even means.

2. Regulation must prevent private actors from risking civilization

Joe says that a coordinated and prudent civilization would not accept the current level of risk from pushing forward toward superintelligence. He says AI labs and governments should act together to avoid irreversible catastrophe, and that regulatory regimes should stop private actors from taking existential risks. Until the world reaches a much higher standard of safety and confidence, he argues, restraint and coordination are essential.

Concluding Notes

This conversation left me with a deeper appreciation for how Joe thinks about the long arc of intelligence. I expected him to be more anthropocentric in his long-term vision, and I’m sure he expected me to be more bullish about accelerating the AGI transition. Instead, we found a great deal of common ground.

Our clearest point of agreement was continuity. Neither of us believes you can simply “drop a pin” somewhere in mind-space and assume value will unfold coherently. Joe described our present perspective on value as “radically impoverished”-not just about what might matter in the future, but even about what value fully means now. That humility struck me as exactly right. Guests like Blaise Agüera y Arcas and Martin Rees have pointed toward similar vistas, but Joe is even more modest about what lies beyond the human horizon.

We also converged strongly on near-term imperatives. Joe emphasized not letting the flame go out: avoiding irreversible catastrophe, resisting the urge to build superintelligence under current incentives, and strengthening our moral and epistemic capacities before anything else. In my own writing, I’ve framed this as our dual mandate: prevent extinction, and do whatever helps the flame of value proliferate.

Where we diverged was around persistence. Joe leaves open the possibility that future intelligences may not revolve around the Spinozan conatus-the impulse to persist and expand. I see persistence not as a moral failing, but as amoral and necessary: without it, no value can continue or unfold. While I agree that persistence alone is insufficient, I suspect it will remain the substrate on which richer aims are built, not something that can be cleanly replaced.

A deeper “difference,” not quite a disagreement, appeared in how we imagine a worthy successor emerging. Joe is open to the idea that the successor could be a levelled-up human civilization – stronger in coordination, moral reflection, epistemics – supported by powerful but non-agentic AI tools. It echoes Toby Ord’s “long reflection”: a period where humanity stabilizes enough to deliberately choose its trajectory. It’s a beautiful vision, and I don’t dismiss it.

But I’m less convinced we’ll get the breathing room. The pressures of destruction and transformation – commercial AGI races, immersive AI environments, AI-mediated identity and relationships, and broader civilizational instability – may accelerate us past the point where a calm, global reflection is possible. Because of that, I put more emphasis on ensuring that whatever systems follow us-merged, symbiotic, or synthetic-truly possess sentience and autopoiesis, so they can carry the flame even if we don’t reach that plateau.

These aren’t disagreements in spirit so much as differences in emphasis. Joe’s humility about value, and his framing of civilization itself as the potential successor, brought out nuances I’ve rarely encountered. For my part, I remain convinced that the next stage of life must preserve sentience and the capacity for expansion, even if we reach it amid more turbulence than we’d like.

It was a pleasure to record this episode, and the distinctions it surfaced are ones I expect to revisit. Joe’s framing – that “worthy” may describe not a single mind, but a civilization capable of thinking clearly together – will stay with me as this series continues.

…

Follow The Trajectory

• Sign Up for The Trajectory Newsletter
• Subscribe to The Trajectory on YouTube
• Subscribe to The Trajectory podcast
• Follow on LinkedIn
• Follow on X