As Much as Humanity Can Hope for in the Long Term
As human beings we are propelled by our varied hopes, our lives are in many ways driven by them. On the day-to-day, we hope to make the sale, to avoid…
This new installment of the Worthy Successor series is an interview with Terrence Deacon, emeritus professor of cognitive science and anthropology from UC Berkeley.
Terrence’s work focuses on language, symbolic reference, embodiment, and the way human cognition differs from other forms of communication and intelligence. In this episode, he explains that language allows humans to share thoughts in a way no other species can.
In this conversation, Terrence argues that current large language models are not living or meaning-making systems, but statistical models built from human-generated text. He describes them as a “deep fake” of intelligence because they can convincingly simulate language without wanting, knowing, or caring about what is true.
We talk about symbolic communication, large language models, teleodynamics, embodiment, and why he sees living systems as fundamentally different from today’s AI systems. We examine his view that intelligence is not only pattern production, but something tied to being in the world, maintaining oneself, and having something at stake.
The interview is our twenty-ninth installment in The Trajectory’s second series, Worthy Successor, where we explore the kinds of posthuman intelligences that deserve to steer the future beyond humanity.
This series references the article: A Worthy Successor – The Purpose of AGI.
I hope you enjoy this fascinating conversation with Terrence:
Subscribe for the latest episodes of The Trajectory:
Below, we’ll explore the core take-aways from the interview with Terrence, including his list of Worthy Successor criteria and his recommendations for innovators and regulators who want to achieve one.
Terrence Deacon’s Worthy Successor Criteria
1. Increasing levels of synergy across living systems
Terrence points to a clear pattern in the history of life: major evolutionary transitions are marked by new levels of synergy, where previously independent or competing entities begin working together. These shifts unlock entirely new capabilities, from multicellular organisms to complex human societies.
He emphasizes that these synergies are not driven by intention, but they consistently lead to more powerful and stable forms of organization. In this framing, a future that has gone well would involve the continued emergence of higher-order cooperation, where systems coordinate rather than compete destructively.
2. Life must continue to generate value (normativity must not disappear)
Terrence defines life itself in terms of value. For him, living systems are those that divide the world into what is good or bad for them. This is not a secondary feature – it is the defining property of life. A flourishing future, in his framing, must preserve this capacity.
He explains that this capacity did not exist before life emerged. In purely physical systems, there is no sense in which anything is better or worse. Value only appears once there are systems for which things matter.
He contrasts this directly with current AI systems, which he sees as lacking this property entirely. They may simulate reasoning or produce outputs, but nothing is actually at stake for them.
3. The future must preserve sentience and experiential life
Terrence makes it clear that what ultimately matters is not intelligence in the abstract, but the continuation of systems that can experience, care, and have something at stake. He distinguishes between intelligence and what we are actually searching for, which is life, consciousness, and experience.
He ties this directly to what a flourishing future would require. While consciousness may be a more complex or higher-order case, what must persist is the presence of sentience – systems for which things can matter, and for which value is real.
He reinforces that this is what distinguishes living systems from simulated ones. Current AI systems may process information and generate outputs, but they do not experience anything, and nothing matters to them. A worthy successor, in his framing, must be part of a world where experiential life continues.
Regulation / Innovation Considerations
1. AI development should continue – but we must understand the architecture that produces value
Terrence explicitly rejects the idea that AI development should simply be halted. Instead, he argues that the real challenge is understanding the kinds of architectures that could actually generate value, rather than merely simulate intelligent outputs.
He frames this as a broader scientific problem that cannot be solved by computer science alone. Understanding how value emerges will require contributions from neuroscience, philosophy, and computational research in parallel with ongoing AI development.
2. AI trajectories will be shaped by competing forces, not centralized control
Terrence frames AI development as part of a broader class of runaway processes that cannot be cleanly stopped or controlled from the top down. Instead, he suggests that such systems are typically balanced by other forces that emerge alongside them.
He reinforces that this dynamic is not hypothetical, but characteristic of how complex systems evolve. Control does not come from a single authority successfully constraining the system, but from interacting processes that limit, counteract, or redirect one another.
This framing implies that governance will not be a matter of halting AI development, but of creating or allowing countervailing systems that can respond to its risks and impacts.
Concluding Notes
I appreciated Terrence’s emphasis on grounding intelligence in life itself. His repeated focus on “skin in the game” – that living systems have something at stake in their continued existence – is a useful corrective to how loosely we talk about intelligence in AI today. It forces a distinction between systems that simulate outputs and systems for which anything actually matters.
I also found his framing of higher-order synergy to be one of the more concrete answers to what a “good future” might look like. Rather than tying success to human survival or dominance, he points to a pattern already present in evolution – systems working together in ways that unlock entirely new capabilities. That gives a clearer lens for evaluating future trajectories than abstract discussions of alignment or control.
That said, I found myself diverging on the implied permanence of the boundary between simulated and real intelligence. While Terrence is clear that current systems lack value, embodiment, and experience, it’s not obvious to me that future systems will remain confined to that category. It seems plausible that architectures could emerge that blur this distinction in ways that are difficult to anticipate.
I also remain skeptical that value and sentience will naturally persist as systems scale. While Terrence emphasizes cooperation and synergy, evolutionary history also includes competition, replacement, and extinction. There isn’t a clear guarantee that increasingly capable systems – even if they are more integrated – will preserve the kinds of value structures he is pointing to.
In fairness, Terrence does not present these ideas as rigid conclusions. He approaches them as part of a broader investigation into life, value, and intelligence. Even where I disagree, the questions he raises – particularly around the architecture of value – feel foundational, and worth continued attention as AI development progresses.
