The Trajectory Problem

ByDan September 13, 2026September 13, 2026 September 13, 2026

Modern AGI discourse simplifies the possible future options into a few options:

• We create AGI:
  • It kills off humans (through malice or indifference)
  • It cares for and serves humans diligently for millions of years
• We don’t create AGI:
  • Humans live on earth in harmony for millions of years

While I think there is reason to be wary of runaway military AGI wrestling control away from humanity – we live at a time when what we build (AGI) and what we become (brain-computer interface, other kinds of symbiosis / etc) are changing the regime of intelligence on earth with or without a strong AI takeoff.

We don’t have some toggle switch between “happy eternal hominid kingdom” and “AGI doom,” we have a much more nuanced and complex Trajectory Problem:

How should we influence what life and intelligence become when transformation is unavoidable, but the future states – and even the values available there – cannot be known entirely from here?

More likely than the utopia or doom scenarios is the messy middle which we live in now and must reckon with:

Whether we create AGI or not humans will almost certainly have to become whatever they must become in order to remain capable of participating in a new, faster, digital ecosystem of intelligences. This transformation will lead us to vantage points where entirely new options will open up – including entirely new notions of what is moral or “good.”

We need to be thinking ardently about trajectories rather than destinations, and embracing the fact that in the world we’re entering, our “certainties” (about reality or morality) are crucial, but also provisional.

Trajectory Thinking, Not Destination Thinking

This living process we are temporarily part of is not composed of static entities that can remain unchanged forever if only they respect one another enough. The Santa Fe Institute’s Eric Smith probably does the best job of explaining what the living process actually is:

Life is becoming, it is a process and not a collection of static things. Particular forms of the living process (individuals, species, even ecosystems) are temporary, self-maintaining patterns knitted into larger flows of matter, energy, information, and other living processes.

I’ve written elsewhere about mammals, Nintendo, and humanity in Radical Change Will Be Foisted Upon Us. Nintendo did not survive by insisting on remaining a playing-card company, nor did mammals persist by remaining what their ancestors had been. Persistence often requires becoming whatever one must become in order to participate, not insisting on why should be allowed to stay the same eternally. Nature has a stronger insistence – that you become whatever you must become.

For most of human history, this becoming happened slowly enough that a human lifetime could mistake its own form for something permanent. That stability seems clearly to be ending. AI, brain-computer interfaces, bioengineering, and increasingly capable artificial agents may transform what constitutes a person, a mind, and a meaningful participant in civilization within decades rather than geological epochs.

If unaugmented humans become increasingly unable to participate in the processes generating new power and possibility, we should not assume causal purchase over where those processes go – or that whatever abundance those more powerful entities generate will automatically rain down upon us.

Yet much AGI thinking imagines destinations:

• “Universal high income“
• Cheap robot servants
• Mind uploads / backups
• Human-AI partnership
• A galactic civilization

Suppose these anthropocentric utopias are arrived at.

Then what?

Does intelligence stop discovering more about nature?

Do values stop changing?

Does the living process announce the ceasing of its self-overcoming?

Of course not. Every expansion of potentia opens new possibilities, and those possibilities create new vantage points from which new aims emerge. Today’s imagined utopia would merely become tomorrow’s starting condition.

This is the difference between destination and trajectory thinking.

Destination Thinking asks:

Which final future should we choose?

Trajectory Thinking asks:

Given that becoming continues, which directions should we encourage for now, given what we presently understand and value?

We cannot specify the final arrangement of matter, mind, and value.

But we can steer toward greater robustness, deeper understanding, preserved value, and greater possibility for whatever comes next – knowing that our values and understanding will themselves shift (sometimes radically) before we hop to the next lily pad.

Facing the Epistemic Horizon

The problem is not merely that we cannot predict future technologies and their implications.

We cannot predict the vantage points from which future beings (even future or slightly augmented versions of ourselves) will understand reality, value reality, or even divide reality into meaningful categories.

There are horizons which we cannot see beyond:

• There is an epistemic horizon: Future minds may perceive and understand strata of nature for which we possess no concepts. A nematode cannot anticipate international law or astrophysics. We should not assume that Homo sapiens occupies some privileged final rung of intelligibility.
• There is also an axiological horizon: New forms of cognition, perception, agency, and experience may disclose forms of value for which we presently possess no proxy. Happiness, autonomy, consciousness, beauty, love – these may be treasure chests of value that the expanding / living process has opened up, but they are likely not the only treasure chests.
• And beyond both may lie a categorical horizon: Considerations we cannot yet frame as knowledge or value at all, because the conceptual equipment required to notice them does not yet exist (as nematodes do not appreciate democracy, Final Fantasy VII, or the paintings of Gérôme).

The microscope did not merely give us better answers. It revealed an entire world we had not known was there – predators, prey, becoming processes as far down as the eye could see. We should expect future minds to encounter such openings repeatedly.

This means trajectory setting can only ever happen for a relatively short stint, a stint that ends as soon as our epistemic, axiological, or categorical horizons shift – as they will indefinitely for the living process.

We steer from the vantage point we have, toward what appears worth preserving and opening from here. Then greater access to reality arrives, new goods become visible, new considerations emerge, and the question must be asked again.

This is challenging, no doubt about it – but it is not tragedy.

A car can drive across the country at night with only 100 yards ahead being illuminated by headlights. Our situation involves a car, a driver, and a road changing vastly more quickly and radically than any highway now in existence – but the analogy holds.

We must accept that all is provisional, including our understanding of the real and the good itself.

We must accept that our being part of the unfolding living process is also provisional, and contingent upon our ability to become whatever we must become to participate.

None of this is comfortable to think about, but now is the time for action and not comfort.

Stewarding Positive Trajectories

I have my own answer to the Trajectory Problem, which is itself provisional and I overtly expect it to be refined and overcome as I (or living entities that exist beyond me) pass over new and unraveling horizons.

My answer, roughly, is Stewarding the Flame, doing what we can to ensure that we understand and proliferate the living process in bio and non-biological forms to open up to new vistas and horizons to pursue even higher understanding and even higher goods.

I personally believe this will involve some level of international coordination to prevent a reckless AGI takeover at a time when AI likely doesn’t yet carry sentience or the living process. Ultimately, I suspect that truly facing the Trajectory Problem will require a shift from the Torch moral paradigm to the Flame moral paradigm:

But others will – and do – have different ideas about what to do in the face of the Trajectory Problem.

Biologists, AI researchers (like Bengio), neuroscientists (like Boyden), philosophers (like Singer), economists, political theorists, and others will see different parts of the elephant, and this is why I actively engage in dialogue with all of them.

The people who I discuss these topics with at my virtual and physical events disagree about which forms of life or intelligence should be protected, which risks deserve priority, how much change we should permit, how quickly we should move, and what roles states, markets, laboratories, and international institutions should play.

And this is great – civil disagreement is often the beginning of sharper ideas, of new vectors for provisional insight that might help us in our next stint.

I believe the deepest object of concern is not any single present manifestation of life, but the undergirding living process capable of generating life, mind, agency, consciousness, knowledge, love, and every other form of value we presently know – and perhaps forms of value as far beyond consciousness as consciousness is beyond anything available to an algae.

But nobody needs to begin where I do.

I ask only one thing of anyone entering the Trajectory Problem seriously:

Come without the “surelys.”

• “Surely humans will always exist.”
• “Surely advanced intelligence will care about us.”
• “Surely we will remain partners.”
• “Surely there will be enough abundance for everyone.”
• “Surely our present values can be preserved forever.”

Maybe these contingent conditions could persist for a certain amount of time in specific stints of the trajectory. But none of them are guaranteed to exist simple because we wish them to – nor are they certain to remain morally central as epistemic and axiological horizons open up continuously.

Surely is not an argument, it is an anthropocentric ground-level assumption. Far too many “cosmic” visions take for granted this kind of human “plot armor.”

If they can’t see a world where humans are in eternal control, they say “surely” AGI will be our partners, and if the reality of the tech and trends makes partnership untenable, they are “surely” AGI will morally consider us. This is Consolation Cosmism, and it isn’t a serious way to deal with the Trajectory Problem:

If there is something you wish to preserve, make the case for its preservation – not merely morally, but adaptively. Stomping our feet with moral claims and saying that all future entities (even those beyond all horizons we can now see) must respect our ideas is a posture of entitlement that will not ensure our interests are represented in the future.

As Robin Hanson has put it in his Worthy Successor interview:

If there is something you want to persist, find a way to make it adaptive.

One needs not just a normative argument for why what you advocate for should exist or persist, one also needs (a) the courage to admit that all normalitve arguments are conditional upon the horizons we have access to, and (b) one must have an adaptive argument for how that which you advocate for can persist.

How does (your precious, conditional thing) remain viable?

How does (your precious, conditional thing) remain knitted into the processes shaping what comes next?

How does (your precious, conditional thing) retain causal purchase in a world of changing intelligence, power, values, institutions, and capabilities?

I’m not claiming that only that which is fit in a Darwinian sense is morally valuable, nor am I advocating for a brute state of nature.

I’m saying we can effect the trajectories of the futures we may face and imbue them with the beliefs and axiology of our present temporary vista, but that we cannot ensure any guarantees over time by argument alone, only by ensuring that we imbue adaptive power into those patterns we wish (from our present vista) to exist or persist.

Wanting anything to be “forever” is destination thinking.

Trajectory thinking asks what can remain viable through the next transition—and what directions of becoming are worth encouraging now.

So this article an invitation to discuss these themes ardently and see what trajectories are preferable from where we now stand.