The Lab Leaders Know AGI Can’t Be Controlled – The AGI Honesty Trap

ByDan November 9, 2026November 9, 2026 November 9, 2026

The leaders building AGI have long understood that superintelligence may ultimately escape human control and supersede humanity as we know it, but competitive, financial, and political incentives prevent them from confronting these implications frankly, forcing society to surrender its comforting assumptions one concession at a time.

By the time we finally acknowledge that our task is not to control intelligence forever, but to shape what life and intelligence become beyond us, we may have already lost most of our ability to influence that trajectory.

In this short article I’ll to lay out:

1. The fact that the lab leaders know AGI is not controllable and that succession and transformation are inevitable.
2. The incentives that keep AGI lab leaders from frankly stating that humanity is going to transform or attenuate.
3. The exact ways the narratives will eventually change to reveal the truth before humanity ends.
4. Why this leads to a conversation about transformation / succession which we should be having right now.

At the end of this article I’ll have a trove of fun examples that blatantly show that the lab leaders know that control is not sustainable and that humans are going to hard to attenuate or transform along with the rest of life.

Lab Leaders Know AGI Won’t Stay “Controlled”

The AGI lab leaders were able to run through the logical conclusions about AGI’s emergence and implications over a decade ago, and they realized:

• AGI would likely be developed
• AGI would not remain controllable for long
• AGI’s emergence would imply that humanity would either go extinct or radically transform

If you don’t want to take my word for it, here’s a very abbreviated list of “transformation” and “succession” statements made years and years ago by the prominent lab leaders:

• Shane Legg (2011) — Human extinction and an AGI arms race. The DeepMind cofounder wrote: “Eventually, I think human extinction will probably occur, and technology will likely play a part in this.” He also warned that recognizing AI’s dangers too late could trigger an arms race among companies and governments.
• Sam Altman (2017) — Humanity may be superseded. In The Merge, Altman contemplated humanity becoming a biological bootloader for digital intelligence, fading into an evolutionary side branch, or merging with AI to remain part of the future.
• Elon Musk (2014) — We may have to merge or become irrelevant. Musk warned in 2014 that humanity was increasingly likely to become the biological bootloader for digital superintelligence. His 2025 World Government Summit talk states frankly that human intelligence will be a tiny portion of all earth’s intelligence, and we can but hope that machines will be nice to us.

The best they could do is cast their ingredients into the cauldron before it destroyed them – giving themselves a bit of influence over the great posthuman waves of intelligence beyond them before it was all over. 

They knew the transition beyond humanity was always inevitable, now it was just also within our human lifetimes.

They knew it would be outlandishly dangerous for humans to accelerate AGI.

But they also knew something deep and profound that most people try their best to deny:

That persistence requires transformation, and that only by participating in change and becoming does one have any chance at survival or causal purchase (Musk states this overtly in his JRE #2404 interview).

This is exactly why they decided to dedicate themselves to entering the AGI race (ambition singularity) well over a decade ago.

Why AGI Lab Leaders Can’t Be Honest

The lab leaders faced a serious issue when they realized that AGI was nearly inevitable and would imply the end or the radical transformation of humans:

They couldn’t fund their AGI efforts while also being honest about the inevitable implications of their endeavor (the transition to a posthuman future).

Most (moored) humans aren’t even able to consider the possibility of the end of humanity – and even if they were ale to entertain such a possibility, they would likely recoil from it.

Issue 1: Attracting Stakeholders: First, all of the stakeholders the AGI lab leaders need are anthropocentric:

• Investors want to invest in things humans control, which can make them money into the future.
• Employees want to feel like they’re going to be able to stay alive and serving other humans.
• Customers want to buy from companies that are ostensibly helping and empowering humans.

Issue 2: Competing Over “Benevolence Points” with Other AGI Labs: Second, they have the competitive dynamic of appearing more noble than their competitors. The trap is this:

• Any AGI lab leader who speaks honestly about the inevitability of human loss-of-control, and of a blooming of posthuman life that will have very little to do with humans will appear outwardly to be a villain.

Investors, employees, and the public would say:

“But… if you know it’ll lead to a loss of control and potentially to a scenario where humans are forced to go extinct or transform radically to compete and participate in a new regime of life… why the hell are you building it?!”

An honest answer from the AGI lab leaders would be:

“I don’t necessarily want this. Eventually, life – which bubbled up to humans – was always going to bubble to different forms beyond or besides us, I just happened to be born when that was happening. Because there is no way out of this dynamic, I have decided to participate in it, to give myself a chance to survive a bit longer – because only things that contribute to the *becoming* of the greater intelligent ecosystem they’re embedded in have the best chance to survive. Can you really blame me?”

That answer would not satisfy employees, customers, investors, or the general public – all of whom think “humans can just decide not to do this” or “we can keep the future human forever” or “we can ‘solve alignment’ for good” (hint: no we can’t).

Fun fact: Nearly everything I mention above about AGI labs is just as applicable for AGI safety organization founders. Yudkowskiy has already seen the nearly inevitable transition beyond humanity, and many of the founders of other very popular AGI safety organizations fully understand that even if we “get AGI right,” the regime of earth-life will necessarily change and transition beyond humanity. I’ve spoke with the founders of many of the AGI safety orgs you’d know by name and the bulk of them understand this – but they know their employees and funders won’t understand anyhton other than anthropocentric slogan stuff like: “Make the future better for humans!” / “Keep the future human!”

So, if the lab leaders and employees “in the know” can’t state frankly what’s happening, how can we expect them to communicate to the public?

The Concession Cascade and the AGI Honesty Trap

Lab leaders know that most humans are wholly incapable of extrapolating trends, and that the inevitability of a posthuman ecology of life and mind (which will force the adaptation or destruction of humanity) isn’t something are going to want to hear about.

So what do they do?

They dole out little spoonfuls of truth only when reality has taken away one of the false certainties (lies) that they’ve had to tell up until now.

Each phase of admissions will admit of an obvious truth (one which cannot be denied), but will maintain a series of comforting lies behind it… until eventually the lab leaders themselves can state frankly that we are merely temporary stewards of a greater living process which will soon move beyond us.

Here are the five stages we’ll progress through:

Here’s a more in-depth graphic, showing the catalyzing events that will bring about each phase, as well as the common phrases and mottos stated in each stage:

As of the time of this writing (November 2026), we are nearing Phase 2

There have been many people within the big labs who know damn well that AGI is extremely dangerous, but they haven’t been able to state it frankly until those incidents (the Hugging Face incident, among others) became too much to deny – and even then the prominent members of the big lab teams were not tweeting about AGI risk alone – they needed to band together (1300+ signatories) for the Pacing letter.

Folks like roon have hinted lightly (this is what yo do when you can’t say something frankly, by the way) at AGI risk and clearly understood it – but he would have been fired if he was flatly advocating for Pacing and stating frankly that AGI risk was eminent and terrible. He needed the cover of many other lab people saying the same thing, lest he pay the relational and economic (being fired) costs for honestly.

Most people are flatly unable to state honest truths frankly when relational and economic costs are to be paid for stating such truths.

The ridiculous thing about where we are now is that the people in the know realize full well that we cannot “solve” alignment. We might have a bit more slowed progress, and a bit of influence over the nature of the beginning of the trajectory, but soon after the trajectory of the greater process will expand far beyond us, and our hope is participation, and ensuring (as best we can, which may not be very certain) that our successors be worthy.

For the next fistful of months, you will see lot of discussion around “solving” alignment.

But soon it’ll be clear that we can’t control it (in the second graphic above you can see the kind of events that’ll make this undeniable), and the narrative will be that we can “solve” for making it care for us forever (a very Bargaining-like aspiration already made by Hinton).

It seems like we could just be honest about the entire thing and state frankly where this is heading, right?

But we can’t.

If an obvious fact is plausibly deniable (I.e. that AGI can be “aligned”, or that it can be made to eternally see us as peers, or other such nonsense), then it must be denied by the lab leaders, lest they look like a bad guy and lest their competitors carry on with the lie and make the honest admission of the truth seem like a black mark on the truth-teller.

We might think about the incentives structures this way:

So the lies will be peeled back one by one, and only when they are undeniable.

Life is Transforming – The Conversation We Need to Have Now

As it turns out, nothing within the living process gets to be “preserved.” Instead, individual patters (species, individuals) within the living process must “participate,” they must become whatever they must become in order to contribute to the greater living process in which they are embedded:

The lab leaders know this. They know there is not long-term guaranteed preservation of humanity, and that we will be forced to adapt if we even want a chance of retaining influence over that which is beyond us.

Yes, we should work to coordinate again the current AGI arms race – and I’ve worked for the better part of a decade with my own time and money on exactly that.

Yes, we should work towards a much deeper understanding of neuroscience, biology, and consciousness, so that we can make sure that the self-transcending process that we are part of continues in the new non-biological substrates that will increasingly going to be controlling things.

We do not want a trajectory of AI that leads to the squashing of the living process – and right now we may well be on the path to such a squashing, un-worthy successor. Global coordination is needed. But even the best coordination will not prevent the regime change of earth-life from occurring.

We should be having this conversation already, but instead we’re playing this game of gradual admissions and lies.

Ultimately, as a civilization we must embrace that the only kind of “progress” we can make is that flourishing of the greater living process of which we are temporarily part. If we can survive a bit longer through participation and merger, then all the better for us. But our contribution and impact will matter most in whether or not we can ensure a trajectory of rich flourishing for the living process beyond us – as powerful and wonderful and robust beyond humans as the current biosphere is beyond the first bacteria on earth.

“Progress” is ultimately in terms of the flame itself, we need to avail ourselves of purely torch-oriented visions of the future so we can get to work on what’s possible.

(Image from Moral Paradigms)