AGI Red Lines and Green Arrows

ByDan March 22, 2026June 11, 2026 June 11, 2026

Recently, there has been a concerted effort within the United Nations to establish a set of international red lines for AI development, with the expressed goal of getting developed nations on the same page about which developments are explicitly dangerous to develop.

In relatively short order, the call for red lines received a number of prominent advocates, including the likes of Yoshua Bengio and other prominent figures in academia and policy.

But this is only the beginning of a greater process of civilizational coordination in the face of wildly powerful and ultimately uncontrollable AI.

International coordination – if we ever achieve it – will inevitably begin with a defensive motive; to “keep humanity in control.”

But this motive will be increasingly untenable as:

• New kinds of agency and power emerge from AI systems
• Humans increasingly offload both strategy and execution to AIs
• Human relationships and cognition are increasingly mediated by AI
• Biotech and neurotech blur the lines of what “human” and “life” mean

Eventually, coordination should serve not only to prevent runaway risks and destruction, but it should also serve to steward the greater process-of-life that is increasingly unraveling in new non-biological forms, and at faster, non-biological speeds.

Watch the full video essay here:

In this article, I’ll outline the process of international coordination we should hope to see among first-world powers in the coming years:

• Risk Realization -> AGI Red Lines: First, humanity will progressively realize how unprecedented and potentially dangerous and destabilizing AGI is, and it will respond more and more with new kinds of coordination to prevent AGI’s potential runaway risks. “Red lines” will start as an idea, and move eventually into some attempts (and potentially successes) at real international law or intergovernmental / inter-organizational coordination.
• Process Realization -> AGI Green Arrows: Next, whether or not humanity is able to achieve reasonable coordination around AGI, it will become self-evident that the rolling, self-overcoming process of life (of which humanity is part, and non-biological substrates are becoming part) is going to keep expanding beyond human civilization and humanity itself. At this point, “green arrows” will emerge – proposals for international innovation and coordination that might lead to the greatest flourishing for the living process beyond man.

Here’s a quick visual diagram to outline the distinction between the two:

I’ll explain both dynamics in the sections below, along with some important considerations for encouraging international coordination around both (a) risk reduction and (b) encouraging flourishing.

Red Lines Emerge from the Realization that AGI is Dangerous / Destabilizing

Throughout history, when humans unlock new powers, we eventually develop new ways to coordinate around them.

The pattern is consistent:

• First, new capabilities expand the range of what is possible.
• Then, civilization develops new forms of governance to prevent catastrophe.

Examples include:

• Mass-produced food: Led to the FDA and new coordination around ingredients, health impacts, etc
• Global aviation: Led to international airspace rules and safety standards
• Nuclear weapons: Led to nuclear treaties and non-proliferation agreements
• Genetic engineering: Led to bioethics norms and laboratory safety protocols

Each case reflects the same underlying reality: When the power of a technology grows large enough, coordination must grow with it.

Artificial general intelligence will force this dynamic to operate at a much larger scale.

AGI is not simply another industrial technology. Many of the researchers building these systems — including Geoffrey Hinton, Yoshua Bengio, and Ilya Sutskever – have suggested that advanced AI may begin to resemble a new kind of cognitive system rather than a traditional machine.

The red line realization will likely be:

AGI will potentially destroy all earth-life, and escape our control immediately.

The resulting coordination will likely be:

• A tapestry of international law, checks and balances among great powers, coordination and shared safety protocols among AGI labs, etc.

AGI red lines have long existed in principle (Bostrom, Yudkowski, others), and they’re now starting to gain more traction in real policy. It will take a powerful global realization of shared risk in order to enact more serious laws.

Nuclear war was seen as totally inevitable even by geniuses like von Neumann, but through a campaign of public awareness and a shared fear of mutually assured destruction, humanity was able to stave off that “inevitable” result. With great deliberation and collective effort, the same might be the case for preventing the brute AGI race.

(This article is more about painting the dynamics of what catalyzes collective effort, not laying out proposals for the “how.” For specific proposals around AGI coordination, see my own essays Unite or Fight and SDGs of Strong AI — or the good work of the CIGI, or the ideas in Narrow Path, both of which have reasonable ideas for what coordination might look like in practice. The ideas of Dean Xue Lan are particularly strong.)

Green Arrows Emerge from the Realization that Intelligence Expansion is Inevitable

After the risks become clear, a second realization will follow.

Even if societies attempt to slow the development of advanced AI, the broader process will continue.

Scientific discovery, economic incentives, and geopolitical competition will all push the technology forward, and will push the evolution of life (increasingly non-biological) forward.

Trends will emerge that simply won’t be denied, and will shatter the paradigm myth of eternal human dominance/relevance:

• Human beings will increasingly have relationships with AI, and live mostly in immersive AI worlds rather than the “real” one.
• AI will appear to have traits very much like life, and questions of AI consciousness and moral patienthood will become more and more prevalent and important in technical and policy discourse.
• Cognitive enhancements will become viable, permitting some humans to have vastly more cognitive ability and memory than unaugmented humans.
• The entire economy may be run by vastly more capable, fast, and proactive AI systems – buffering humans out of any position of relevance, not just in the muck-work, but in the strategy, vision, and creative roles.

Attempting to permanently halt this expansion of intelligence would likely fail. And even if it succeeded, it would suppress many developments that could expand human knowledge and capability.

The green arrow realization will likely be:

The ever-changing process of life itself is evolving through symbiotic and non-biological forms, and is going to extend beyond humanity no matter what.

The resulting coordination will likely be:

• Some level of coordinated international efforts around innovation directions to influence the trajectory of life itself in directions more likely to lead to the flourishing of life itself.

If red lines are coordinating around prohibitions, green arrows are shared orientations for development. The latter follows from the former:

They identify areas where expanding intelligence is most likely to increase the flourishing of life and knowledge.

Examples of possible green arrows include:

• Deepening understanding of intelligence and consciousness
  • Research into cognition, consciousness, and mind
  • Exploration of the biological and computational foundations of intelligence

• Human–machine cooperation or symbiosis
  • Technologies that enhance human capability rather than replace it
  • Safe forms of brain-computer interfaces and cognitive augmentation

• Scientific and medical discovery
  • Accelerating research in medicine and biology
  • Expanding understanding of complex living systems

• Understanding the processes of life itself
  • Research into self-organizing and self-creating systems
  • Deeper study of how complex living systems generate new capabilities

The purpose of green arrows is not to rigidly plan the future, but to influence the direction of the rolling process of change to lead to what are hopefully better aggregate outcomes for life.

(Again, this article is not a full-blown proposal about what the international green arrows should be. I go into some proposals for these kinds of priorities in Our Final Imperatives, and in SDGs of Strong AI. These kinds of initiatives should continue to be fleshed out thoroughly, but we should imagine that they will not gain real international traction until after the red lines are in place, and until the green arrow realization occurs.)

We Need Both Red Lines and Green Arrows

Ultimately, we will need both selective prohibitions and selective directions of development.

Both together will form a kind of conscious evolution of the greater process of life on earth.

In the philosophy of Spinoza, potentia inevitably expands, as life (which has a primary mandate to persist – a conatus) needs to constantly unfold new powers and capabilities to ensure its survival in an unpredictable and changing world.

But there are times when too much power threatens to destabilize the entire system, and so the living process bounds some of its powers in a selective way (think: Traffic laws, nuclear non-proliferation treaties, etc). Spinoza refers to these forces as potestas.

Here’s a visual explanation of both forces:

The goal of these bounding forces of potestas is not to eternally bar the living system from expansion; it is to stabilize a certain set of powers, so that new magazines of powers (potentia) might unfold beyond it.

In other words, for Spinoza, potestas is good only when it allows for the continued expansion of potentia.

This is his core metric – and it should be for us as well.

The international community will inevitably see (and hopefully coordinate to prevent) AGI’s risks well before it will digest the painful truth that life is an ever-expanding process, not a collection of static species with one inevitable “main character” social mammal.

The hope is that green arrows will follow.

And maybe it will be more than a hope if we work diligently towards proliferating the conversation about AGI and the trajectory of life. This conversation-fostering is, above all else, the purpose of this blog, our interviews with AI and policy leaders, and our physical events.