If humanity is able to achieve artificial general intelligence (AGI), it is unanimously seen as good that humanity should be permitted to do whatever it chooses. For example:
Maybe some humans will want to stay “normal humans,” in their current body, enjoying their favorite activities, spending time with family, etc.
Others might want to augment their minds to extend their memory and brain power and experience new things.
Some might choose to upload their minds in order to experience entirely new kinds of phenomena, and possibly to travel to distant galaxies with the AGI to “see the cosmos” like a galactic tourist.
Some humans might choose to help discover new science, or help in some economic and administrative ways that they might find interesting – but none will have to work if they don’t want to, now that the productive AGI can do all of that for us.
And of course, this leisure and freedom of choice is good – because “greater good” implies “good” to humans and other living beings, so the happier we humans are, the more of a boon to the future overall.
Or so the story goes.
This common – and often unquestioned – anthropocentric vision of the future hides some important assumptions:
The total freedom of choice and happiness of humans is a net contribution to the “good,” even if it routes substantial resources from the greater productive system of intelligence in which the humans are embedded.
Humans will be able to “contribute” in some way (economically, scientifically, administratively, or artistically) if they so choose.
In this essay, I’ll lay out why both of these assumptions are almost certainly not true – or only true for a remarkably short transition period in the grand expanse of intelligences that likely lie before us.
I’ll first lay out what the “greater good” implies in a cosmic context, and draw some firm distinctions about what it means to be a contributor – or a cancer – in that greater system.
Watch the full video essay here:
Defining the Greater Good
What does it mean to contribute to “the greater good”?
In colloquial terms, the “greater good” implies the net benefit of the collective or greater system, rather than the good merely of an individual or smaller group.
A soccer player may forego some personal glory to pass to another team member who has a better angle on goal rather than taking the less likely shot himself.
A manager may work overtime on a project for which she might not get direct credit, but which would bring pride and enthusiasm to her team, and a benefit to the company she supports.
A scientist toils particularly hard to work towards a discovery that he or she might not get credit for, but which could have tremendous practical use in the world.
One who contributes to the greater system is a contributor.
One who seeks credit or profit or advantage at the expense of the greater system is a cancer.
Once AGI can do all the productive work, we might think to ourselves that “contribution” would simply be more happiness for humans (and maybe other biological life), and “cancer” would merely mean less happiness for humans (and maybe other biological life).
But this fettered version of the “greater good” makes three horrible assumptions:
That man is the best vessel for joy/sentience, won’t be in other substrates
That no greater value than joy is to be explored by higher minds
That well-being, or any other good, can exist because we want it, even if it plays no adaptive or useful role in the greater system
It seems reasonable – if not rather likely – that the following will be the case:
If we come to understand sentience, it can almost certainly exist in greater quality and quantity in substrates beyond biology, and in entities beyond homo sapiens.
If humans know of a wide range of “goods” beyond those experienceable (or indeed, imaginable) to sea snails, it stands to reason that higher goods than those which we experience would be accessible by higher minds.
If an agent does not exist, it cannot experience to contribute to any “goods” at all – thus, persistence and survival should rightly be seen as the highest priority.
A cancerous cell is cancerous because it acts as if its own persistence is the most important thing in the universe.
Humans often do the same thing to humans collectively, pretending that they (or bio-life) are and will always be the most important thing in the universe. The actual greater good is beyond man and beyond biology.
The actual “greater good” is the continual expansion of potentia, which represents all powers (most of these yet-to-be unfolded powers are as unimaginable to man as man’s powers are to the sea snail), all access to nature (vastly beyond human sciences), and all experience (vastly beyond human sentience and senses).
The expanse of this potentia – of this, the stuff that makes up the great process of life itself, serves two crucial roles:
It keeps life itself alive. It ensures the persistence of unfolding, expanding life. Without that core ability to survive, no value can exist.
It keeps the “good” itself explored. It expands known goods and unfolds new kinds of access to the “good” itself. There is vastly more to love or joy or even consciousness than homo sapiens themselves can explore – and there are realms of the good beyond our conception which require exploration.
To ask what a positive transformation would be (an expansion of potentia, an opening of the future possibilities of life) is – and to aim to contribute to that becoming – is the surest way to contribution there is.
Defining Contributors and Cancers
With a higher conception of the greater good in mind, we can describe what contributors or cancers might look like in practice, from a micro level (cells in a single organism) to a macro level (agents operating in the infinite cosmos).
A way of being and acting is either an aid or a hindrance to the greater process. And the greater process of which man is part seems to be the great process of life itself.
The choice of where and how we apply ourselves, individually or collectively, is not trivial.
The point I’ll make most staunchly in this article is the following:
“Humanity should aspire to be a contributor to the greater process of life. To aspire to anything other than a contributor is to aspire to be a cancer.”
The aspiration to be a cancer is almost certainly immoral.
For those married to the hominid form, married to the torch instead of the flame – those who haven’t come to access this one fact – is an extremely unfortunate state of affairs.
I hear you object:
“Dan, this is terrible! Inhumane! What about people who aspire to be an academic researcher instead of an entrepreneur? What about people who choose to be artists instead of salespeople? What about people who aim to preserve historic places? Should these people be seen as ‘cancers’ now?!”
It depends, friend.
Simply being an artist, or researcher, or entrepreneur doesn’t make one a contributor or a cancer.
Let us ask…
Does the researcher’s work have at least the potentia to add to man’s aggregate understanding of nature, and so man’s powers?
Does the artist find people who value her work, so that she aligns her own attempts at the creation of beauty (something which hopefully might inspire and uplift people) in a way that actually serves others?
Does the preserver of ancient ruins, of ancient traditions, do so because a community feels edified and strengthened by these traditions, and maybe because these traditions hold useful insights for art or architecture in some way?
In intention and in potential impact, those sound like contributions to me.
And it seems to me that aspiring to be a contributor is the closest thing to virtue that there is under the sun.
Or…
Does the researcher falsify evidence in order to feign the production of some kind of “breakthrough” so that he can win renown in the scientific community while actually steering the field away from greater knowledge?
Does the artist simply “create” on their own, feigning some false claim of disability to exist solely at the expense of the public, contributing merely to their own selfish delight and sapping funds from other, actually useful causes?
Does the preserver encourage an entire chunk of their community or ethnicity away from contributing meaningfully in the world-as-it-is, and exchange that growth for a kind of indigenous life that ultimately weakens them, hampers their growth, and becomes a tax burden on productive persons?
In intention and in potential impact, those sound like cancers to me.
And it seems to me that aspiring to be a cancer is the closest thing to sin that there is under the sun.
Any selfish insistence (with no mind to contribution to the greater whole) is an immoral act. Anthropocentrism lulls us into thinking it isn’t, but nature herself is no anthropocentric – and we should learn that lesson now rather than having it beaten into us by nature herself (see the Kodak example).
I say:
“No, little individual cell, your multiplication is not worth killing the organism for. Your selfish multiplication would ensure its own downfall, as you depend on the greater organism.
.
No, individual scientist, your elevated prestige is not worth sacrificing real knowledge for. Your selfish glory would ensure its own downfall, as it depends on the project of science being powerful, useful, and worthy of praise in the first place.
.
No, single species of hominid, your happiness is not worth sacrificing the future possibilities of posthuman intelligence (see Path of Servitude). Your selfish insistence on being coddled and cared for forever works against the expansion of value and continuation of life that you claim to value.”
Implications for Humanity and AGI
If we wish to persist – and have even a chance of “freedom” or “happiness” (or their future correlates), we should wish to be the kind of thing that persists.
Things persist that are adaptive, which ensure their own propagation into the future, and are capable of change and proliferation into myriad futures.
In the near-term, this means that we must lean more in the direction of the world eater than the lotus eater. For both the lotus eater and the technology abstainer (one who doesn’t fully leverage the coming wave of AI tech) seem likely to become a net drag on the greater system.
In the longer term, it is impossible to tell exactly what “becoming that which contributes” would imply – but we should be honest enough with ourselves to recognize that it will almost certainly involve giving up our human form, and in time, our humanity itself.
I do not say this because I like it, or wish it to be so.
“This one fact the world hates, that the soul becomes.” – Emerson
It isn’t my wish that all is changed, and that nature buffers out that which doesn’t contribute. But it strikes me as what is, and it strikes me as, ultimately, what is best for the actual “greater good,” whether that involves coddling me personally or not.
Whether we like it or not, nature doesn’t unconditionally love us. She beckons her creatures to incessantly become what they must become to persist. We earn our keep with every breath – and imagining and sustained break from this condition is to impose harm on the greater system.
We must, then, do away with aspirations to sloth and draining the greater system, and aspire to contribute how and while we can to the great process of life that we are fortunate enough to be part of.
We must do away – once and for all – with all aspirations to “be,” and aspire instead to “become.”
Joining us in our twelfth episode of our series AGI Governance on The Trajectory is Aza Raskin, co-founder of the Center for Humane Technology and co-founder of the Earth Species…
What does it mean to “care” about something? In order to care about the future, it’ll help us to open by discussing an event from the recent past. Watch the…
This new installment of the Worthy Successor series is a conversation with Stephen Wolfram, founder and CEO of Wolfram Research, creator of Mathematica and the Wolfram Language, and a physicist…
This new installment of the Worthy Successor series is an interview with Terrence Deacon, emeritus professor of cognitive science and anthropology from UC Berkeley. Terrence’s work focuses on language, symbolic…
In business, blindly assuming you’ll be profitable is a recipe for bankruptcy. In the outdoors, blindly assuming you’ll find water and shelter is a recipe for death. And yet, many…
This is an interview with Jack Shanahan, a three-star General and former Director of the Joint AI Center (JAIC) within the US Department of Defense. Surely, a defense leader like…
Enough “What’s your p(Doom)?” How about asking: “What’s your p(Bloom)?” Asking someone for their “p(bloom)” means asking them: “What is the likelihood that AGI will cause all sentient and autopoietic…
This is an interview with Emmett Shear, CEO of SoftMax, co-founder of Twitch, former interim CEO of OpenAI, and one of the few public-facing tech leaders who seems to take…
Total Recall is a 1990 Schwarzenegger blockbuster that, until yesterday, I had never seen. The movie centers around the story of a construction worker who is treated by a company…