Epistemic Horizons and “Tents of the Night”

ByDan October 7, 2026October 7, 2026 October 7, 2026

The eye is the first circle; the horizon which it forms is the second; and throughout nature this primary figure is repeated without end. It is the highest emblem in the cipher of the world. – Emerson, Circles

People discussing the future of AGI make a lot of claims about what should happen.

• Humans should always remain in control of all future intelligences.
• AI should always respect human autonomy by doing XYZ.
• Any kind of consciousness entity should always have its preferences respected.
• A good future must include this; an advanced intelligence must never do that.

From our present vantage point and in our present circumstances, some of these claims may make rational sense, and may even be a prudent course of action.

But there’s a serious problem with should, must, never, and always when we’re standing at the edge of entirely new kinds of minds coming into existence.

Every claim about what is real, what is good, and what should happen is made from a particular cognitive vista. My own beliefs are produced by a human brain, five rather limited senses, a particular life history, a particular civilization, and the sliver of nature modern science has managed to expose.

Change the vista and the conclusions may change with it.

This is already obvious within human life. Our beliefs change when we encounter new evidence, experiences, technologies, or perspectives – and they also change as a consequence of the developments of our brain as we age (toddlers aren’t physically able to empathize in the same way a mature adult can, for example). Across cultures and centuries human values and beliefs have altered radically. The scientific method has not wiped away these disagreements, but has given us shared vistas to stand from, from which we can (and have, in nearly every field of endeavor) find new vistas to look out from.

This dynamic becomes vastly more impactful if we consider minds vastly different from our own.

Three Humility Horizons

“It seems to be a rule of wisdom never to rely on your memory alone… but to bring the past for judgment into the thousand-eyed present, and live ever in a new day.” – Emerson, Self-Reliance

For an unconscious single-celled algae, there is no real (what is) or good (what should be).

For a trout in the ocean, there may be a salient sentient experience of what is real, but no appreciable notion of morality.

For a lemur, there is a relatively clear idea of the real, and also many salient notions of the good in the form of social norms, reciprocity, and inter and intra-clan dynamics.

Are we to presume that these are the final categories of all possible experience simply because they’re the two that we humans have access to?

An attitude of “the only categories of consideration are those which I (the present observer) currently have access to” is clearly flawed. Through the development of minds, nature has unfolded not just “more” or “different” access to the real (physical senses, cognitive ability, etc), but entirely new categories of consideration.

For this reason we might consider three separate horizons that bound our present human vantage point:

The Epistemic Horizon is the boundary of what a mind can presently know, perceive, detect, or understand.

• A nematode cannot see.
• A dog inhabits an olfactory world largely unavailable to us.
• Before microscopes, humans could stare directly at pond water without knowing that an entire living world was moving before their eyes.

The microscope did not create that world. It expanded our access to reality.

There is no particular reason to believe that our present access is final. New instruments already allow us to detect things our biological senses cannot – and new augmented senses might open up new magazines of access to nature that make possible entirely new kinds of tools that we can’t today imagine (as a blind mole couldn’t imagine the telescope).

Future AI minds may possess vastly greater memory, new sensory modalities, different cognitive architectures, or entirely new ways of modeling nature.

The Axiological Horizon is the boundary of the goods and bads a mind is capable of apprehending.

• A nematode cannot appreciate Shakespeare.
• A rodent cannot value constitutional government or scientific discovery.
• Our primate ancestors could not debate animal rights or the moral status of artificial minds.

As cognitive capacity expanded, so did the space of things that could matter. I’ve argued something similar for years under the term Moral Singularity: sufficiently different and expanded minds may develop modes of valuing inaccessible to us, just as much of our moral world is inaccessible to simpler creatures.

Humans often presume that the pain-pleasure axis of conscious experience must be the final frontier of all moral relevance – but because there are creatures with no sentient experience, it is reasonable to presume that we humans, similarly, are wholly incapable of experiencing new magazines of value for which we have not the imagination or faculties to speculate about.

This doesn’t mean we should rush recklessly into that unknown. I’m advocating in this essay, and in a hundred others, that we do not.

But, again, even epistemic and axiological horizons likely do not exhaust the problem…

The Categorical Horizon is the limit of the kinds of considerations we can formulate at all.

Real (epistemic) and good (axiological) are categories available to us. Consciousness, individuality, rights, experience, matter, selfhood, agency: these too are ways human minds divide and interpret reality. We should be cautious about assuming that the conceptual furniture available to Homo sapiens exhausts every category through which vastly expanded minds could encounter existence.

This is not an argument that future minds will necessarily be wiser than us in all cases. Greater intelligence might discover terrible things, adopt terrible aims, or destroy itself – these seem to be trajectories we should avoid.

It is an argument about epistemic humility.

As potentia expands, cognitive light cones expand with it. New facts become visible, new experiences possible, new goods or values may become accessible. It seems rather likely that entirely new categories of consideration emerge beyond humans, as they emerged leading up to humans.

As we climb the heights of new strata of mind, we must take the fleeting nature of our vantage point into account as we set trajectories into a future moving at blazing speed.

Normative Claims as Tents of the Night

“We are by nature observers, and thereby learners. That is our permanent state. But we are often made to feel that our affections are but tents of a night. Though slowly and with pain, the objects of the affections change, as the objects of thought do.” – Emerson, Love

Claims about truth and value are provisional.

They are our best guesses from the vista we now look out from – and they can’t be claimed to be anything more.

When it comes to past moral claims, we totally understand this dynamic. Here are a fistful of examples of moral perspectives that were absolutely “normal” until rather recently in the developed world – and which are still upheld by huge swaths of the human population today:

• Non-human animals were seen as inanimate property.
• Marriage was a practical, economic, and social contract rather than a romantic partnership based on love.
• Domestic violence was seen as a private matter, not a legal one.
• Slavery was seen as an obvious and natural part of the human condition by many civilizations.
• Interracial marriage was seen as an obvious detriment to one’s bloodline or social standing.

Circumstances and perspectives changed, and new normative beliefs became common over time. Yet each generation lulls themselves to sleep with the notion that all those past moral oversights were must building up to the inevitable and inviolable moral norms that exist today. The “right” normative beliefs, the “sacred” ones.

But all is provisional.

Even the most obvious-sounding normative claims are obviously fluid when we examine them up close – all of them would be wholly thrown into question with new evidence or new circumstances.

Sacred Premise 1: Humans should be in charge of all future intelligences

“Keep the future human,” they say. AI systems might be allowed to manage supply chains, help to inform policy, do all the physical labor, and perhaps even make the bulk of all scientific discoveries – but they should report to humans, and ultimately serve humans.

Inconsistencies in putting this idea in practice:

• Do humans who augment their brains drastically to wield more control over artificial intelligence count as “human”? We are already turning into cyborgs in a thousand ways, do we expect a worldwide ban on any deeper forms of human-machine symbiosis than we have today?
• If companies who use AI outcompete those who don’t, then companies managed by AI may soon outcompete those that rely on direction from human owners.
• When the outputs of AI come at such speed and complexity that humans cannot understand or integrate them. Already we see the AGI labs using AI to build their AI, and AI to monitor the “safety” of their AI – a process clearly hurling beyond human control. Is this not a canary in the coal mine for the rest of the economy, and perhaps the sciences and government as well?

Factors that could undermine this “sacred” belief:

• Cognitive assemblages: The future fill be full of distributed systems in which humans, machines, algorithms, institutions, and information flows jointly perform cognition and produce outcomes. Competitive organizations in the economy and the sciences will need to integrate multiple kinds of minds into its systems to stay relevant and competitive. Asking “Who is in charge?” becomes increasingly ill-posed. Agency becomes a property of the assemblage rather than a single sovereign actor.
• Machine sentience: Already we see moral concern for artificial intelligences becoming a viable philosophical challenge, with entire roles at some of the large AGI labs for “model welfare.” Whether AI is conscious or not, if humans believe that they are conscious, and increasingly treat them as partners and persons rather than as mere tools, then mandating eternal servitude of these entities to human whims may be seen as morally untenable – especially as more and more people depend on AI for friendship (18% of U.S. adults under 30 say they have an ongoing personal friendship with an AI chatbot – per YouGov), learning, and other relationships previously reliant on other humans.
• Human augmentation: As humans not only immerse in AI relationships, but potentially into VR / AR worlds, and potentially with invasive or non-invasive brain augmentation – the very definition of “human” will be up for grabs, as will the concept of “identity,” broadly. Saying “humans should be in charge” will mean humans-as-they-are to some people, but will imply an accelerating transhuman lineage to others.

Historical precedents of similar change:

• Normative perspectives of who should be “in charge” have changed drastically. Some nation-states think they should be in charge. It was common for races or religions to believe that they should be in charge. Men were “in charge” in the household in the developed world until recently – and in a great many places they still are. Circumstances change, societal norms change, ideas of “rights” change. They don’t “change until the present moment, then get locked in forever,” they are ever in flux – and as new minds and cognitive assemblages emerge, they’ll likely shift much more radically than before.
• In the theocratic middle ages, God was seen as obviously and rightfully “in charge.” A fifth to a third of all land was owned by the church during some periods of the middle ages, and tithes and rents to the church itself were seen as the obvious natural order of things. The rise of cities and trade, an increase in labor’s bargaining power after the Black Death, and other factors led to more modern nations and then to more independent ideas of commerce and property.

Who is “in charge” has been conditional and contingent in the past, is conditional and contingent now, and will be conditional and contingent into the future.

Sacred Premise 2: All sentient beings should have their preferences respected

If anything sounds like moral progress, this does.

Humans shouldn’t be tortured or enslaved. Dolphins shouldn’t be needlessly killed in fishing nets. Chimpanzees shouldn’t be subjected to agonizing medical experiments. As our circle of moral concern has expanded, it has become increasingly natural to suppose that sentience itself – not species membership – ought to command moral consideration.

Maybe so.

But even this apparently enlightened principle becomes slippery almost immediately.

Inconsistencies in putting this idea in practice:

• We already radically grade the preferences of sentient beings. Most societies prohibit many forms of animal cruelty while permitting factory farming, animal experimentation, pest extermination, pet ownership, fishing, hunting, and killing animals for food. Even among people who agree that animal suffering matters, there is enormous disagreement about how much it matters relative to human preferences.
• “Respect their preferences” has no obvious stopping point. A chicken presumably prefers not to be slaughtered. A zebra presumably prefers not to be eaten by a lion. A cat prefers killing a mouse; the mouse prefers remaining alive. Nature is composed of mutually incompatible interests. Respecting every preference is not merely difficult – it is logically impossible. Even if predation and parasitism could be completely eliminated (almost certainly a terrible idea, this decreases the thrival of living systems), the same matter and energy would be preferred to be used by multiple entities at once. Trade-offs do not disappear.
• Sentience itself does not tell us how moral weight should be distributed. Does a human’s intense preference outweigh ten chickens’ preferences? Does a whale count more than a mouse because its cognitive world is richer? Does temporary suffering matter differently from death? “Respect sentience” leaves nearly all of the difficult allocation questions unanswered.

Factors that could undermine this “sacred” belief:

• Discovery of what consciousness actually is: We presently do not know why or how particular physical processes produce subjective experience. Suppose consciousness turns out to be distributed, field-like, graded in unexpected ways, temporally nonlocal, or dependent upon organizational properties extending beyond an individual body. The morally relevant unit might no longer map cleanly onto one skull = one subject = one bundle of preferences. A radically different theory of consciousness could produce a radically different theory of what deserves consideration (see Levin on collective intelligence).
• Continued inability to determine what consciousness is: The opposite outcome causes trouble too. If consciousness remains scientifically inaccessible, civilization may increasingly rely on proxies: cognitive complexity, integrated agency, cognitive light cone, self-modeling, communication, planning, creativity, or other observable capacities. Future AIs, cyborgs, collectives, and cognitive assemblages may possess these proxies in combinations and degrees unknown today. Moral consideration could consequently become graded along dimensions quite different from the simple binary of sentient/not sentient, and there may be individuals or collectives that demonstrate vastly more of these proxies than individual humans (as humans demonstrate more of our current proxies of consciousness than dogs or beetles have today). What then?
• New kinds and scales of experience: Future minds might contain thousands of distinguishable streams of experience, merge and divide, copy themselves, operate at radically different subjective speeds, or share information so intimately that ordinary individuality becomes difficult to locate. What constitutes a being, a preference, or even one experience could cease to be obvious.

Historical precedents of similar change:

• In almost every society (and surely in the case with many existing societies), it was only men – or potentially property-owning men – whose “rights” mattered in any appreciable way. Behooving the preferences of women, or men of a foreign (i.e. bad and inferior) race or religion, or children would be seen as a sick extravagance, ridiculous at best and actively undermining to civilization and order at worst. This perspective has changed and expanded over time – and obviously continues to do so – and would obviously continue to do so when minds beyond humanity come into being.
• Whaling was among the most important part of many eastern and western economies leading up to the industrial era. It is remarkably recent than “animal rights” and things like “dolphin-free tuna” (do we care not for the [somewhat obviously sentient] tuna itself?) have become popular. This is not some inevitable march towards a singular and obvious direction of abstract “moral progress,” but a shifting of norms as our epistemic grounding (specifically: our understanding of whales’ role in the broader ecosystem we depend on, and the discovery of lighting methods that didn’t involve the dangerous and difficult pursuit of killing whales).

Sacred Premise 3: Humans should not have to change or evolve if we don’t want to

So maybe a business has to change to accomadate new customers, new supply chains, and new circumstances.

And perhaps all other species now in existence (outside of the temporary and contingent custodial niche of being in a zoo or being a human pet) have had to consistently change and become in order to thrive.

But we can recognize those demands for change is immoral and bad. If we want to, we can step outside of the need to participate with the greater living process. Such participation is for animals, and for companies, but not for humans, who can and should be above this process.

Let’s put this one to the test as well.

Inconsistencies in putting this idea in practice:

• We already have to change in order to participate. “Be whatever you want to be” has never meant “the world must reward whatever form of participation I prefer.” A martial arts instructor can love teaching martial arts, but if his community primarily wants children’s classes, fitness training, or private lessons, he has to adapt what he offers to what people around him actually value. He can teach martial arts when he does so in a way that is net-additive to the living ecosystem in which he is embedded. He does not get claim to equipment, a building, and revenues because he “wants” to be a martial arts teacher. He can only persist as one by participating – and the same can be said for any career or ambition.
• Modern humans constantly augment themselves and change. Literacy, formal education, eyeglasses, pharmaceuticals, smartphones, search engines, software, vehicles, and institutions dramatically alter what an unaided human can do. We usually don’t call this “ceasing to be human” because the technologies arrived incrementally and became familiar. No one today demands a white collar salary without computer literacy / typing skills – we change constantly (in our morals, in learning new software, in moving to new cities with better housing prices, etc). This drift will continue into myriad forms.

Factors that could undermine this “sacred” belief:

• Most people may voluntarily choose substantial augmentation: If memory enhancement, disease resistance, new senses, BCIs, or intimate AI symbiosis become safe and useful, millions or billions of people may want them. “Protect unaugmented humans” is one proposition; “prevent other humans from becoming something different” is another entirely.
• Participation requirements will almost certainly rise drastically: A person once needed handwriting and arithmetic to participate in much of commercial life; later came typing, computers, email, spreadsheets, smartphones, and an expanding stack of digital tools. If future scientific, economic, or political systems operate at machine speed and complexity, remaining causally involved may require deeper forms of augmentation (read: World Eaters, Ambitious Tech). The cost of staying a participant will be the cost of changing from being a human, just like every species of living pattern before you.
• The unavoidable tradeoffs of living systems: Life persists partly through consumption, displacement, competition, predation, cooperation, reproduction, and transformation of environments. An ethic requiring the indefinite satisfaction of every existing preference could conflict with the generation of new life, new minds, new ecological possibilities, or new forms of value. The question may increasingly become not “How do we satisfy every preference?” but “How should conflicting forms of value and becoming be weighed?”

Historical precedents of similar change:

• Our ancestors repeatedly transformed their way of life to remain viable in altered environments. Hunter-gatherers became agriculturalists; agricultural societies urbanized; industrial workers acquired entirely new disciplines of time, literacy, machinery, and organization; information workers continuously acquire new technical capabilities. None of these transitions was painless, uniformly voluntary, or universally beneficial—but none left human life unchanged.
• At the deeper biological level, persistence has never meant preservation of form. Our lineage did not persist by insisting upon remaining aquatic vertebrates, early mammals, or ancestral primates. Bodies, senses, brains, social arrangements, and behavioral capacities changed as environments and opportunities changed. The present human form is itself the temporary product of innumerable departures from previous forms.
• Technological capabilities repeatedly move from luxury to advantage to expectation. Literacy, mechanized transportation, telephones, computers, and internet access began as capabilities most humans lacked. Eventually environments reorganized around them. What once constituted augmentation became ordinary participation. Grandmother saw electricity, airplanes, television – but she also only had to learn one thing in her career: to type. We have to learn 50 software programs per year to have a chance of keeping a job. Soon the demand for relevance will be radical transformation beyond humanity.

…

We could use another hundred “sacred” moral values and explain their flimsiness in the face of epistemic, axiological, and categorical horizons.

We can have favorite moral principles, we can have beliefs and habits that we consider best – but we must preface “best” with “best given our current vantage point.” We may live firmly by principles today, but be flexible enough to change all of them as reality demands.

The time of static notions of the real and good is gone. At the dawn of minds beyond humanity, we must be prepared for all vistas to open.

Adaptive Normative Claims

“Beware when the great God lets loose a thinker on this planet. Then all things are at risk. It is as when a conflagration has broken out in a great city, and no man knows what is safe, or where it will end. There is not a piece of science, but its flank may be turned to-morrow; there is not any literary reputation, not the so-called eternal names of fame, that may not be revised and condemned… The things which are dear to men at this hour are so on account of the ideas which have emerged on their mental horizon, and which cause the present order of things as a tree bears its apples. A new degree of culture would instantly revolutionize the entire system of human pursuits.” – Emerson, Circles

Okay, so our vistas of moral consideration are all temporary.

What are we supposed to do?

Throw away our normative ideas entirely?

Certainly not.

I value consciousness, the self-transcending / blooming quality of the process-of-life, discovery, beauty, and many other things accessible from my present vista. If I can help it, I’d aim to ensure that those values and capabilities become more deeply understood so that they might carry forth to the next strata of intelligence – be that augmented humans or especially to non-biological intelligences.

Our best understanding of reality and value should bear heavily on decisions about AI, brain-computer interfaces, bioengineering, and the future of intelligence – we must make the best (albeit provisional) choices about our trajectory from where we now stand.

But we need to reckon that these sacred notions of the real and good – they are but tents of the night.

How long they last, and in what form, and how they are supplanted by something different or higher – we cannot know. Some people thought Newton’s notion of gravity would never be questioned, and some people in India still believe society should be staunchly segregated by caste.

When climbing the towering and foreign mountains we’re about to climb here at the dawn of AGI, we must pitch a tent and discern where to go from here.

There are two tenets we might bear in mind when we make normative claims from our present vista, if we wish for our claims to be honest and useful:

1 – Provisionality

Making a normative claim from one epistemic and axiological vista does not make it so from all future vistas.

All ideas about the real and the good are either provisional, or they are dogmatic.

We’re speaking here most about normative claims (the good), but these are often draped on epistemic undergirdings which are themselves clearly up for serious revision as new tools, access to nature, and more capable / divergent minds come into being.

If you believe humans should retain autonomy, say so.

But acknowledge the vista from which you make the claim. What new facts about consciousness, identity, agency, or value might alter your conclusion? How long are you actually claiming this arrangement should persist?

There are very few honest “forevers” available to a creature that knows so little.

2 – Adaptivity

Saying that something deserves to persist is different from explaining how it could persist.

All ideas about the good are either adaptive, or they are entitled.

If human autonomy matters, how can humans remain meaningful participants in a world of increasingly capable intelligence? If biological humanity matters, what technological, political, or ecological arrangements make its continued existence viable?

A normative claim is not a persistence mechanism.

Don’t merely pound the table about what the future owes what you cherish. Explain how what you cherish can remain knitted into becoming.

Law in much of the Western world is inherently provisional. Amendments, new judgements, all can help society and legal systems adapt to new technologies, situations, evolving preferences, etc.

A Call for Adaptive Normativity

We need normative claims. We need to decide what matters, what should persist, and where we should go from here.

But adaptive normativity is not a preference for change. It is a capacity for change.

Individual rights, autonomy, democracy, human persistence – all may be defended fiercely today. But we should defend them because of what they protect and enable from our present vista, not because we imagine that our present moral arrangements are written into the cosmos.

Adaptive normative claims should:

• Name the value beneath the norm. What does this principle actually protect or make possible?
• Name its scope conditions. What facts about minds, identity, consciousness, scarcity, or civilization make it sensible now?
• Name its revision triggers. What could we learn, or what could change, that would make another norm more appropriate?
• Separate the Flame from the Torch. Preserve and expand what matters without demanding that its present form persist forever.

For most of history, change was slow enough for each generation to lull itself into believing that its moral world was the moral world.

We have less excuse now.

Minds beyond humanity are now emerging. Humans themselves are changing. Identity, consciousness, agency, and moral consideration may take forms we cannot presently imagine.

Our norms are tents of the night.

We need them. We should build them carefully and sometimes defend them fiercely. But we should not mistake them for the landscape.

At the edge of rapid transformation, static normativity becomes maladaptive. Our task is not to dictate the moral constitution of eternity from one fleeting human vista. It is to preserve known value, avoid ruin, expand potentia, and keep the Flame alive and blooming into whatever vistas come next.

Then, from there, we choose again.

And this can (must) be seen as an opportunity – even if it be challenging.

The adaptivity and provisionality of all processes – from biological life to what we now call “morality” – is the reason why the worm was able to turn into a fish and was able to turn into you. It is the reason why the tribal clannishness or feudal serfdom of the past was able to turn into the civilization we now enjoy, which makes possible the internet through which you read this very article.

Has this transformation ended?

Are we to pretend it could be eternally frozen?

Of course not.

Ultimately, values and morality are not in service to our present torch (our homo sapien form, our present civilizational structure, etc), they are – as all torches are – in service to the flame – to the greater process-of-life itself.

…

NOTE: Yes this is an Emerson fan page. Always has been.

Header image credit: Space.com