What Is Reality?
Have you ever wondered whether reality is real? I mean, like, really real? This question may not take up much of your time, but great thinkers since ancient times have pondered the nature of reality and our relationship to it. Philosophers have advanced a variety of positions in this long-running debate, and quite a few have concluded that the world we perceive is primarily a construct of our own minds, rather than anything like objective reality. The idea that our senses (or, indeed, those of any conscious agent) don’t disclose reality is the main idea of The Case Against Reality by Donald Hoffman, a cognitive scientist at UC Irvine, who argues that in addition to its ancient pedigree, this counterintuitive view of reality and perception is also supported by the latest science.
I thoroughly enjoyed this book, and I’d recommend it as both a refreshingly new approach to some age-old questions and a primer on the science of perception. The book is filled with fun optical illusions, including quite a few in a full-color insert, and Hoffman uses them well to demonstrate that our minds construct the world we perceive in a probabilistic way. I appreciate the boldness with which Hoffman presents his big ideas, and this book has certainly had an impact on my own thinking about these topics. I like a lot of what Hoffman is doing here, but I think the book has some flaws in its argumentation that undermine its overall case.
The central argument of The Case Against Reality is based on evolutionary game theory, but Hoffman also assesses evidence from his own field, the science of perception, as well as recent discoveries in physics. He presents these multiple lines of evidence as converging on a picture of the world as fundamentally a network of conscious agents interacting with each of us via the interface we call the physical world of space, time, and material objects. This is a radical position for a modern scientist to take, but it’s not unprecedented in the history of philosophy, and I think it’s more plausible than it might initially seem.
Although Hoffman briefly discusses some of the philosophical precursors to his ideas, from Plato’s cave to Kant’s noumena, the book is pretty light on philosophy overall. This isn’t necessarily a bad thing, but I think there are a few points where this book suffers from a lack of conceptual clarity. I think Hoffman overstates his case with claims like “evolution by natural selection does not favor true perceptions—it routinely drives them to extinction” (Hoffman xv). It seems much more reasonable to adopt pragmatic notions of truth and reality based on our experience as conscious agents, rather than making unnecessarily counterintuitive claims about our relationship to the world.
The Case Against ‘Reality’
The first place where things get a little slippery is the title, The Case Against Reality. Strictly speaking, the book advances a realist position with respect to objective reality, rejecting subjectivism and solipsism in favor of an unorthodox metaphysics Hoffman calls “conscious realism” (Hoffman 185). The word “reality” in the title is meant to indicate the familiar physical world of our senses, so the title is accurate insofar as the book does make a case that this world we perceive is not fundamentally real. But again, this isn’t a case against reality, just a case that reality is not what we perceive it to be.
In Hoffman’s view the physical world isn’t exactly unreal or an illusion; it just isn’t what we think it is. It’s not that material objects don’t exist, just that they only exist inside consciousness, as icons allowing interaction with a deeper reality. These icons might not have a form that corresponds to any structure in the reality they represent, but they have content that corresponds meaningfully to some relevant aspect of this reality. Like icons in a desktop interface, they tell us genuine truths about the things they represent and allow us to interact with the underlying system in a sensible way.
Frankly, I think the suggestion that one is not in contact with reality is a mild psychological stressor, and plenty of people have lost their grip on things already. The phenomena of direct experience are undeniably real to the one experiencing them, and the world I experience myself as living in is real enough for me. I may not perceive anything of the agent-independent structure of the world, but for all practical purposes that structure is just an abstraction. Whatever reality really is, its nature is such as to manifest itself to me as the world of my experience.
The title itself may not be worth taking so seriously, but it gets the reader off on the wrong foot regarding the real substance of this book’s claims. I was a bit perturbed by what I perceived as an anti-realist streak in this book: Hoffman makes plenty of assertions like “natural selection does not shape us to perceive the structure of [objective] reality,” and “as senses grow more complex, they have less chance to disclose any truths about objective reality,” and that the contents of our perception “need resemble nothing of objective reality,” even going so far as to say that “no perception is veridical” (61, 69, 86, 87). While I agree with a lot of the points Hoffman makes in this book, I think statements like, “my perceptions can’t show me the truth,” are based on a misleading idea of what truth is supposed to be (176). The fact that my senses give me the world from my perspective rather than the view from nowhere doesn’t mean my perceptions are not veridical.
The idea that the world is not as we naively perceive it has been elaborated in many different ways throughout history, and I think there are good reasons for believing that mind is more fundamental than matter, but I think Hoffman goes too far in asserting that our perceptions disclose nothing of objective reality. In particular, I think he gerrymanders the definitions of terms like “truth” and “objective reality” to exclude facts about agents and their relationship to the environment. This book uses evolutionary game theory to argue that organisms inevitably evolve sensory systems “that hide external reality and encode fitness payoffs” (101). In other words, we are sensitive to the opportunities and threats the world offers us rather than any agent-independent structure the world might have.
In this sense I completely agree with the basic argument, but I disagree with the way fitness payoffs are excluded from the definition of objective reality. For a particular organism, it is objectively true that its environment offers certain fitness payoffs, which constrain the set of experiences and actions available to it at any given moment, and it is no less objectively true that the same environment might offer different payoffs to different organisms or even the same organism at different times. What exactly these fitness payoffs are is not clear, however, and how they relate to the content of experience is another question altogether.
The Hard Problem
The Case Against Reality opens with a discussion of the hard problem of consciousness and the difficulties of searching for brain states that correlate with conscious experiences. The first chapter cites a series of illustrious thinkers, including Leibniz, T. H. Huxley, and William James, expressing their astonishment at the possibility that consciousness could somehow emerge from the behavior of unconscious matter. Hoffman himself had the privilege of discussing this problem with Francis Crick, while Crick was working on his book about the emergence of mind from matter, The Astonishing Hypothesis. The exchange between these two thinkers is particularly entertaining, as Hoffman seems to be denying object permanence, which Crick finds hard to swallow.
Hoffman draws on his expertise in the science of perception to make a case that “seeing is an active, constructive process” that presents a visual world tailored to the mind of a particular observer, leaving us with “no direct knowledge of objects in the world” (Hoffman 40). He strongly disagrees with Crick’s confident assertion that science will someday furnish a satisfactory explanation of consciousness in terms of the physical activity of the brain, and it’s easy to construe The Case Against Reality as Hoffman’s response to The Astonishing Hypothesis and the materialist paradigm represented by Francis Crick.
Hoffman effectively turns the hard problem of consciousness on its head, assuming consciousness to be fundamental and proposing that the physical world exists purely as an interface between conscious agents. This approach to the mind-body problem is radical for a modern scientist but, again, not unprecedented in the history of philosophy. Hoffman mentions a few of the notable predecessors to his view, including the idealist views of George Berkeley and Immanuel Kant, and while he justifies his view primarily on scientific grounds, I also think his inversion of the hard problem is a good move philosophically.
Hoffman’s consciousness-first approach has the philosophical advantage of reversing the explanatory gap between third-person accounts of physical phenomena and first-person accounts of conscious experience. As Hoffman says about the failure of physicalist theories to provide a workable explanation for conscious experience as a physical phenomenon, “I think the failure is principled: you simply cannot cook up consciousness from unconscious ingredients” (Hoffman 183). While we may not be able to imagine any description of physical phenomena that can satisfactorily account for our experience of life as conscious agents, we can easily hypothesize conscious agents as the relevant causal powers behind any physical phenomena. Besides being the norm throughout human history, this tendency to interpret the world in terms of consciousness and agency is seen in the magical thinking of children, and even educated modern people have the ineradicable cognitive habit of anthropomorphizing the world. It seems that by default human beings relate to the world on personal terms, seeing it as made up of agents like ourselves.
This is a natural way to interpret the world because, starting with our experience of ourselves as agents, we can use the complex cognitive machinery of our own self-awareness and goal-driven behavior to make inferences about the behavior other people, animals, plants, and even inanimate objects by projecting onto them a system of beliefs and desires appropriate to explain the behaviors we observe. This may very well be a quirk of our own minds with no deeper implications for the nature of reality, but The Case Against Reality suggests that people outside the frame of secular modernity are right to interpret the world as full of conscious agents. This may sound silly, but I think it’s more sensible than it initially seems.
The Case For Consciousness
In his second book on free will, Freedom Evolves, Daniel Dennett has a little fun with the notion that the world is made up of conscious agents:
[Atoms] are just tiny places where there is happening, not doing. But that doesn’t stop us from simplifying our vision of them by treating them as if they were agents—very simple, single-minded agents. That carbon atom clings tenaciously to those two oxygen atoms, preventing them from wandering off, forming a persistent molecule of carbon dioxide—a modest task for a carbon atom. Other carbon atoms play more exciting roles holding together gigantic mega-atom proteins, so that the proteins can do their thing, whichever thing it is (Dennett 55).
One remarkable thing about The Case Against Reality is that it really does make a case for atoms as “very simple, single-minded agents” or at least a simple interface between human beings and other agents of some sort. Hoffman really does argue for the view Dennett calls “intentional systems all the way down” and characterizes not unsympathetically as belonging to “the childhood of civilization,” although Hoffman seems confident that modern science is on his side (ibid.). Leaving aside for now the question of what an atom really is, it might still be reasonable to use the existence of consciousness as a starting point for understanding the nature of the physical world. After all, no facts about the physical world can be known without first entering into consciousness in one way or another. Whatever the ultimate nature of reality is, it must be the kind of thing that includes conscious experiences such as my experience of writing this right now. Does it include other kinds of conscious experiences?
I can easily imagine that the people I interact with (and, indeed, all human beings) have conscious experiences analogous to my own, including your experience of reading my thoughts right now, but the case is a lot less clear when I consider the rest of the animal kingdom, let alone other kinds of living things like plants and fungi. Animals clearly respond to their environment in ways that suggest some sort of awareness, and even plants are observed to act in their own interests as if driven by internal purposes. Does this mean every living thing is a center of conscious experience and action the way I am? Do the simpler patterns of behavior other organisms exhibit imply simpler forms of consciousness? Hoffman argues that I have no grounds for denying that anything outside myself is conscious. In his view, physical objects are icons in the human interface with the world, which is fundamentally a network of conscious agents, meaning physical objects are not conscious themselves but are icons that allow us certain kinds of interactions with the other conscious agents that make up the world around us.
The face I see in a mirror, being an icon, is not itself conscious. But behind that icon flourishes, I know firsthand, a living world of conscious experiences. Likewise, the stone I see in a riverbed, being an icon, is not conscious nor inhabited by consciousness. It is a pointer to a living world of conscious experiences no less vibrant than my own—just far more obscured by the limitations of my icon (Hoffman 188).
The point Hoffman is making here boils down to the problem of other minds. How do I know the people around me are conscious in the way I am instead of just moist robots that exhibit human-like behavior with no corresponding inner experience? This has an interesting sort of resonance with an argument made by C. S. Lewis in an early chapter of Mere Christianity. The original argument is about the existence of a universal natural law, but it applies just as well to consciousness:
There is only one case in which we can know whether there is anything more, namely our own case. And in that one case we find that there is. . . . Surely this ought to arouse our suspicions? In the only case where you can expect to get an answer, the answer turns out to be Yes; and in the other cases, where you do not get an answer, you see why you do not. Suppose someone asked me, when I see a man in a blue uniform going down the street leaving little paper packets at each house, why I suppose that they contain letters? I should reply, ‘Because whenever he leaves a similar little packet for me I find it does contain a letter.’ And if he then objected, ‘But you’ve never seen all these letters which you think the other people are getting,’ I should say, ‘Of course not, and I shouldn’t expect to because they’re not addressed to me. I’m explaining the packets I’m not allowed to open by the ones I am allowed to open.’ It is the same with this question.
As Descartes demonstrated with his famous cogito, the existence of my own mind is a given, and the existence of the “objective” physical world must be inferred from certain aspects of my experience. One of the commendable things about Hoffman’s backdoor approach to the mind-body problem is that he takes seriously the greater certainty we have in the existence of consciousness than in the existence of the physical world. The Cartesian method of doubt may not be sufficient to prove the existence of a whole self, but I cannot dispute the reality of my ongoing experience. I may be hallucinating my world, but my experience of the hallucination is no less genuine; regardless of the “objective” truth, I currently experience a particular, definite set of sensations – external, internal, and affective – bundled together into the phenomenological unity I call myself. One further character of this (or any) experience is that I can respond to it with any of a number of actions and even anticipate some salient consequences of these potential actions. I feel like this is all obvious to me by direct introspection, more clearly apprehensible than the words in front of me, and if you don’t share my intuitions, well, I’m sorry you had to find out like this but you might be a zombie.
Donald Hoffman, however, does share my intuitions on this point, and it is from this sort of bare-bones phenomenology that he constructs his “conscious agent model.” This mathematical formalization of basic phenomenology is one of the most intriguing ideas presented in The Case Against Reality, and it is treated much more technically in some of Hoffman’s academic papers. In summary, Hoffman’s notion of a “conscious agent” is defined by “a repertoire of experiences and actions” as well as a feedback loop by which the agent's actions affect its experiences (Hoffman, p. 125). To complete this loop, the model includes a basic representation of the world as being in one of multiple states relevant to the agent and capable of changing from one state to another based on the agent’s actions. Hoffman characterizes this “perceive-decide-act” (PDA) loop in very simple terms:
Based on its current experience, the agent decides whether, and how, to change its current choice of action. … The action of the agent changes the state of the world. The world, in response, changes the experience of the agent… (ibid.).
The life of a conscious agent thus proceeds from perception to decision to action to the consequences of action on perception via some change in the state of the world. It assumes the existence of both conscious experiences and actions, but I find both of these categories to be phenomenologically given in a way that renders them absolutely undeniable. We act, and we experience, and we come to understand stable aspects of the world via the regular patterns of perception and action afforded to us. We need make no assumptions about the nature of the external world beyond the obvious one that our interactions with it engender particular patterns of experience that allow us to posit stable external phenomena such as physical objects.
Physical phenomena are, at some level, not observed directly but hypothesized to explain the regularities we do observe in the way actions modulate changes in experience. Evidence for anything in the world must affect experience either directly through the senses or indirectly through the mediation of some instrument, and this is a fundamental feature of how we can know anything outside ourselves. As Sharon Hewitt Rawlette writes in Psychology Today:
If we look carefully, we can see that all of the physical properties that science has so carefully measured and cataloged ultimately derive their meaning from the effects they produce on a conscious observer: the person who’s holding the yardstick or looking at the fMRI or gazing at the interference pattern produced by the double-slit experiment. Even the properties of the basic particles of physics derive their meaning from the ways that these particles ultimately affect the conscious observations we make of them. Scientific experiments cannot tell us what a photon or an electron is in itself. Science can only tell us that, when a photon or electron is present, our conscious experience of the equipment detecting these particles will be affected in a certain way.
We cannot escape the perspectival nature of our knowledge. The only phenomena knowable to us are those that in some way affect our phenomenology. Conscious experience is the necessary starting point for any epistemology, and many different schools of thought have taken consciousness as the foundation of their ontology as well. Of course, by taking conscious experience as an ontological primitive, Hoffman absolutely punts on the question of what exactly the nature of this experience is, but I’m willing to let that slide because, as far as I can tell, the ontology of physics has no firmer foundation. What exactly is_matter_, anyway? How can reality be just a bunch of mathematical equations, deep down?
The Truth of Fitness
I have mixed feelings about the book’s first big idea, the Fitness-Beats-Truth (FBT) theorem, which states that organisms that perceive only the fitness payoffs afforded to them by the environment inevitably outcompete organisms whose perceptions match the structure of reality. I agree that perceiving fitness payoffs is a better strategy than perceiving more “objective” aspects of reality, but I’d frame the situation a little differently. Fitness payoffs are not subjective: it is an objective fact whether any particular organism lives or dies, eats or goes hungry, successfully reproduces or ends its line without offspring. An organism that sees the fitness payoffs on offer isn’t failing to see reality; it’s seeing precisely those aspects of reality that are most relevant to that organism.
The book introduces the FBT theorem with a discussion of evolutionary game theory and a simple experiment involving simulated creatures foraging for food. The simulated environment is home to two types of organisms, which Hoffman calls Fitness and Truth. Each organism is capable of two perceptual states (gray and black in the image below), but the ways in which these perceptual states correspond to the state of the world differ between the two species.
The Truth creatures can see whether there is more or less food available, while the Fitness creatures can see whether there is enough food for them or not, and naturally Fitness beats Truth in the evolutionary simulation. It makes perfect sense that the Fitness creatures would be more fit, since their perceptual system is more complex. While both are capable of the same number of perceptual states, the perceptions of Fitness relate to the world in a more complex way than those of Truth. This can be seen in the image, as the perceptions of Truth have one threshold where gray transitions to black, while those of Fitness transition at two thresholds, giving its perception function two comparisons to make instead of one. Fitness is more fit because its perceptual system incorporates more information in order to present it with a view of the world based on relevance, as opposed to the remote objectivity of Truth. In an important sense, Truth is restricted to the view from nowhere, while the perspective of Fitness includes facts about its own relationship to the world. This doesn’t mean that Fitness is not in contact with reality; on the contrary, its contact with reality is more pragmatically effective because it’s informed by the state of the organism. This has nothing to do with the subjectivity of conscious experience, either, since it’s an objective fact whether one of these simulated organisms has enough food, or too much, or too little. It’s just that Fitness perceives too much and too little as the same irrelevant gray, allowing it to zero in on the black ‘Goldilocks zone’ where the amount of food is just right. Its perceptions work because they inform decisions about what to approach and avoid in ways that aid survival. Truth has perceptions that map in a linear way to the quantity of food available, but why should quantity be more real than fitness payoffs? If anything, the fitness payoffs are more real because they incorporate more information and thus present a higher-fidelity rendering of reality to the organism.
When Hoffman claims that “although fitness payoffs depend on the true state of the world, they also depend on the organism, its state, its action, and its competition,” he is defining “the world” in unnecessarily narrow terms (Hoffman 55). Is the organism not part of objective reality? Are facts about the organism’s state and its relationship to its environment necessarily subjective? I’d contend that there are plenty of objective truths about how I fit into my environment, and these truths are the most relevant to the content of my perception.
For instance, I am currently sitting in a chair. This is an aspect of my experience that I perceive directly through proprioception and the pressure of the chair’s seat and back supporting me, but it is also an objective fact verifiable by any observer (watching, say, though my webcam). Whether the chair is comfortable is something else I am perceiving directly, and an observer might assume it’s reasonably comfortable (it is) based on the fact that I’ve been sitting here for a while, but this can only ever be an inference from my observable behavior. A hypothetical observer could creep out of her surveillance van and sit in my chair herself (while I’m at work, say) to get a direct perception of the chair’s comfort level, but all she learns in doing so is how comfortable the chair is for her. Suppose it is very uncomfortable. The “comfortableness” is a property not of the chair but of the chair’s relationship to each person. The experience of comfort or discomfort may be subjective, but the fact that the chair is more comfortable to me than to my hypothetical sneaky observer is an objective fact not directly present in either person’s experience of the chair. For any given person, the chair affords a certain level of comfort, and this is an objective fact about how that person stands (so to speak) in relation to the chair.
A Matter of Taste
Returning to the book, let’s talk about feces, which Hoffman brings up a couple times to demonstrate the agent-relative nature of perception. The fact that feces “offers big payoffs for hungry flies, but not for hungry humans” is objectively true (Hoffman 55). It is scientifically verifiable that feces affords nourishment to certain kinds of flies while offering human beings nothing but the risk of illness. When Hoffman says, “The payoffs of feces, and thus their tastes, differ crucially between us and coprophages,” I assume he is right, although I admit I have a dearth of personal experience in this area (Hoffman 88). Like a chair that can comfortably seat people of some body types and not others, feces is appetizing to some organisms and disgusting to others, and these differences can be analyzed in a systematic way based on our scientific knowledge. We can understand objectively how and why coprophagia benefits those species that engage in it, and conversely, why such activity is of little benefit to human beings and other coprophobes.
One of the true strokes of brilliance in this book is the definition of an illusion as “a perception that fails to guide adaptive behavior” (87). Given such a pragmatic definition of illusion, it’s disappointing that this book does not include a similarly pragmatic definition of truth. In fact, just a few sentences before giving his definition of illusion, Hoffman asserts that in his theory of perception, “no perception is veridical” (87). Maybe there’s a crucial asymmetry that escapes me, but it seems reasonable to define veridical perception as the opposite of illusion, simply a perception that successfully guides adaptive behavior. In practically every respect, Hoffman does away with correspondence theories of truth, but his assertions that our perceptions hide reality, we see nothing of the truth, etc. rely on just such a notion of truth-as-correspondence. This book is at its best when it’s making pragmatic arguments grounded in phenomenology, and I think its overall case would be strengthened by a pragmatic definition of truth.
The words ‘true’ and ‘real’ are notoriously hard to define, but George Lakoff and Mark Johnson do a good job of defining these words pragmatically in their magisterial 1999 work on cognitive linguistics, Philosophy in the Flesh. Their definition of truth is based on understanding.
A person takes a sentence as “true” of a situation if what he or she understands the sentence as expressing accords with what he or she understands the situation to be (Lakoff & Johnson 510).
Likewise, their definition of reality is remarkably practical and down-to-earth:
What we mean by “real” is what we need to posit conceptually in order to be realistic, that is, in order to function successfully to survive, to achieve ends, and to arrive at workable understandings of the situations we are in (109).
Both of these definitions seem quite resonant with Hoffman’s view of the world, but instead of embracing pragmatism in a way that allows us to feel ourselves embodied and embedded in the world, Hoffman takes a rhetorical tack aimed at cultivating in the reader a sense of alienation from reality, which I find a bit distasteful.
Hoffman presents the taste of feces as “a baffling problem for the standard account” in which “our perceptions are normally veridical,” but I don’t think it’s problematic at all that the same substance might taste different to organisms with very different digestive systems (88-89). The disgust you or I might feel at the taste of feces is a veridical perception that this is a substance teeming with bacteria which would be harmful for us to ingest. In general, different organisms should be expected to respond differently to the same stimuli, based on the different costs/benefits afforded to the organism in question.
It should be clear now what sort of confusion underlies the following series of rhetorical questions:
Whose perceptions are nonveridical—ours or those of coprophages? Are we right that feces truly have a loathsome taste? If so, do pigs, rabbits, and billions of flies suffer a taste illusion? Or are they right that feces truly are delicious? If so, is our disgusting experience a taste illusion? (88)
Yes, feces truly do have a loathsome taste to human beings. And yes, feces truly are delicious to those pigs and rabbits. There is no contradiction here, and there’s no need to invoke any of the esoterica of subjective experience. All that’s needed is the recognition that terms like “disgusting” and “delicious” are 2-place words, describing how an organism stands in relation to a potential food source rather than intrinsic properties of the food source itself. The sensation of taste is an interface between an animal and a potential food source.
On Interfaces
The most counterintuitive idea in The Case Against Reality is that the world each of us perceives is not objective reality but an interface based on concerns relevant to our evolutionary history. The physical properties of the world we perceive are aspects of how our minds interact with reality rather than being aspects of reality in itself, and they no more resemble this underlying reality than the letters C-A-T resemble an actual cat. I tentatively agree with this framing of the issue, but I think Hoffman doesn’t give interfaces enough credit. An interface may not be structured like the system it represents, but it carries semantic information about that system. An interface discloses to a user certain relevant facts about the interactions it makes possible, and the information conveyed through an interface can be assessed as true or false at the semantic level. The word “cat” may bear no resemblance to our furry friends, but the word alone is sufficient to evoke that association in anyone who knows English. Words typically have no intrinsic relationship to the things they name, but this arbitrary relationship of meaning is an objective fact about the semantics of English or any other language.
The example of writing an email comes up a few times in this book, as a demonstration of the way interfaces offer a user-friendly experience by hiding irrelevant information:
Suppose you’re crafting an email, and the icon for the file is blue, rectangular, and in the center of the desktop. Does this mean that the file itself is blue, rectangular, and in the center of your computer? Of course not. … Indeed, the file has no color or shape; and the location of its bits in the computer is irrelevant to the placement of its icon on the desktop (76).
This example should take a bit of the counterintuitive edge off the idea that our perceptions might be different from reality—as different, perhaps, as a GUI is from a circuit board. The location of the file on the desktop does seem totally arbitrary, but the blue color and rectangular shape of the file probably carry important semantic information about the file (namely, that it is an email). Typically, the file icons of a desktop interface are of different kinds depending on the type of file they represent, whether the files contain text, images, executable programs, or any of a variety of content types. The icons for a plaintext (.txt) file and a Word document (.doc) probably look different, but they ought to be more similar to each other than either one is to, say, a PDF icon, because they are more similar in terms of the nature and structure of their content as well as how they allow the user to interact with them.
On my computer, the .txt file icon looks like a sheet of paper with a dog-eared corner, while the .doc icon looks like the same dog-eared sheet of paper with a blue ‘W’ on it as well. The appearance of these icons may be arbitrary, but once I’m familiar with the conventions of my operating system, I can ‘read’ certain kinds of semantic information about the files in my computer directly from their associated icons. The dog-eared-paper icon indicates that the file represented is a text file, which gives me important information about how to interact with it. Of course this does not mean that the underlying reality of these text files is a tiny sheet of paper inside the computer somewhere, and the dog-eared corner appears to be a pure stylistic choice with no semantic content, but thanks to my interface I can tell at a glance whether Microsoft Word is the right application for editing a particular file. The image of paper is a kind of metaphor, indicating that text files can be read and written on like physical paper, but even a hypothetical user who’d never seen a sheet of paper would quickly learn that this rectangular white shape means that the file it represents is a text file. It tells the user something about the underlying reality of the computer file, in terms of the interactions it affords to the user.
Returning to Hoffman’s example of the email, he is right to say, “The blue icon does not deliberately misrepresent the true nature of the file,” but he goes too far when he continues, “Representing that nature is not its aim” (ibid.). Presumably, the fact that the icon is blue and rectangular rather than, say, green and triangular or red and octagonal, tells the user something true about the kind of file it represents and thus what the user can expect when interacting with it. The color blue here is certainly more arbitrary than the dog-eared-paper shape is for a text file icon, but it is meaningful given that files of a certain type are reliably represented by icons of a certain color. The blueness of the icon doesn’t correspond to anything blue in the computer, but it represents something about the file in the idiom of desktop icons; it has a meaning in the “chromatic semantics” of the operating system, and this meaning might be accurate or misleading to a user such that we could call it true or false.
Message Over Medium
Writing an email is an instructive example because it raises questions about the kinds of things that exist at different levels of abstraction. If I’m writing an email in my browser window, then the content of the email exists in my mind as a message I want to send, on the computer screen as a sequence of characters I can read, and in the computer’s memory as a sequence of bytes corresponding to the characters on the screen. If I save my email as a draft, the bits representing my message get sent over the internet to be stored on an email server. Assume, for the sake of argument, that my computer and the email server have different ways of encoding and compressing this data, so that the bit sequences representing my work-in-progress are very different on the two machines. The difference in the bit sequences is irrelevant to me, as long as they encode the same message. The message is an abstraction that can take many different concrete forms. It can be physically instantiated as a pattern of voltages in the transistors of a computer chip, a pattern of neural activity in my brain, a pattern of pixels in my computer screen, or a pattern of ink on paper. These are all meaningfully the same pattern, because each one encodes the same content in its particular context.
The purpose of an interface, such as a desktop icon or a text editor, is to hide the details of its implementation behind an abstraction barrier so that human beings can work at a level of abstraction that’s comfortable for them. Hoffman is right to recognize that email is only possible because of an abstraction barrier that conceals “tiresome details on transistors, voltages, magnetic fields, logic gates, binary codes, and gigabytes of software,” (76). These implementation details form an important part of the context in which writing an email is a sensible thing to do, but they should be irrelevant to the task of writing. As Hoffman says, “If you had to inspect that complexity, and forge your email out of bits and bytes, you might opt instead for snail mail” (ibid.). A text editor allows us to work with letters, words, and sentences, just as we do when we write with pen and paper; although the motions of our hands in these two media are radically different, the cognitive work of putting one word after another is identical. If I want to email a friend
I experience, as I’m typing this, the way my mind proceeds from one word to another at the semantic level, and the detailed choreography of my fingers across the keyboard remains hidden behind an abstraction barrier, relatively opaque to introspection unless I bring my attention down from the semantic level to analyze the physiology of my typing motions, which slows down my typing considerably. If I lost access to this semantic information and had to compose an email by consciously orchestrating patterns of change in the tension of my hand muscles, I would be quite lost indeed. I might say that whether typing an email or writing a letter by hand, I benefit from an interface with my own body that allows me to “cash out” semantic information into stereotyped movements that record that information in a particular medium. A good interface uses abstraction barriers like bridges over the Turing tarpit, to keep users safely at the semantic level of the message instead of bogged down in the representation details of the medium.
The semantic information in a message tells us how to inscribe it in any particular medium, and we have no trouble recognizing that the same message will inevitably take different forms in different media. The differences between an email and a handwritten letter are many, and if we looked only at their physical forms we might not be able to recognize them as the same kind of thing at all, but they can convey the same message with equally high fidelity by being identical at the semantic level. As long as all the same words and sentences are there, it matters little whether they’re encoded as voltages in transistors or squiggles of ink on paper. The semantic information has a reality that transcends any particular implementation; its meaning is recognizable on the human side of the abstraction barrier in a way that persists across drastically different modes of representation. This is not necessarily to say that the email as a semantic entity is more real than the pattern of voltages that instantiates it in my computer, but it is at least as real as its physical representation.
The Moon and the Spoon
Hoffman offers a further illuminating example:
To ask whether my perception of the moon is veridical—whether I see the true color, shape, and position of a moon that exists even when no one looks—is like asking whether the paintbrush icon in my graphics app reveals the true color, shape, and position of a paintbrush inside my computer (77).
Yes, that’s right, he believes the moon isn’t there when nobody’s looking at it. He explicitly agrees with George Berkeley that “to be is to be perceived,” asserting that not only the moon but all physical objects are “data structures we create and destroy” as we perceive them (79). The icons on my desktop don’t exist when my computer is turned off, or asleep, or even when the desktop is hidden by an open, such as right now as I’m typing this in my browser. I can minimize my browser window and see each icon in its proper place, but they were not really “there” all along, hidden “behind” the browser window. They are constructed anew each time the desktop reappears, and they appear the same each time because they’re constructed the same way from the files which do persist as patterns in the hard drive. So the paintbrush icon might not represent any real paintbrush, but it represents a program that really exists (physically, as a distribution of voltages) inside the computer and allows its user to create art. Color, shape, and position are not the right way to describe this program, but the fact that the icon looks like a paintbrush, again, tells the user something true about the program. The function of this program as a tool for creating 2-D visual art is conveyed metaphorically by the image of the paintbrush. This correspondence at the semantic level is meaningful and non-arbitrary: the program has the characteristics it does because it was designed as an artistic tool. It may be a digital artifact, but at a certain level of abstraction a graphics program has more in common with a physical paintbrush than with, say, a desktop chess program. By the same token, a game of chess played on a computer is in an important sense identical to a game of chess played with physical pieces, if the game is understood as a sequence of abstract moves that can be played out in different kinds of physical media. I might play chess online with someone whose chess program looks very different from mine, but the differences in our chess programs are irrelevant insofar as they allow us to participate together in the game as an abstract process.
The moon is there every time I look, presumably because it represents some stable aspect of reality that persists even when unobserved and reliably manifests in my experience as the moon. Whatever this persistent aspect of reality is, then, must be the “moon that exists even when no one looks.” It might be true that all of the physical properties of the moon are as arbitrary as the color of a desktop icon, but the stability of those properties implies a stability in the underlying reality and how we can interact with it. So what is the moon, really? Well, as a physical object, it’s an icon in the human interface, but as a thing-in-itself, it’s simply whatever is “out there” that’s responsible for our experience of the moon: a “moon file” on the hard drive of reality. The moon we perceive is constructed from the moon-in-itself according to the perceptual program of the human operating system.
You and I might construct very different icons of the moon when we look at it, but we can agree on basic things like where it is in the sky, what phase it’s in, and how much light it gives on a particular night. In light of what we can agree on, deeper questions about what the moon “really is” outside of our experience seem irrelevant. Take, for example, the phases of the moon: is the first-quarter moon the same moon as the third-quarter moon? Hoffman would say no, the moon physically exists only when perceived, and each night our sensory systems construct a slightly different moon icon to reflect a slightly different state of the underlying moon-in-itself. We can use the moon to keep time because our experience of the moon changes in a periodic way so that the span of time from new moon to full and back again is always the same. It doesn’t even matter if the moon is “the same” from one full moon to the next—as long as the pattern continues in its regularity, any difference in the underlying reality responsible for this pattern is hidden behind the abstraction barrier of our experience.
As an icon, the moon’s purpose is more informative than interactive—giving light and tracking time with its phases—but human beings have venerated moon gods for a long time in an attempt to establish some two-way relationship with this mysterious heavenly light. Recently, human beings have discovered that the moon is a place we can visit (!), and this seems to obviate the older interpretation of the moon as some kind of conscious agent, but who knows, maybe these are just two different ways of interacting with the same underlying reality. To scientifically-minded modern people, the moon appears as a giant rock in the sky, which can be analyzed using the tools of science, but which doesn’t afford any genuine personal relationship to human beings. More mythopoetically inclined cultures might have developed interfaces that allowed them to experience the moon very differently. They would of course have seen the giant rock in the sky, but the moon may have regularly appeared to them in much more significant ways, say, as an omen for good or ill, or a giver of wisdom in dreams.
Coming back down to earth, let’s look at something that affords a bit more exploration by the average reader:
A spoon is an icon of an interface, not a truth that persists when no one observes. … I open my eyes and construct a spoon; that icon now exists and I can use it to wrangle payoffs. I close my eyes. My spoon, for the moment, ceases to exist because I cease to construct it. Something continues to exist when I look away, but whatever it is, it’s not a spoon, and not any object in spacetime (79).
Here I think the book suffers from imprecise terminology. As with the moon, the spoon-as-a-physical-object exists in the human interface as a representation of the spoon-in-itself, and, crucially, it informs us as to how we can use it. It’s messy business, but I can use a spoon with my eyes closed. Hoffman would say that the visual icon of the spoon disappears when I close my eyes, but my grasping the spoon creates a tactile icon that allows me to manipulate it sight unseen. Are my visual and tactile spoon icons in any sense the same spoon? Hoffman seems to say not exactly. The different types of sensory icons are different representations of the mysterious “something” that accounts for my reliable experience of a spoon. I’m not sure why he feels the need to say, “whatever it is, it’s not a spoon,” since I’d rather say, whatever it is, is what the spoon really is. The physical properties of the spoon—size, shape, texture—may be aspects of an icon that don’t correspond meaningfully to any underlying reality, but it’s undeniable that I am in the presence of “something” which functions for me like a spoon. If that’s not a spoon, I don’t know what is. You might say I can count on this aspect of reality to manifest “spoonishly” in my interface, or I can use it “spoonwise” (e.g. to eat cereal, scoop ice cream, or stir coffee). It seems more reasonable to say that affording these kinds of interactions with human beings is simply what it means for some part of the objective world to be a spoon. What a spoon “really is” when nobody is looking is another question, but unlike the moon, the spoon is a human invention, so it’s probably something much less interesting.
The Reality of the Virtual
Whatever the ultimate nature of the world is, we experience it from a human perspective, and the concepts we use are tailored to the ontology of the human interface. The Case Against Reality contains a few discussions of virtual reality, with the suggestion that the physical world we perceive has a similar ontological status to the virtual world of a video game. If you and I are playing Minecraft together, we might each see the Minecraft world’s square moon on our own screen, but we understand that we are not perceiving the same physical object. Nevertheless, our two images of the moon will have certain properties in common, such as their phase and their path across the night sky, which presumably are represented somewhere in the Minecraft server hosting our virtual world. We are each accessing the same server for information about the same world, and although the relevant data might take different forms in the circuits of our computers, our screens end up with images of this world that agree with each other.
Now suppose I throw a pickaxe at you (in Minecraft). I see the corresponding pickaxe icon disappear from its slot in my inventory, and I see the pickaxe leave the hand of my avatar and end up floating in the space between our avatars. You don’t see the change in my inventory, but you do see the pickaxe leave my avatar’s hand and come to rest in front of your avatar where you can easily walk forward to pick it up, upon which it appears both in your hand and in your inventory. Of course no physical pickaxe changes hands in this interaction—what I’ve described as “the pickaxe” is really just an icon, but unlike the icon in your inventory, this icon appears to both of us as part of the game world. I have “given” you the pickaxe by taking an action that renders the functionality of this pickaxe no longer accessible to me, while you take an action that equips you with this functionality, and this change in functionality is represented by the “public” pickaxe icon on both our screens leaving the hand of my avatar and ending up in yours. My action has an effect on how each of us experiences the game world, such that the affordances of the pickaxe are “relocated” from you to me.
Notice here that the pickaxe as a pixelated image in your avatar’s hand isn’t even responsible for removing blocks of stone from the virtual landscape when you click on them; rather, it represents to you, the player, that you can now more easily remove those blocks of stone by clicking on them. The movement of the pickaxe icon when you mine a block doesn’t cause the block to disappear, but it does signal that you are taking this particular action which will result in the disappearance of the block. From within the Minecraft world, it’s sensible to act as if the pickaxe is what does the job of mining, but the pickaxe is only a symbol for a set of affordances: it can mine stone quickly and non-destructively or tear up minecart rails in an instant; it can take certain kinds of enchantments that enhance its capabilities; and it even makes a decent weapon. This is what it means to be a pickaxe (in Minecraft).
Hoffman seems to say that the same sort of thing is happening when I pass you a spoon across the table. The spoon in my hand is an icon in my interface, representing that the affordances of the spoon are available to me. You have your own icon for the same spoon, and the sight of it in my hand tells you that I have access to the spoon’s affordances. I take an action I interpret as reaching across the table with a spoon in my hand, but since the physical world is just an interface, this is just the movement of an avatar and might be as epiphenomenal as the swinging of a pickaxe in Minecraft. This action, however, is followed by the experience of the spoon leaving my hand, at which point I see it in yours and understand that the affordances the spoon once made available to me are now available to you. In short, I have passed you the spoon. This might be a painfully abstract way to describe the situation, but it allows us to see that the spoon’s status as a physical object is less relevant for our purposes than the kinds of experiences and actions the spoon makes available to us as human beings.The spoon is defined by its affordances, and our perception of it as a unified physical object is a kind of shorthand for the particular way these affordances cluster together.
Without reference to the physical, we can define an object as a cluster of affordances. The essential nature of a thing consists in the patterns of perception and action it makes available to a conscious agent. This means that different agents might carve the world up into very different ontological categories, but I have no philosophical problem with that. The world-in-itself is an absolute unit; it’s only conscious agents who need to discriminate between separate “things” in the world for the sake of perception and action, and they can make these distinctions quite differently because they do so on pragmatic grounds relative to their various capabilities and objectives.
To wrap this up, I really do think there’s common ground between the view Hoffman presents in The Case Against Reality and the mainstream view among scientists and philosophers that we perceive reality as it is. What we perceive are affordances, and those affordances are really there. They don’t exist in the agent-independent structure of the world, but they are objectively real aspects of how agents relate to the world around them. The physical world that we experience doesn’t have to be structured like the objective world-in-itself in order to give us semantic information that is really true about that more fundamental reality. Despite my disagreements with some of his rhetorical choices in this book, I think Donald Hoffman has made a valuable contribution to the discourse around perception, reality, and the nature of consciousness. I’ve barely touched on half of the interesting things in this book, so again I highly recommend that you read it for yourself.