Intelligence and Spirit by Reza Negarestani

This book is about intelligence. Intelligence as in that which relates to the intelligible.This book is an attempt to answer hard questions about intelligence, and by extension also consciousness, by drawing from several areas ranging from theoretical computer science over to 19th century German philosophy. Ahead of us lies a tempest of confusing thought experiments, eclectic variety of schools of thought and a discussion of the question what role philosophy plays or should play in the 21st century.

i. Human, All Too Human; Or why emotions are not exotic 7-spheres

One of the central themes in Intelligence and Spirit is the notion of human bias. When thinking about intelligence we easily fall prone to the specific notion of human intelligence. Negarestani wants to first remind us of this issue and as a consequence develop a point of view that decouples the “human” part from intelligence. This is specifically important as AGI becomes a real possibility rather than an abstract absurdity. And rather than raising the common questions about AGI (how can it emerge, what can it look like,...),we can also see this as an opportunity to learn something about ourselves. That is, by looking at other forms of intelligence, we can also observe human intelligence “from the outside”. In his words: “Our main objective is not to investigate the possibility of constructing an artificial general intelligence or to review the popular narratives of posthuman superintelligence, but to think about AGI and, even more generally, computers, as an outside view of ourselves.”[36]

What are examples of such human bias? Going back to Kant, our sensory input is something specific to us and with it the thoughts that it constitutes. Our ability to observe, and more generally to experience, enables and, simultaneously, limits our ability to think. How we conduct pattern recognition, create concepts and structure them, is bound by this. To wit, the political compass is a two-dimensional plane, we are inclined to think in low numbers, and physical objects, which are dominant in our world, determine our world view. Why do we not think of emotions as a 7-sphere? Or even an exotic 7-sphere? Why do we not naturally come up with a classification of 1000 species rather than first distinguishing between “fish” and “vertebrate”? And why does water play a more important role in our mythologies than liquid nitrogen? Well, because we never needed to think in higher dimensions or memorize hundreds or thousands of data in a single list. And we did not even know what liquid nitrogen is, let alone that it existed, until rather recently. This is by far not an exhaustive list of examples and the takeaway here is: Whatever I do, I do it as a human. A different form of intelligence does not need to abide by this.

Pictured: My interior life, a 7-sphere, it is very complex and you wouldn’t get it (Source).

Human memory is messy. We can suppress unpleasant experiences, highlight other parts of our experience, reshuffle the order of events etc. to generate certain narratives about ourselves and the world as we perceive it. A computer on the other hand has a pretty reliable memory, as in a hard disk. The implications of such a rigid memory in both the positive (and negative) way seem severe. This goes back to the introduction: When asking about intelligence, we also need to ask about the intelligible. Different modes of intelligence will have different ways to manifest themselves and we need different scales to measure it.

ii. A thought experiment

The main body of the book consists of what Negarestani calls a thought experiment. Concisely put, we are building an AGI from scratch. Now this is not a book about transformers or reinforcement learning, and the word “neural network” has but a few honorary mentions in the book. So this is not an attempt at a manual, but an attempt at dissecting the different layers of “intelligence”. At this point you might rightfully ask for a definition of what intelligence is and alas Negarestani does not provide us with a clean-cut answer. Rather:”...this book is a rudimentary attempt to undertake the urgent task of presenting a philosophy of intelligence in which the questions of what intelligence is, what it can become, and what it does can be formulated.”[37] That being said, we see recurring themes, such as language or self-reflection, but presented in a rather enigmatic way. Anyway, the “embodied automaton” in the thought experiment is defined via the following features:

"It has been programmed to instantiate a number of diffuse and recurring goals centred on the maintenance and preservation of the system qua agent.”[38]
“It has been wired to engage in activities that increase the probability of goal-fulfilment.”[39]
“...the agent is equipped with different specialized sensors and different modules for integrating sensory data both within a specific sensory modality (multiple data associated with one sensor) and across different modalities (data associated with different sensors).”[40]
“The automaton is furnished with a sufficiently complex and functionally flexible memory capable of encoding, retrieval, consolidation, discarding, and transfer of sensory impressions.”[41]

So it is an agent, who optimizes for an unspecified goal(s), is multimodal and possesses a memory. By today’s state-of-the-art this “thought experiment” is becoming increasingly less “thought” and increasingly more “experiment”. But note that this book was published 2018, i.e. before the current hype of LLMs.

Based on this thought experiment, we now ask questions such as “Can this automaton see?” or “Will it be (self-)conscious?”. And in the process of answering this question, we are forced to state more precisely what we mean by that. In general the questions that Negarestani covers are not relevant in terms of AI capabilities, they are more of intellectually curious nature. As I said earlier, if anything we will learn more about ourselves than AGI.

We can take this one step further: As the thought experiment goes further, it becomes evident that it is rather a description of an infant growing up and learning to use language than a superintelligence scenario. In fact, he denies the possibility of superintelligence as is common to us. We will get back to this later.

This thought experiment is the leitmotif for most of the book. But a big portion of the book is not about Kanzi, which will be the name of the above described agent; rather it is about topics revolving around it and which then gets applied to it. I will refrain from giving an exposition of this lengthy thought experiment and I will later justify why. For now, I want to raise your attention as to how Intelligence and Spirit approaches this thought experiment.

iii. The (Un)reasonable Effectiveness of Formalism in Intelligence

In 1960 the physicist Eugene Winger published the article “The Unreasonable Effectiveness of Mathematics in the Natural Sciences” arguing that mathematical structures are an unexpectedly strong tool for our understanding of physical phenomena. Not only do they model current observations very well, but they often also indicate more general patterns which also turn out to be correct.

That being said, it is not clear if this principle should hold in other areas. In Intelligence and Spirit we see a frequent employment of mathematical concepts, notably category theory. This is not done completely out of nowhere, see for example applications of category theory in cognitive science or neuroscience. But, at least in Negarestani’s case, it is not clear to me what the upshot is. There are paragraphs where I think that he falls for category theory-ism, scientism’s closely related sibling, and he ends up presenting ideas in an unnecessarily convoluted way. In fact, he himself makes a remark about this: ”While the generality of category theory makes it a powerful tool for the study of mathematical structures, it is well-suited neither for the study of all types of mathematical structures nor for the modeling of all forms of physical systems and phenomena.”[42] This makes me only more curious why he sometimes refers to and uses a mathematical/formal language, which falls short of providing any insight. And in fact me being more familiar with the mathematical side of things just increased my perplexity.

He does at least try to provide a refreshing Ansatz to solve hard questions. In introducing language as constituent of intelligence, rather than a means of communication, he portrays a theory of interaction and game (as in agent vs environment or agent vs agent). And he goes on to show how that can give a response to the Chinese room experiment:”Yet to claim, as John Searle does, that syntax by itself is not sufficient for semantics is a recipe for the inflation of what meaning is, along with a myopic interpretation of what syntax is and what syntactic expressions do. This is of course a claim encapsulated in the Chinese room thought experiment, an argument that simultaneously presupposes a potentially mystifying account of meaning, a peculiarly anaemic interpretation of syntax, and an outmoded understanding of the relationship between syntax and semantics.[...] While syntax by itself does not yield semantics, it does so when coupled with interaction.”[43] But again, the recurring issue persists of using formalism as a tool that seems stronger than what it is.

iv. Human All Too Human II; A critique of a critique of human bias

Now that the topic of AI safety/alignment is getting exponentially more and more attention, I see recurrent arguments to dismiss it. One such example is:“…[T]here are many types of constraints that need to be in place for anything like cognition to be realized; and […] the myth of a superintelligence or an unbounded posthuman intelligence is precisely the product of biases ingrained in the flat or unconstrained picture of function. In other words, such views inexorably forgo the task of explaining what it means to call something intelligence, and describing the exact structural constraints by virtue of which something can be identified as exhibiting intelligent behaviours. In this sense, naturalistic accounts of superintelligence fall into a contradiction: committed to a physicalist account of intelligence and a thesis about an unbounded intelligence, yet unwilling to go through the hard work of identifying the structural and behavioural constraints and taking them seriously.”[44] Nooooo, you are not intelligent, singularity is categorically impossible I scream as I am rapidly being processed into paper clips.

I really would like to understand or even believe this. And I don’t necessarily disagree with it. Maybe there are these, to me obscure, constraints to what constitutes intelligence and one main point of the whole book is to clarify them. But either I don’t get it and/or I don’t agree with it. Negarestani too often argues along the lines of “Well people naively believe A, but of course the more intricate B is the correct approach.” But he doesn’t goes into the details of why B is more justified to a satisfactory degree. And when faced with a high complexity of thoughts I am in the dilemma of whether there is some valuable insight hidden inside or if it is just metaphysical sleight of hand. The alternative conception of what intelligence presented simply isn’t convincing enough. That doesn’t mean that it is completely wrong, but that the way it is presented is not sufficient to make me really change my mind to the degree of certainty that he is giving.

Anyhow as of right now, we are getting a rapid increase of capabilities. And it is questioning many of our benchmarks of what we understand as intelligence.

(Source)

Here of course Negarestani would raise accusations of the form “You are anthropomorphizing what intelligence means”. But to me it raises the question of “How are you sure you aren’t?”. Pursuing this path to the end is futile in the similar lines of a radical skeptic who questions everything. I am still conflicted on this and I want to say that there is virtue in trying to de-bias ourselves and open us for other options. But I am also not sure how or what to do about that.

Going one step further, there is something quite ironic in that Negarestani criticizes that “the fanatic Kantian critical crusade against metaphysics only leads to an illusory disillusionment as one ends up with a stock of unexamined and unacknowledged metaphysical assumptions”[45]. But really while reading this Intelligence and Spirit I felt that his critical crusade against parochialism (here meaning human biased perspective) only leads to an illusory disillusionment as one ends up with a stock of unexamined and unacknowledged parochial assumptions. As is usual in philosophy, criticizing is easier than creating. This holds for Intelligence and Spirit. And this book review is probably no exception.

v. Where do we go from here?

Hedge fund managers come up with like 500 ideas per week and I assume 100% of them are absolutely crazy but 1% are maybe not that off and 0,01% are strangely on point. This has also been my experience reading this book. Out of nowhere Chu spaces appear as something that “...offers advantages which are necessary and crucial in the construction of a big toy model of AGI”[46], but after a few pages they are never discussed again. Suddenly, Hegel’s view of self-consciousness arises and a few pages further we jump back to something something type theory.

This parkour between the myriad of ideas can be frustrating, especially when they are picked up deus ex machina to ostensibly resolve questions about functionalism and dropped off as quickly. Negarestani never goes into as much detail or gives concrete explanations. Given the scope and length of the book this would be also hardly possible without deteriorating into a multiple “Infinite Jest”-long excessive stream of consciousness.

As you might have noticed, I didn’t give any deeper insight summary, but more broadly the motives that persisted. That is mainly because of this all over the place nature of the book. If there is a main point in the book, I confess I missed it. And the dense style in which it was written, does not make it easier. Not that I didn’t enjoy say the discussion on the asymmetry of time and certainly I didn’t run out of food for thought. But the wide scope of impulses don’t permit a closer inspection, or only a very opaque one. And this trade-off is not worth it to me.

Furthermore, I wonder who this book is written for. You get thrown into all this diverse variety of subjects and while there is a superficial explanation and further references for each of them, it is very cumbersome to work through it. Would I as a consequence recommend this book? Well, would I recommend Marx’ Capital to someone who wants to learn how (modern) economy works? Probably not. Maybe if you like fighting your way through a jungle of convoluted claims à la french 20th century psychoanalysts. But you will have to accept walking the tightrope between triviality and incomprehensibility in hopes of finding a novel idea.

vi. Post-mortem: (Academic) Philosophy in the 21st century

“It would be a paltry complaint to point out what is now obvious: that academia was conceived to push thinking to its ultimate unanticipated conclusions, but that academic philosophy today is a bureaucratic regime bent on containing thought within what is most predictable and mundane.“[47]

While reading Intelligence and Spirit I noticed that I was dissatisfied with the points mentioned in the previous section, but it did raise the more fundamental question “What do I actually expect from it?”

Why do we come up with all this gibberish about why intelligence is such a difficult concept and hard to define? Why not something like thinking of intelligence “... as a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience.”[48] Going one step further, we can start dismissing many questions as pseudo-problems. To me, this is a sign of resignation. I don’t understand really much about “consciousness” but I also cannot deny it and in face of this difficulty it feels wrong to me to just resign. We can’t ignore the problems away and you can make the argument that, historically speaking, thinking about these problems can be correlated with valuable outputs.

To begin with I would have hoped to not only find a good answer to the question “What is it that we are talking about?” but also “Why do we care about it?”. Ideally, I would find a more clear answer that goes beyond polemics. Whether the failure to provide a more clear answer to a wider range of people can be traced back to gatekeeping, learned behaviour or something else – I don’t know. This is a burden of modern academic philosophy and this book also didn’t make the big jump for me.

References:

Negarestani, Reza. Intelligence and Spirit. MIT Press, 2018.