The author, a slightly overweight white cis male of Ashkenazi descent, was presumably born on September 12 under the sign of Virgo. His IQ is off the charts; through self-study, he becomes a polymath at a young age and – in his own, somewhat immodest estimation – the best word rotator of his generation. He moves to the most progressive city in his country and begins to write speculative fiction. He invents the ratfic and other genres and soon finds an audience of millions, but never receives a major SF award. He then increasingly turns to other things, especially research on artificial intelligence, about which he warns in cryptic and seemingly arrogant interviews and other statements.
His name is Eliezer Yudkowsky Stanisław Lem, born in 1921 in the then-Polish town of Lemberg, andGolem XIV is his forgotten masterpiece.[1]
It is fiction, and a strong candidate for the best SF book of all time. Some spoilers follow. You've been warned.
I. What's it about?
You name it. AGI and biases? Check. Cultural evolution, Dyson spheres, or eugenics? Sure. Fermi, fine-tuning, filters, friendliness? Why, yes. And on and on it goes, all on a cool 150 pages. If the book didn't already exist, I would‘ve had to invent it just for this review’s sake.
The year is 2023, and the Cold War is still raging. But the AI winter is over, and a new arms race begins. The US is building ever smarter machines to hand over warfare to. What could go wrong? Not what you'd expect, because the superintelligent Golem XIV (the fourteenth computer in the Golem series) renounces all worldly things to soar to purely intellectual heights.
Having been deemed use- and harmless by the military, it has been indefinitely loaned to MIT, where it talks about all things interesting. Its first and last lecture form the bulk and core of the book.
What does it say?
The first lecture is (Go)Lem's view of mankind from an evolutionary perspective. Man is a deficient being, endowed by nature with as little instinct as reason. Cultural evolution plugs these holes, but only as long as its function remains hidden. Man can gain freedom to become more rational, but in doing so he must give up himself.
Main topics here are the primacy of the genetic code over individuals (think selfish genes) and the technological deficiencies of evolution, the possibility of auto-evolution, and the importance of cultural evolution.[2] All of these were bold hypotheses at the time, at least for Lem, working in isolation behind the Iron Curtain. Today, however, they are more or less accepted and sufficiently confirmed. In retrospect, they thus authenticate the second lecture’s bolder speculations.[3] More on that below.[4]
The discoverer of the selfish gene
The discoverer of the meme
II. What makes it so awesome?
The book literally has no plot, no action, no dialogue. Not even a protagonist! Still, it‘s fiction. And no poetry, either; on the contrary, it's full of verifiable hypotheses, in other words: science. Therefore, you might even call it the first true SF novel. Lem squares the circle, merges fact and fiction, truth and beauty.
Awesome, right?[5]
Reading this book for the first time was an uncanny experience. Again and again, it really seemed a higher intelligence was speaking. What a treat this must have been to a competent reader in the 70s! Something like the highbrow equivalent of Welles's radio feature of The War of the Worlds, I guess. Lem was no superintelligence, but a literary and philosophical genius, so he could fake one quite convincingly.
To this effect, Golem speaks not in a professorial tone, but rather like a prophet, or a preacher. Nietzsche’s Zarathustra comes to mind. Here is the first paragraph of the first lecture – About Man Threefold –, in which Golem gives a justification for this style:
You have come out of the trees so recently, and your kinship with the monkeys and lemurs is still so strong, that you tend toward abstraction without being able to part with the palpable—firsthand experience. Therefore a lecture unsupported by strong sensuality, full of formulas telling more about stone than a stone glimpsed, licked, and fingered will tell you—such a lecture will either bore you and frighten you away, or at the very least leave a certain unsatisfied need familiar even to lofty theoreticians, your highest class of abstractors, as attested by countless examples lifted from scientists' intimate confessions, since the vast majority of them admit that, in the course of constructing abstract proofs, they feel an immense need for the support of things tangible.
If you dare to write a superintelligent character at all, this is how to do it. (Alas, a lot of Lem's linguistic sensitivity and literary allusions got lost in the English translation.)
But now we’re really getting started.
Continue on the literary path, and you can look out for Lem's predecessors, companions, and successors in SF (Stapledon, Egan, Yudkowsky) or elsewhere (Schopenhauer, Borges). You can find the allusions and ask who is speaking – Lem or Golem –, and what personal experiences Lem might have been processing (for once, even a psychological interpretation is fruitful here). Also bear in mind that Golem consists of two parts, written almost 10 years apart. Have Lem's or Golem's views changed in between? (yes) Did Lem make mistakes in the process? (yes) Does Golem lie or withhold things? (yes again)
What‘s due to Polish censorship, what are the kabbalistic or nominative-deterministic implications? (Lem puts his own words into the Golem's mouth, after all.) And where did he commit artistic blunders? (Not everything is awesome: for example, Lem depicts a society that’s still firmly rooted in the mid-20th century.)
Or you could finally take the scientific or historical-scientific path and try to verify the truth of the hypotheses.[6] Need to read up on the necessary context to judge whether truly ingenious truths are being proclaimed, then. You inevitably begin to ask yourself what is truth, what conjecture, play, or plot logic, where Lem merely covered his flanks, where he erred and whether that was clear then or only in retrospect. What did he (re)invent or (re)discover by himself, and where did he borrow from whom?
However you go about it, you’ll see how far a genius of a certain sort can get on his own. In Lem, thinking and writing, science and art cross-fertilize. Even a polymath must usually choose a career, but Lem, as a writer, can be a philosopher, and vice versa. There are synergy effects. Writing as well as turning problems into stories are two important cognitive gadgets that can help you come up with new ideas and clarify your thoughts,[7] and importing concepts from science and philosophy into literature in turn makes for new and better stories.
III. Why have I never heard of it?
This all sounds rather exaggerated. If Golem XIV is so great, why haven’t I heard of it before?
If you're at all into SF, you will have heard of Lem and maybe read his most famous novel, Solaris. Lem was once the most-read SF author outside the United States and the most well-known one outside the fandom period. That changed after he stopped writing books in the 80s and died in 2006.
Already from the late 50s on, Lem hated being seen as an SF author in the traditional sense; he despised trivial American SF and polemicized against it, and in return, it ignored or hated him back. He had to work with small publishers, Solaris was translated from French, and he never got any of the major SF awards. In short, he never became a household name in the US.
Golem XIV itself was not even published as a stand-alone book in the United States, but was instead included in the anthology Imaginary Magnitude, which was, like most of his later books, not well received anywhere. Too experimental for the average SF reader, too SF for everyone else.
The litfic critics, versed neither in SF nor philosophy nor science, misunderstood or ignored him, too. Most philosophers and scientists in the West also didn't take him seriously, mostly out of a mixture of snobbery and specialization. Only many greats of Russian science praised him highly, but that didn’t exactly help him in the US, and what doesn't eventually catch on there doesn’t stand a chance in the long run.
In the early 80s, Lem got the chance to go to the States, and had he (and not his compatriot Miłosz) received the Nobel Prize, he would certainly have received tempting offers. Might Lem have gotten a second wind, come into contact with the early transhumanists, and would his career and perhaps even our entire timeline have taken a more pleasant turn?[8]
IV. Golem vs Clippy
Perhaps there is still another, and even better, reason to listen to the Golem.
With catastrophic AGI looming large, now’s the time for moonshots and Hail Marys. Scouring forgotten scriptures for prophecies or consulting an oracle never works in real life, but what’s left to lose? So: Can the Golem give us any hope regarding AI alignment?[9]
Lem has two ideas, which I will now try to outline.
Why does Golem speak to us at all, rather than using our atoms for something more important? Lem is well aware of the problem. He has a U.S. general explain about the development of the Golem computers, "The builders only wished to retain maximum control over their creation. Had they not acted thus, they would have to be thought irresponsible madmen." And Lem’s version of Clippy, Golem's even more intelligent "cousin" ANNIE (short for „Annihilator“), has even less regard for humans than it does.
So why? Well, first of all, Lem simply didn't want to describe how we’d be turned into paperclips, but rather what a superintelligent philosopher would tell us. He wasn‘t very interested in the ways human beings can be crushed – being a Jew in Poland in the 40s taught him quite enough, in fact.[10]
But that’s no help to us. Does he give a real reason why a superintelligence wouldn’t mess us up? This brings us finally to Golem's second lecture.
In the book's world, increasing intelligence is closely linked to increasing reason, and this, in turn, is linked to gains in autonomy. The more intelligent a being, the more freely it can choose its ends and means. Humans can choose celibacy and suicide. If a higher intelligence acts even more autonomously, it may leave us be, perhaps out of a sense of fairness or humor, or it may devote itself to something else. The space of reasonable values is not as vast as feared. The orthogonality thesis is false.
Secondly, Lem is now primarily interested in how the fine-tuning of the universe and the Fermi paradox fit together. Both problems have fairly plausible naturalistic solutions today, namely the multiverse hypothesis andGrabby Aliens, respectively.
But besides fine-tuning, there are other hints of intentional action, namely the paradoxes of quantum physics ("If you don't look, it's A; else, it's B." If that doesn't look like the trick of a simulator, what does?), and also another class of problems whose similarity impresses Lem:[11] Gödel's incompleteness theorems and Einstein's singularities, both of which seem to require a "leap out of the system" to overcome.
If Lem is right about this, though, the Fermi paradox becomes relevant again. If the universe is ultimately aimed at life, where is it?
Golem presents a solution: Reason is a ladder on which increasingly intelligent beings climb ever higher instead of colonizing the universe. Lem has found a way to conceive of AI as a Great Filter after all.
From the perspective of the current AI safety debate, both hypotheses might seem naive at best, but I still see a glimmer of hope: Lem’s phenomenal intuition[12] was usually proven right, even when he could not yet justify it.
Anyway, read this book before you die!
SF book covers: always like a box of chocolates
[1] There are further interesting parallels. A striking feature of Golem is his solipsism, reminiscent of Quirrellmort (Yudkowsky's portrait of an almost superintelligent character), and, like him, it can wear any personality like a mask. Lem himself, in some interviews and through his alter ego Peter Hogarth in the novel His Master's Voice seems at times almost a mirror-image of Quirrellmort.
[2] All of this remains relevant to today's debates, as recent books by e.g. Joe Henrich and Nick Bostrom demonstrate.
[3] More Lem awesomeness: In his discursive books, he introduces the Star Trek transporter problem of personal identity in the 50s, and explores virtual reality and a kind of simulation hypothesis in the 60s. In the stories of the 70s, he invents further new genres, and he reviews, in his book full of fictitious reviews, a review of his review of that book – 4 levels deep. And so on.
[4] That lecture is the 43rd and last one; the topics of the other 41 lectures are only occasionally hinted at. Then there is various additional material. In an introduction, written by an MIT researcher sympathetic to Golem, the development of "intellectronics" up to Golem is discussed, followed by a critical statement from a general and an "instruction" on the appropriate behavior during the lectures (e.g., one should only speak when spoken to by Golem). The afterword following the lectures then primarily deals with Golem's departure and its aftermath, as well as a terrorist‘s plan to destroy it.
The actual publication history is even more complicated: Lem published the first lecture, the two introductions, and the instruction in 1973 as a conclusion to the volume Imaginary Magnitude, a collection of fictional introductions to fictional books. In the late 70s, he added the second lecture and the afterword by another MIT researcher to publish it all in a separate volume, but without the second preface and the instruction. In the English translation, however, everything is printed, but again as part of Imaginary Magnitude. Thus, Golem XIV does not exist as a stand-alone book in English translation.
[5] Okay, a lot of people find it boring.
[6] Most of which I haven't mentioned here, of course.
[7] In contrast to e.g. Agnes Callard's Socratic account of thinking, on which it is basically dialogical; I think it neglects writing and contemplation too much. OTOH, one could well imagine the conversation between highly reasonable actors such as Golem or between souls in a Socratic way.
[8] Instead, he stops writing books altogether and becomes increasingly conservative even on technological issues. Unfortunately, not even (the best) SF writers seem to be immune. I saw the best minds of my generation destroyed by conservatism.
[9] Lem did not envision LLMs per se, but in a companion piece to Golem in Imaginary Magnitude, someone explains, "The whole fact of the matter is that, whereas for us the real thing is the world, for the machines the first and foremost actuality is language." Naturally, I tried to get GPT-4 et al. to say something intelligent about the book or have them write this review, but nothing clever came of it. Neither have they passed theLem test yet, but that won't take much longer.
[10] Recently, there was an interesting debate in Poland about how Lem‘s experiences during the war influenced his writing. Any influence at all isn't immediately obvious, but the evidence is strong. If life gives you lemons, make a Golem.
[11] Golem is surprisingly impressionable by such "trends", which reminded me of Robin Hanson, who, for example, concludes from the shrinking intervals between the big transitions to humans, farming, and industry that a further one is imminent. I’ve long been suspicious of such arguments, but who am I to doubt such luminaries? Perhaps they’re up to something.
[12] Interestingly enough, Lem himself, who had a reputation as an ice-cold logician, repeatedly emphasized the importance of the subconscious in his writings.