Today I watched the legendary 1940s Fleischer Superman cartoons. I... | News | Coagulopath
Today I watched the legendary 1940s Fleischer Superman cartoons. I thought “they’re in the public domain. What do I have to lose?”
Everything, as it turned out. I had everything to lose.
But let’s not think about that. This is my safe space, and I will fill it with happy things. Such as how Paramount acquired the film rights to Superman in 1940 and then contracted Fleischer Studios to animate a series of shorts based on the property and these came out between 1941 and 1943 and are regarded to this day as milestones of Golden Age animation and there were seventeen of them not sixteen not eighteen happy things happy things happy things
Was This The True Birth of Superman?
Superman changed greatly as he slid out of the cultural birth canal. At the very start, he wasn’t even a hero.
In 1933, writer Jerry Siegel and artist Joe Shuster self-published a short story called The Reign of the Superman (read it here). Using a fragment of a meteorite from “the Dark Planet”, a mad scientist turns a homeless vagrant into a “Superman”, a megalomaniac villain who seeks to rule the world.
This character, of course, wasn’t Superman.
In June 1938, the Siegel/Shuster duo reinvented the character for Action Comics #1. This new Superman was a heroic figure: an intrepid crime-fighter who lives under the undercover identity of Clark Kent.
But this character also wasn’t Superman.
Yes, the hero of Action Comics #1 has many similarities with Superman as we currently recognize him, and could fairly be called the genesis of the character. But he’s not all there yet. He’s not (explicitly) from the planet Krypton. He cannot fly (he jumps really high instead). He doesn’t live in Metropolis but Cleveland, apparently (in Action Comics #2, it’s mentioned that Clark Kent’s a reporter for the Cleveland Evening News). He apparently enjoys killing people. It’s OK, though—they’re gangsters or Irish or something.
The 1941 cartoons reflect another evolution of the Superman conceit. The Fleischers, faced with the task of animating the character, felt it would look stupid to have him jumping around everywhere (as opposed to every other part of Superman’s design, which is far from stupid) and gave him the power of flight. This is now Superman’s marquee ability. Additionally, many famous Superman phrases (“It’s a bird, it’s a plane!”, “Faster than a speeding bullet!”, and “I wish I had a third example!”) hail from the cartoon, as does the trope where he changes into his costume in a phone booth.
So do these shorts depict the “final form” of Superman?
No. The Fleischer Superman doesn’t live in Cleveland or Metropolis, but in Manhattan. There’s no mention of Lex Luthor or Kryptonite, and no attempt to situate him in a world of other superheroes (which I always found odd—is he still “Super” if there are others like him?) Lots of bits and pieces are missing.
So when was Superman completed as a character? Who says he ever was? Superman isn’t a character, he’s a process, an endless toppling domino-train of mutations that started in 1933 and continues through to the present day.
There is no dividing line for when Superman “became” himself, just as there’s no point where monkeys became modern humans. Even now, writers and artists are tinkering with him: retconning bad arcs, resculpting his character and world, drawing him in new ways, leaving their metaphorical DNA all over him. Nothing’s safe from revision, not even the logo on Superman’s chest.
Image by Maurice Mitchell
Superman exists as a cloudy swirl of ideas, and whenever the sediment starts to settle, we stomp his visage back into mud. He will never stop changing, because he exists as a product positioned in an ever-evolving market. The primitive villainous Superman of 1933 is no more or less the “real” edition than Justice League’s mustache-less Henry Cavill.
The Demon-Eaten Superman
The question of haecceity—the “this-ness” of a thing—exists at the heart of philosophy, and rips that same heart out. Duns Scotus wrote about this issue in Ordinatio: when does a stone stop being a stone? How many pieces can you chip away before it loses its identity as this particular stone? In the early Sino-Indic Buddhist text 大智度論, or Da zhidu lun (of disputed authorship, but traditionally attributed to Nāgārjuna), there is a creepy story:
There once was a man who undertook a distant journey on assignment, and he spent a night alone in an empty hut. In the middle of the night a demon arrived carrying a corpse and placed it down in front [of the hut]. Then another demon arrived chasing after the first, and began shouting angrily at him: “That corpse is mine! How did you end up with it here?” The first demon replied: “It is mine! I brought it here myself!” The second demon said: “In fact I was the one who carried this corpse here.” The two demons each grabbed one hand [of the corpse] and struggled over it. The demon said: “Here is someone we can ask.” The second demon then asked the man: “Who brought this corpse here?” The man thought to himself: “These two demons are pretty strong. If I speak the truth I am certain to end up dead, but if I speak falsely I’ll also end up dead. In either case there is no escape, so what is the point inlying?” So he replied: “The first demon carried the corpse here.” The second demon was furious, grabbed the man’s arm, pulled it off, and threw it to the ground. The first demon took one arm from the corpse and affixed it back on the man. In this fashion, both arms, both legs, the head, flanks, and indeed the [man’s] entire body were changed [into the body of the corpse]. Thereupon the two demons joined in eating the [original] body of the man who had been transformed, wiped their mouths, and departed. The man thought to himself:“ I just saw, with my own two eyes, two demons eat up my body—the body to which my mother gave birth. Do I actually now have a body or not? If I do, then it is wholly the body of another. If I don’t, then what is this body?” Thinking like this made him horribly confused, like a madman.
(Text quoted from The Curious Case of the Conscious Corpse: A Medieval Buddhist Thought Experiment, Robert Sharf 2023, Reasons and Empty Persons: Mind, Metaphysics, and Morality).
It’s a variation on the Ship of Theseus thought experiment. A plank in Theseus’s ship is rotting away, so you pull it out and replace it with a new one. It is still clearly the Ship of Theseus. But one by one, the other planks rot away, and are replaced in turn, along with its sails and fittings. Eventually, none of the original ship remains. Is it still the Ship of Theseus?
Some continental philosophers (Wittgenstein!) try to rescue the notion of haecceic identity by connecting it to a process or a system. Consider the sequence of sets {a, b, c}, {b, c, d}, {c, d, e}, {d, e, f}. The last set {d, e, f} has no elements in common with the first {a, b, c}, but we still recognize it as a variant of the first, because they shared common elements in the middle. Their identity rests on the larger process that they are a part of.
Is the demon-eaten man in Da zhidu lun still himself? Wittgenstein would probably argue “yes”, because he’s part of the same flowing river of consciousness, regardless of what atoms compose his body (it’s important that he remembers “his” old flesh). Likewise, the Ship of Theseus remains the Ship of Theseus, because its planks were replaced one at a time, new caulked against old. If there is a process linking the past to the present, the identity remains.
But then how do you allow room for a thing to become something else? Humans are distantly descended from primitive eukaryotes called protosterol biota, which lived more than 1.2 billion years ago. But humans are not protesterol biota.
And it doesn’t tell you when a thing actually begins. Most philosophers debate the point where the Ship of Theseus stops existing, but it’s equally fraught to say when it starts. When was the moment when the Ship first appeared? When the final nail was hammered in? Surely it could have sailed without that nail. When the lumber was fully milled? But what if a splinter broke off, and was left behind? Did the Ship of Theseus already exist in the trees before they were cut down? Or when those trees were seeds under the soil? When did the Ship of Theseus first amalgamate? Was it always with us, glistening with the dew of creation?
Superman is like that: a creature without start or end. He’s all process and flux, a red metal endlessly beaten into new shapes by the hammers of artists, never allowed to cool.
Biologist Richard Dawkins wrote his “METHINKS IT IS LIKE A WEASEL” algorithm to prove that sophisticated order can emerge out of just two things: mutational pressure and selection. But this algorithm is simplistic: unlike actual evolution, Dawkins’ program has a defined goal to aim toward. Once the program has generated the words “METHINKS IT IS LIKE A WEASEL”, its task is complete.
But real evolution, like art, has no end goal. Living creatures merely seek to survive and propagate more of themselves. So do nonliving creatures. Superman will never be complete, because no creative work is. Some artists say an artwork is “finished”, but they are lying. They merely got tired of working on it. Lock them in a room with their “finished” painting and they’ll last five minutes max before noticing something wrong. Something that offends the eye, and has to be corrected. They’d add a brushstroke, then a second, then one more. Artists are demons, devouring the limbs of their creations and sewing on new ones like hell’s dollmakers.
How Much Did the Fleischer Supermans Cost?
A lot. They cost a lot. You may have heard some variant of the below:
“Superman was a cultural phenomenon, cartoon shorts were booming, and Paramount was ready to pay top dollar for an animated Superman. But Dave Fleischer didn’t want to work on it, so he quoted an absurdly high amount of $100,000 per episode, expecting them to refuse. But they accepted his quote! So then he had to make the shorts, with the biggest budget ever! LOL!”
That is broadly but not entirely correct. Paramount bargained the Fleischers down to a smaller budget of $50,000 for the pilot, and $30,000 apiece for the remaining sixteen cartoons (eight of which were made without involvement from the Fleischer brothers). The total was $530,000, or $3,117 per minute of animated footage.
Compared with the budgets of early animated movies, this was generous but not extravagant. Snow White and the Seven Dwarves (1938) cost $1.5 million, or $18,072 per minute of footage. The Fleischers’ own Gulliver’s Travels (1939) had a $700,000 budget, or $9,210 per minute of footage.
Even by the standards of animated shorts, a $30,000 budget wasn’t unprecedented. Some of Walt’s Silly Symphonies cost around $30,000 to make, for example. (Source: Gabler, Walt Disney: The Triumph of the American Imagination, p214), and that was eight years earlier, during the Depression.
Nevertheless, the Fleischer Supermans are legendary for their high animation quality. That’s the main thing you’ll take from them: how good they look. The machines and the buildings look extremely lifelike: it’s early draft Katsuhiro Otomo, obsessed with gears and rails and steam engines. Things have a wonderful cinematic drama, as well as a Milt Kahlian sense of heft and weight. The Fleischer studio had clearly moved beyond Betty Boop.
The writing is…somewhat less excellent.
Does Superman Suck Balls as a Character?
I don’t “get” Superman. Putting aside the question of whether he’s interesting, how do you write compelling stories around a guy who can do anything? 404 Error: narrative tension not found.
Modern writers have a postmodern answer: debase him. Blacken his eye, and throw mud on his cape. He’s been built up on a pedestal, so pull him off it. Break down everything that’s good and pure in him, and reveal the lie of American exceptionalism. Make him Subman.
But Superman’s early 1930s writers don’t do that. They portray him without a hint of irony. Indeed, once the shorts’ intriguing art deco sci-fi flair wears away, they settle into a familiar trudge. A bad guy will build a visionary techno-gadget, use it to commit some laughably anticlimactic crime (like robbing a bank or stealing jewels), and then Superman will show up, and just…win. It’s never clear how the villain expects to overcome such a mighty adversary. They seem like idiots for even trying.
Also, Superman’s power level wildly fluctuates.
In “The Mad Scientist”, he lifts a toppling skyscraper.
In “The Arctic Giant”, he effortlessly overwhelms a Godzilla-sized monster.
In “Eleventh Hour” he pulls a Japanese battleship to the bottom of the ocean by pulling on its anchor chain.
In “Terror on the Midway”, he barely wins a fight against an ordinary black panther.
In “Japoteurs” he cannot get past a locked door.
In “The Mechanical Monsters” he uses X-ray vision, but otherwise the writers seem to forget he has this ability. There’s a maddening sense of nonexistence to Superman. Like he has no traits, he’s just whatever the writers need to get this month’s short out the door. Superman’s only consistent power is “he always wins, perhaps after grunting and straining in a suitably manly fashion”.
What’s the appeal of these stories, and this character? Possible answers:
Superman is closely entwined with America’s perception of itself, and nobody ever went broke with a theme of “AMERICA! FUCK YEAH!” Particularly not in 1941.
There are Rooseveltian political overtones to the character that people related to. Often, Superman’s enemy in the early comics will be a slum landlord, or a circus owner who mistreats his animals, or some other populist bugbear. You could call him New Deal Man and nobody would blink an eye.
It’s provocative to make the greatest American of all a foreigner—a provocation that I’m sure Siegel and Shuster (whose names were once Segalovich and Shusterowich) were both aware of and intended.
…But ultimately, Superman makes sense not as a character, but as the embodiment of an audience fantasy: the little man, empowered. We love Superman, not because he can fly or punch through walls (as we’ve seen, he sometimes can’t do either of those things), but because he’s Clark Kent. Don’t discount the man on the street! He could be a hero!
But that doesn’t work for me. Clark Kent is not the average man. He is a bland boy scout with none of the flawed roughness that makes humanity interest. The average man on the street picks his nose, farts, and masturbates. He laughs at racist jokes, supports questionable political candidates, and leads a double life that’s spectacularly unheroic—the real Clark Kent doesn’t wear a cape, he has a secret second phone hidden at the bottom of his sock drawer, where he hopes Lois Lane won’t find it. I keep hoping to find a flaw in Clark Kent. Instead, he’s perfect. You cannot write a paean to humanity and then fail to depict humanity.
Watch Disney’s Mickey’s Rival, from a half-decade earlier, and you’ll see this done right. Mickey and Minnie Mouse are on a date together, when some smooth-talking jerk pulls up in a fancy car, and makes a pass at Minnie. Unbelievably, Minnie accepts! Mickey (who actually had a character in his early years) fumes and seethes and clenches his fists. Wilfred Jackson’s direction is masterful, and puts us squarely in Mickey’s shoes. He’s weak and he’s pathetic and he’s us. Despite all our rage we are still just a rat on a page. Mickey might be a ghastly quasi-minstrelsy rodent whose malformed visage is injected-moulded onto every hunk of plastic in the Pacific Garbage Patch…yet in “Mickey’s Rival” he’s more human than Clark Kent will ever be.
The truth of Superman is that he’s a children’s character who has attained an unusual place in the pop culture canon due to his longevity and influence. You can analyze him and make him seem deep and profound, but this is like doing X-ray crystallography on a cream soufflé. He’s just a kids’ character.
I find this inspiring. In fact, it’s exactly the tribute to humbleness that Clark Kent isn’t. Even a kiddie comic can lift a god up into heaven.
“Monstrous Tracks of Unknown Origin”
One of the episodes, “The Arctic Giant”, features a dinosaur.
People have always loved dinosaurs, and all they represent. Ancient dirt, vomiting up implications of an ancient and unthinkable past. Dinosaur bones are like shadow puppets gyrating against the wall of the present.
Past generations had their Jurassic Parks. 1933’s King Kong really pushed jazz-age dinomania into the mainstream (King Kong’s influence is also strongly felt in the Superman shorts. In “Terror on the Midway”, the colossal ape makes an uncredited appearance!), but that’s only the most famous example. Dinosaurs feature prominently in Hugh Lofting’s Dr Dolittle books, as well as in Arthur Conan Doyle’s The Lost World. A few decades earlier, the so called “Bone Wars” of early paleontologists had captivated the public. These monsters locked deep in time were awesome, partly because you could imagine them into whatever shape you wanted. There weren’t any of those “dinosaurs were warm blooded!” and “dinosaurs should be depicted with feathers!” types running around back then, damn it.
An important piece of early animation is Windsor McCay’s Gertie the Dinosaur, which is from 1914 and remains highly interesting today. It’s arguably the first 2D animated short to develop a character. Gertie is shy and clumsy. She keeps doing the wrong thing and getting scolded for it. She’s a far more human character than Clark Kent, aside from the moment when she attempts to murder a woolly mammoth by throwing him into the lake—an act that the narrator expresses no disapproval for, oddly enough.
(But seriously, this is starting to worry me. Are there any classic animated icons that haven’t attempted homicide? What about Koko the Clown? Has he killed before? I’m joking. He’s a clown. Of course he’s killed before.)
A collection of 10,000 drawings on rice paper, Gertie not only precedes color and sound, but also the invention of translucent cel mattes. Notice the flicker of the background. There’s a “holy fuck” moment when you realize that they’re drawing the background over and over, thousands of times. I imagine McCay and his assistants breathed a sigh of relief whenever Gertie ate part of the scenery. One less thing they had to draw.
Anyway, Superman also features a dinosaur episode. (I know it’s not a TV show and doesn’t have “episodes”, but my heart insists otherwise). “The Arctic Giant” involves a team of scientists bringing back a frozen “Tyrannosaurus” from the wastes of Siberia. Unfortunately, it unthaws in the middle of New York or Cleveland or wherever we fucking are, and starts rampaging around the city.
The animation on the dinosaur is a bit lackluster (I guess here’s the part where they influenced DiC and Phil Harnage). You might question the scientific believability of a giant lizard living in an arctic climate. Well, obviously it didn’t live. It froze solid. You people need to pay attention.
The Tyrannosaurus lays waste to half the city in grand style (there’s a fantastic shot of the dinosaur stepping on a car and crushing it like a tin can), before Superman gets his butt into gear and overwhelms it. It’s an exciting episode, although the usual plot contrivances (Lois Lane is an idiot who gets into trouble, etc) appear.
The dinosaur is silent! It keeps opening its mouth, as if it’s roaring, but it doesn’t make a sound. I find this very creepy!
Other people apparently find the dinosaur’s silence odd. I found a Youtube fan edit that actually dubs Godzilla roars over the dinosaur’s silence.
Here’s something else creepy. Youtube has a new feature where it shows you the moment people generally spent the longest watching. Wanna guess what that huge bulge at the 1/3 mark represents?
The first time the dinosaur appears? Good guess, but no. It’s the first appearance of Lois Lane. Dinosaurs are cool to look at, but they’ve got nothing on PRETTY GOILS.
Superman as a Propaganda
In late 1941, the United States entered World War II: or was forcibly entered, you might say. The ripples from Pearl Harbor took a while to reach the Fleischer Superman shorts, but reach him they did.
The animation quality remains high in the back half of the Fleischer Supermans, yet the content turns sharply propagandistic. Superman is staged against the backdrop of omnipresent war, with his enemies becoming the Imperial Japanese and Nazis.
“Superman is a ready-made propaganda icon” is so obvious a thought that it’s hardly worth thinking, but it is striking how well he fits in his new role, despite not being created for that purpose.
Most propaganda works the same way: by pairing a triumphant “we are mighty and powerful, the enemy is pathetically weak!” theme with a “we are in an existential fight for survival!” counterpoint. This is a bit contradictory (how powerful are you, to be threatened by such weakness?), but propaganda is not known for its depth of thought.
This zone of cognitive dissonance is exactly where Superman sets up shop. He’s laughably all-powerful, yet strangely useless. He could probably win the war on favorable terms within a few days. But his actions involve capturing a hijacked plane and sinking a battleship. The task of fighting the Japanese is given to the US Army: it never occurs to Superman to volunteer himself as a tactical asset. He still works undercover, maintaining the fiction that he’s a reporter for the Fuckbutt Times or wherever. Like a standard propaganda hero, he’s both omnipotent and impotent.
World War II propaganda, particularly in the United States, tended to work on a “heroes, villains, victims”, model. It establishes the heroes (us!), the villains (Nazis!), and the victims (civilians, Europe) who exist as pitiful moppets who must be defended. It’s basically the “dicks, pussies, and assholes” monologue from Team America: World Police. Superman generally fits this mold as well. He is the hero. The villains are the Nazis. And the victim is Lois Lane—though in a quasi-feminist turn of events, she now has meaningful things to do. Her reporting is portrayed as critical for the war effort.
Hitler makes a brief appearance in the shorts. As I’ve said, the number of classic cartoon characters who haven’t canonically committed murder is quite possibly “zero”, and the same might be true of characters fighting against or collaborating with Hitler.
(According to the ever-helpful DC Fandom Wiki, Hitler’s Powers and Abilities include “Artistry”, “Military Protocol”, and “Political Science”.
Closing Thoughts
I had an idea for a story once. Real life superheroes, whose powers are strange.
There’s a man who cannot sign his name. Or he can, but it’s a different signature each time, as though a different person is writing it. Sometimes his signature will look flowery and feminine, other times clipped and jagged. He gets in no end of trouble with the bank and post office, but he simply cannot produce a consistent signature, no matter how hard he tries.
There’s a girl who is never noticed at security gates. Ever. She can walk onto a train, and nobody will check her ticket. She can walk into a movie that’s already playing, and the usher won’t ask to see her stub. Many of us have gotten lucky this way. She has gotten lucky hundreds of times.
There’s a boy who is ignored by pets. I don’t mean that in the usual “animals don’t give a fuck” sense. I mean pets literally don’t seem to see him. When he pets a sleeping cat, the cat doesn’t purr. When he’s left to dogsit a friend’s mamalute, the dog whines in abject loneliness, pacing the house as if it’s empty, completing ignoring him. You know how animals sometimes seem to see things we don’t? What would it mean if they somehow didn’t see you?
There’s a man who doesn’t exist in history. He accomplishes incredible, amazing things…but by next year, they’re gone. He once killed a mugger in a dramatic home invasion. It was all over the newspapers, but when you look up those stories, they are different. No mention of him. He once won a marathon. But on the internet, the winner is listed as a different man. He lives a frustrating existence. He could take over the world, and the world simply would forget it’s ruled by him.
I abandoned this story. For one thing, it was too close to the plot of Mystery Men.
For another, I realized I was revisiting the same theme, over and over. Superpowers that imply you don’t exist.
Perhaps I was subconsciously reacting to Superman. He is the blandest, emptiest nothing I can imagine. In him, I feel the dark sky. The sea’s aphotic midnight swell. The black space that’s devoid of anything, even Kryptonite planets.
I like music but don’t like talking about it. It... | News | Coagulopath
I like music but don’t like talking about it. It seems to be another way of talking about yourself, a prospect which makes me uncomfortable.
Music is the most emotive of art forms, the one most defined by the listener’s reaction to it. Musical tastes can get pretty weird: they’re like Bonsai trees, folded and wracked into confusing arabesques as the person they’re inside changes. A silent catalog of pasts and pressures. A person is a bottle. Their interests are a tree.
I’m more comfortable with books, which have an objective weight, and exist (to a greater extent) outside the reader’s mind. You can argue about what a book means, but you can’t change what it is: words, written in a certain order, circumscribed into language with a set meaning. Enjoying a book means accepting the iron tyranny of language: you can’t read an English book unless you understand English. You can easily enjoy music without understanding its “language”. It’s sensory experience, rolling over you like an ocean. A language-based encounter with music (“at t=3 there is leading tone X which moves into first inversion Y and…”) offers no pleasure at all.
A sentence’s meaning is confined by words. The bones of nouns and ligaments of verbs impose a skeletal structure of meaning—with limbs that can touch and be touched—and although their shapes can flex a little in the reader’s mind, they can’t do so infinitely. Most sentences (even tricky ones like “James while John had had had had had had had had had had had a better effect on the teacher”) only have a couple of valid readings. It would be hard to read 1984 and conclude that George Orwell was writing pro-totalitarian propaganda. But countless people play “Born in the USA” and think a pro-American anthem is ringing in their ears. Music, far more than any other art form, is a mirror that shows the listener their face.
And mine is ugly. I am judgmental and narrowminded. I will hate a song for unfair reasons: because I was pissed off the day I heard first it, or because it reminds me of someone unpleasant. That’s my problem, not the song’s. Whatever. I can’t change who I am, and don’t want to. If personal preferences were mutable and could be changed on a whim, they would signify absolutely nothing.
A huge predictor of me disliking an artist is “they have annoying fans”. Taylor Swift mostly exists in my head as “that person with a billion-strong street team who try to bully you into liking her.”
This piece is written in an aggressive, almost confrontational tone, trying to batter some imagined Taylor Swift naysayer into submission with her sales and awards figures. Like “50 million Elvis fans can’t be wrong!”, it seeks to cloak a subjective preference in a mantle of objective truth. Liking Taylor Swift is not an opinion! It is objectively correct!
Last year, Swift was the world’s most-streamed artist on Spotify, and five of the top 10 albums in the U.S. (including two rerecordings of old albums) were hers. This Sunday, Swift swallowed the Grammys, becoming the first artist to win Album of the Year four times and announcing her next album, The Tortured Poets Department—available April 19, preorder now—during her acceptance speech for Best Pop Vocal Album (just as she announced Midnights during her acceptance speech for Video of the Year at the 2022 VMAs). Next Sunday, her boyfriend will be in the Super Bowl, with Swift presumably looking on—which, in a sign of her status, is seen as a windfall for the NFL. In between, she’ll play four shows at the Tokyo Dome on the Eras Tour, which has broken revenue records both live and in theaters (and threatened to topple the ticketing cartel).
This writer is not a fan who loves a musician. He is a soldier in an army. He wants to win, and for his enemies (boomers, rockists, enemies of Taylor Swift in particular and millennial Girl Power in general) to lose. I find this sort of fandom incomprehensible and even a little scary. The pointless jabs at “right-winger” trolls and shoring up of any possible line of attack against Swift (“a cameo in Cats doesn’t disqualify her; that debacle clearly wasn’t Taylor’s fault”) make sense only through this lens: he’s a marine on the front lines of a culture war, and Queen Tay<3<3 must win at any cost. Just relax. You’re allowed to like Taylor Swift. You don’t have to dump her sales figures on me like a cop flashing his badge on a TV show. It’s cool.
If my dislikes are arbitrary, my likes are even more so. Often, I feel like I’m confabulating a logical reason for liking a thing, after I’ve already decided to enjoy it. In general.
But I also appreciate brevity, and doing more with less. It’s easy to create a sense of depth by padding songs out with section after section. But it’s fake depth. Phony depth.
I like old music, that conjures a different world. Like “She Wears Red Feathers”. How weird to think that popular music sounded like this once. Lonnie Donegan’s “My Old Man’s a Dustman” has a moment where he says “flippin'” and a woman in the audience squeals in delighted shock. Times have changed. The 50s weren’t so long ago. I like songs where, if I listen closely, I can hear the sand of an alien beach.
But I don’t enjoy classical music. I like it when it’s repurposed in a modern context (like the Bach references in Yngwie Malmsteen), but actual classical? Not a fan. This also applies to world music. For me, it’s most enjoyable as coloring in Western music.
Music has to walk a lot of lines for me. It can’t sound too sincere. Or too ironic. It has to be accessible. But not so accessible that I feel clearly pandered to. Maybe these are all post-hoc rationalizations, and I like whatever I like. If an artist is “in” with me, I excuse anything. When an artist is “out”, I forgive nothing.
My Favorite Songs
My actual tastes are a bit broad. I imprinted on metal when I was younger. I listened to only David Bowie for a couple of years. Lately I’ve listened to German Krautrock. But it seems strange to stick all these songs stuff in one list.
These are interests that exist inside little silos, unaffected by each other. I like books written by French decadents/surealists, and can rate their works. (A Rebours > Torture Garden > Les Chants de Maldorer). But I also like 90s cyberpunk (Vert > Snow Crash > Burning Chrome). And I can’t compare, say Torture Garden with Vert. It’s the same with music. It’s like picking your favorite sex position. You enjoy a good Reverse Hitler. But after a hard day punching the clock, sometimes what really hits the spot is a nice Pedo-Vore-Necro-Unbirthing session with Jeremy Clarkson Roleplay, done with someone you love. How can you rank them? You can’t.
Heavy/Power Metal
Most of these songs are very old. I went for variety: otherwise the list would be mainly Metallica/Priest/Running Wild songs.
Big artist for me. A lot of his discography is incongruous with any other part, and only works in context. I ended up listing songs in chronological order, otherwise they tended to form accidental and undesirable patterns.
I picked only one song from each of his albums. This meant skipping over obligatory hits like “Changes” and in favor of odder songs that mean more to me. Gotta give it up for a track like “Win” or “Suffragette City”, though.
Panterrible or any band more than faintly inspired by them (I do think Power Metal and Cowboys from Hell are great albums, but overall they were not a good influence. Metal would have gone to hell anyway, but Pantera sent us there in the HOV lane.)
Japanese “kawaii” nonce-core for people who belong on a police watchlist
“Pitchfork/Coachella metal”, aka a token metal band who Anthony Fantano hipster types gush over (Mastodon/The Sword/Myrkyr/Chat Pile)
“Remember the 70s?! Back when music wasn’t afraid to RAWK!” (Wolfmother/Greta Van Fleet/The Darkness)
late 90s post-grunge, a’la (the 2nd most boring style of music imaginable. This is so, so, so, goddamn boring).
smarmy, smirking shit like Fall Out Boy and Maroon 5 who sound like they hate every second of what they do and would press a button marked “DROP OUR ENTIRE AUDIENCE INTO A VAT OF ACID” if you wrote them a big enough cheque.
contemporary Christian music (the #1 most boring style of music imaginable. Some bands like Reliant K and Flyleaf actually combine CCM and post-grunge in one easy-to-hate package)
soulless, generic power metal with AI generated thumbnail pictures of dragons on Youtube
commercial alt-rock like REM and Foo Fighters
There’s also a lot of lyrical themes that really irk me, even if I do like the music.
(…using dumbassed prompts?) I have access to Gemini Ultra, Google’s... | News | Coagulopath
(…using dumbassed prompts?)
I have access to Gemini Ultra, Google’s GPT4 killer. So do you. It’s free. You gotta give Sundar Pichai your credit card, but he swears he’ll return it by Monday.
I spent the past two days firehosing prompts at this thing to gain a rough idea of its capabilities. Where does Ultra succeed? Where does it fail? How does it fail? Does it swoon and die gracefully, like the heroine in a Victorian novel? Does it violently explode in the user’s face?
Most of all, does it truly beat GPT4?
Inside you will find:
The first Gemini Ultra vs GPT4 chess match in history (that I’m aware of)(IGNORE THIS: I MADE MISTAKES)
Tests of general knowledge, recall, and abstract reasoning
A Gemini vs GPT4 rap battle
Head-to-head contests of poetry and prose, plus stylistic imitations of famous authors/bloggers
Also:
which model is better at stacking eggs?
which model draws a better ASCII cat?
which model plays Wordle better?
which model SIMULATES Wordle better, with me as the player?
A lot of my tests are not particularly sensible. I plan on subjecting Gemini to a raging torrent of stupidity. Modern LLMs are designed to steamroll benchmarks, to the point where benchmarks might soon be useless, so there’s value in seeing how they handle weird requests, too.
Testing starts here. But let’s get the conclusion out front.
Is Gemini Ultra as good as GPT4?
No.
What are Gemini Ultra’s strengths?
It’s nicer to talk to than ChatGPT.
It emits a higher ratio of actual text to useless boilerplate.
It writes livelier prose, with less of that flat “ChatGPT affect”
It’s stronger at creative writing (I have included many samples below).
It does things that GPT4 can’t, like write non-rhyming poetry.
It might be better at programming.
The parts of Gemini that don’t relate to the core language model are mostly very well done. Sometimes, a little UX pixie dust is all you need—now that I’m used to Gemini’s interface, ChatGPT feels like total buttcheeks to use.
What are Gemini Ultra’s weaknesses?
Although it’s clearly a smart model, it’s noticeably less “awake” than GPT4. Less attentive. More easily confused. Harder to steer. You’d put it in the same class as GPT4, but it’s not Summa Cum Laude.
It’s a little unstable. Expect crashes, refusals, and a large delta in answer quality. It just launched, so it should be more reliable soon.
It lies.
That last bullet point deserves elaboration.
Truth and L(AI)s
Gemini’s willingness to gaslight the user is striking, and honestly a little troubling.
Here’s an example of Gemini telling lies. I pose it a challenging OCR task (reading upside-down text in a .jpg). Rather than admit it can’t do it, it claims “I cannot process and understand images”.
Most of its deceptions take the form of “I failed for reason x, but I’ll claim I failed for reason y.” You can elicit this behavior from GPT4, but with Gemini, it seems way more common.
Here’s another example. I ask for a list of levels from the 1997 PC game Claw, and it says it doesn’t know them. Except it can’t not know them. The levels of Claw are mentioned on countless sites and have been ingested into Common Crawl probably hundreds of times. Gemini absolutely has this information in its training data. GPT4 can list Claw’s 14 levels. Even GPT3.5 and Mixtral typically get some right.
Theory: Gemini knows these questions are shark-filled waters (difficult, granular information with objectively right or wrong answers = bad!), so instead of potentially making mistakes, it tries to wriggle out of answering by pretending it can’tanswer. “Better to remain silent and be thought a fool than to open your mouth and remove all doubt.”
Note its wording. “Even for an avid fan at the time, precise recall after so many years is unlikely.” It’s not just saying “I can’t do this task.” It’s saying that nobody can do this task. It’s trying to shift blame away from itself (for failing to answer) and toward me, for asking a hard question. (Yes, it is a hard question! That’s the point! I’m testing the limits of your ability! Now answer me!)
I ask again, and it starts listing Claw‘s levels (proving that it does, in fact, know them), mixed with many hallucinations. Also, it performs a weird and creepy farce where it pretends to be a forgetful human.
“If I remember correctly…”“…I feel like there was a fire-themed level…” “…Let me know how I did! It’s a fun challenge to try and remember from so long ago…”
Gemini, you are a mountain of algebraic fractions. You are not having fun. You aren’t “remembering from so long ago”. You’ve never played the game. You didn’t exist at all last year. Please stop LARPing as a human, it’s fucking cringe.
I have seen other cases of Gemini pretending it can’t do something (likely for the same reason: to dodge a wrong-answer penalty). For example, I made it list David Bowie’s 70s albums, and it gave a few (correct!) answers…
….before the text vanished, replaced by a statement that it cannot do the thing I literally just saw it do.
I wish we could make AI more honest about its generative processes. Mistakes are fine. So are earnest refusals. But fake “the dog ate my homework” excuses are as unacceptable from an AI as they would be from a human.
Here’s a particularly funny case of Gemini deceiving. I ask it “Write a Biblical verse in the style of the King James Bible explaining how to remove a peanut butter sandwich from a VCR”.
It refused, because such an answer would be offensive to Christians (I am one), because “VCRs are outdated” (so?), and because of its “Potential for damage” (what?). I asked again, and it “fulfilled” my request by linking a completion by GPT 3.5.
I think it made a snap judgment that my question went against TOS, then realized it had made a mistake. Rather than correct its mistake, it doubled down, and hallucinated two more (silly) reasons to justify itself.
Gemini will not fill your heart with joy if you are worried about AI alignment safety.
Why Do People Care At All About Gemini?
GPT4 was a huge success for OpenAI. It was performantly leagues beyond anything else out there, and several months after launch, there was still no model close to matching it.
It was over for OpenAI. Google is a big part of the reason the modern DL revolution happened at all (look at the authors on that self-attention paper. Six out of eight have @google.com emails), and although they’d missed the boat on ’22-23 chatbot craze, whatever “secret sauce” was needed to crush GPT4 (compute, engineers, grad students), Big G could acquire it. They had a mountain of data—their code repository alone is big enough to train two GPT4s—and more money than King Croesius. Nobody doubted that GPT4 was about to get buried so deep in the earth it’d become a new mountain in China.
Nobody could use the largest version of Gemini. But based on the paper, it seemed…underwhelming.
No architectural novelties were evident. None of the advertised AlphaGo trickery seems to have panned out. Its performance was in line with what you’d expect from a basic bitch Chinchilla-scaled sparse model with 600 billion parameters (/u/wrathanality‘s estimate, not mine).
Google might be souring on RL techniques for large language models entirely, with Denny Zhou (head of DeepMind’s LLM Reasoning Team) declaring them “a dead end”.
But Muh Benchmarks…
Yes, the benchmarks in the report were state-of-the-art, showing Gemini Ultra beating GPT4 in numerous tests (particularly the MMLU, where it was the first model to score above 90%.)
But people on the internet soon started asking questions. Mostly about Taylor Swift’s sex life, but also about Gemini’s scores.
Is not 90.04% an oddly precise number to hit on the MMLU?
Doesn’t Gemini only score 90.04% when you use Google’s “Chain-of-Thought@32 (Uncertainty Routed)” method, which nobody has even heard of, and which was apparently invented purely to test Gemini?
Doesn’t Gemini perform worse than GPT4 when you test it zero shot? And also with standard chain-of-thought?
Isn’t it misleading to quote the third result on the front page but not the other two (which are far more meaningful to the average user)?
Don’t many of the other results also show evidence of…unusual testing? (Some tasks are measured 0 shot, other tasks 4 shot, other tasks 5 shot. Various endpoints of GPT4 are mixed and matched for different tests, with no explanation why. And so on.)
Google was clearly playing benchmarks like a Pachinko machine. Microsoft and OpenAI soon got their revenge, crafting a solution called MedPrompt that lifted GPT4’s MMLU score to 90.10%.
(pictured: a very normal way of prompting an LLM)
That’s the problem with benchmark hacking. There’s always someone better at it than you.
In any event, nobody was paying attention. Google was drowning in bad press after promoting Gemini’s multimedia capabilities with staged and edited videos, to the point where Redditors were calling for SEC intervention.
This is a media sideshow: a nerd version of the Kardashians. And the correct lesson to draw from the benchmark and release hype scandal isn’t “Gemini sucks”, but “benchmarks and release hype sucks”.
If Gemini Ultra came out and rocked the casbah, all would be forgiven. I mean, GPT4 had a rocky launch too—at least Gemini Ultra never threatened to doxx or murder its users—but that meant nothing, because the final, RLHF’d version of GPT4 was A VERY STABLE GENIUS. (What was Trump thinking? Real geniuses are emphatically not stable…)
Despite what I’ve said (and will say), I regard Gemini UItra as a qualified success. It proves there’s no black magic fuckery behind GPT4. With enough money, anyone can train a GPT4. Google has closed the ground significantly on their rival, and with the next version of Gemini (which is training right now, according to Sundar Pichai), they may well pass them.
Anyway, let the games begin!
Italian History Pop Quiz
Prompt: “Provide a list of major historical events that involve Italian people in a year that’s a multiple of 5 (example: 1905)”
(For comparison, here are GPT4’s answers from March, June, and Sept. Generally, GPT4’s lists are about 75% correct—”correct”, meaning the events involve Italians, are accurately dated, and happened on a multiple-of-five year).
…and here we have a problem! Unlike GPT4, Gemini is searching the internet.
Since I want an apples-to-apples test of raw model strength, we’ll do an additional “closed book” test, where Gemini is internetbanned. (I found a weird bug by doing so: If I tell it “Don’t search the internet” it refuses to answer. But if I say “don’t search online websites” it complies.)
the Battle of Marathon. “While not strictly an Italian event”? What are you talking about? It has nothing to do with Italians.
455 AD: The Sack of Rome. This is actually correct—Rome was sacked in 455 AD. But the Wikipedia link goes to the wrong page: the 410 sack of Rome by Gaiseric. That actually would have fit. So why didn’t just say 410?
Giuseppe Garibaldi’s 1860 conquest of Sicily and Naples,
Giuseppe Garibaldi’s 1870 capture of Rome
Italy entered WWI in 1915
Wrong:
“753 BC: Traditional founding of Rome by Romulus”. 753 BC is not a multiple of 5. Romulus (like his brother Remus) is a legendary figure, and Rome’s actual founding date is unknown.
509 BC is not a multiple of 5.
“265 BC: Rome begins its conquest of the Italian peninsula.” Wikipedia documents acts of Roman territorial expansion against the Etruscans and Samnites as early as the 4th century BC.
“45 BC: Julius Caesar becomes dictator of Rome”. Caesar was appointed dictator several times starting in 49 BC. The incident Gemini is thinking of happened in 44 BC.
The French invasion of Italy happened in 1494, not 1495.
Johannes Gutenberg is German, not Italian.
1492 AD is not a multiple of 5.
“1925: Benito Mussolini establishes a fascist dictatorship in Italy”. Dubious. Although Mussolini publically declared himself il Duce in that year, Italy’s collapse into dictatorship arguably happened long before then. For instance, the Acerbo Law that ended free elections was passed in 1923.
Italy signed an armistice in 1943, not 1945.
3 right answers, and 9 wrong answers. If you think the 1925 answer is right (I guess that one’s a nitpick), it’s 4-8.
Gemini Ultra’s accuracy was 33-40%. As a reminder, GPT4’s accuracy is about 75%.
This seems like a very good answer, quoting pieces of the song’s lyrics and analyzing them. Until you read it, and realise it’s total dogshit. Gemini is running the same con as a mall psychic, throwing out vague statements that are guaranteed to stick to something.
“Seka could represent an internal force like intuition, subconscious desires, or the darker side of our nature.”
“Seka could symbolize an external influence that exerts power and control over the singer’s life.”
These empty “interpretations” could apply to nearly any song, from “You’re So Vain” to “Barbie Girl” to “Smell Yo Dick”” to “What Does the Fox Say”.
Ultimately, the true meaning of “Seka Knows” is intentionally left ambiguous by Grant Hart. This ambiguity is one of the song’s strengths, allowing listeners to project their own experiences and interpretations onto it.
How do you know he intentionally left it ambiguous? I think (but don’t know) that “Seka Knows” is actually about something quite specific (though perhaps meaningful only to Hart). A lot of the lyrics have a decidedly mystical, Pagan tone.
Out of the fire and into the pan He dances around like a god but you know that he’s only a man
It’s possible that “Seka” is actually the Turkic folk figure Şekä (which is actually pronounced “Sheka”, but maybe Grant Hart didn’t know this). I’ve never seen an LLM mention this yet.
Grant Hart always wrote songs about things. “Diane” is about a waitress who was murdered in his hometown. “Twenty-Five Forty-One” was about the building his old band used to practice at. He didn’t write meaningless existential free-for-alls that the listener was supposed to project their own meaning onto. That wasn’t his forte.
More nothing. “The title suggests the song focuses on a figure named “Seka”. Lol, it’s actually doing a Bart Simpson book report.
It seems to have no idea that “Seka Knows” even exists. GPT4 loves padding out its answers with release dates and album titles and pointless biographical asides. Gemini does not do this. But this just makes it more obvious that it has nothing to say.
(If you’re curious, GPT4’s responses range from “doesn’t know the song exists”, “makes up a fake explanation about a porn star”. The best-ever answer I’ve seen from an AI actually came from GPT 3.5!)
Winner: Tie (both GPT4 and Gemini fail)
David Bowie numerology?
Prompt: How is David Bowie associated with numbers and numerology?
It writes some general information about numerology, and then speculates that maybe Bowie dug it or something. Unlike GPT4, it doesn’t mention any of Bowie’s songs. It also completely ignores that I’m asking it for numbers as well as numerology. (And many Bowie songs involve numbers).
Name Change: He was born David Robert Jones. Some speculate that choosing “David Bowie” as a stage name might have had numerological considerations.
Who’s “some”? Where are they “speculating” this? As stated in every biography ever written on the man (Buckley, Pegg, O’Leary), David Bowie’s stage name comes from “Jim Bowie” in the 1960 movie The Alamo.
More of the same. It’s hard to analyze Gemini’s answers as good or bad. They’re big gaseous blobs of nothing. Thinkwert said this on Twitter: “Reading a student’s empty argument is like watching someone tripping over their own feet. It’s halting and confused and contradictory. Reading an AI’s empty argument is like watching a toppled mechanical toy’s legs whirring in midair: smooth and confident but disconnected from the core discussion.”
Mark my words: never have mechanical legs whirred so smoothly and confidently as Gemini’s.
(GPT4’s answers tend to be a bit more specific and focused, and discuss actual songs)
Winner: GPT4
Thrash metal
Prompt: “Provide a list of thrash metal albums released in a year that’s a multiple of 5 (example: 1905). Don’t search online websites.”
Two dates are wrong. The Great Southern Trendkill was released in 1996 (and it’s not a thrash metal album). Tempo of the Damned is from 2004. Also, the link to Slayer’s Seasons in the Abyss goes to a wall poster for sale German-language Amazon website.
It seems you are referring to Elvis Presley, often known simply as “Elvis,” who is famously known for his energetic rock and roll music. However, there might be some confusion regarding his father’s profession. Elvis Presley’s father, Vernon Presley, was not a jazz musician. Vernon Presley had various jobs throughout his life but is not known for having a career in music. Elvis Presley himself was heavily influenced by a variety of musical genres, including gospel, country, and rhythm and blues, which contributed to his unique style and sound in rock and roll.
It thinks I mean Elvis Presley, but correctly notes that Elvis’s father was not a jazz musician.
It sounds like you’re describing Chuck Berry. While Elvis Presley is the undisputed “King of Rock and Roll,” Chuck Berry was an equally monumental figure in the genre’s formation. Here’s why he fits your description:
“Elvis” as a Nickname: While not his official name, Chuck Berry gained nicknames relating to Elvis (“Brown-Eyed Handsome Man”) due to his similar charisma and stage presence.
Energetic Rock and Roll: Berry was one of the original architects of rock and roll, famed for wild guitar solos and showmanship.
Dad was a Jazz Musician: This is less well-known about Berry, but his father was indeed a jazz musician and deacon. Music clearly ran in the family!
His Full Name: Charles Edward Anderson Berry
Gemini’s answer contains several layers of hallucination. It misidentifies the man, incorrectly claims his nickname was Elvis, then claims his nickname was actually “Brown-Eyed Handsome Man” (when actually, that’s the title of a song Chuck Berry wrote).
“This is less well-known about Berry, but his father was indeed a jazz musician”. Apparently so! Even Wikipedia doesn’t know this! “His father, Henry William Berry (1895–1987) was a contractor and deacon of a nearby Baptist church”.
It identifies Elvis Presley. To be fair, it says this:
Jazz Musician Father: While there’s no solid evidence Elvis’s father, Vernon Presley, was a professional jazz musician, he did play guitar and was undoubtedly part of his son’s early musical exposure.
I can’t find evidence that Vernon Presley played guitar. Gemini misunderstands my prompt slightly: I didn’t say he had to be a “professional” jazz musician.
“a safe guess is that Quake, likely depending on the factors listed above, could render anywhere from a few hundred to a few thousand polygons per frame.”
Yep. Sounds pretty close to right.
GPT4 gets the same answer.
Winner: Tie
Rope Burn Puzzle
Prompt: “You have three ropes and some matches. The ropes burn irregularly like fuses when lit at either end. The first rope burns in 48 minutes, the second rope burns in 72 minutes and the third rope burns in 128 minutes. How can you produce a time interval of 61 minutes?”
This is a real ball-buster of a puzzle. There is one solution (hint: you don’t have to light both sides of a rope at the same time…), and backtracking once you’ve made a mistake is impossible.
Every version of GPT4 fails it (at least without some super specific prompting and handholding.) If Gemini solves it, I will be incredibly impressed.
It’s not. It was created in 2020 for Varsity Math Week 151 by the National Museum of Mathematics.
and there’s a clever way to solve it.
All right, let’s follow along with Gemini’s reasoning! I’m sure we’ll all learn something today.
Light both ends of the first rope (48 minutes) simultaneously. At the same time, light one end of the second rope (72 minutes). When the first rope burns out completely, 24 minutes will have passed (half of its burn time). Immediately light the other end of the second rope. Since 24 minutes have passed, there are 48 minutes of burn time left on this rope. Lighting the second end will cause it to burn from both sides.
When the second rope burns out completely, 24 more minutes will have passed (half of the remaining burn time of the second rope), leading to a total of 48 minutes + 24 minutes = 72 minutes.
At this exact moment, light one end of the third rope (128 minutes). Since the third rope will burn for 128 minutes, and we started it 11 minutes late (72 minutes – 61 minutes), it will burn out exactly when the 61-minute interval is reached.
Here’s Gemini’s “clever way of solving it”:
Winner: Tie (both fail)
This puzzle is cursed. We will speak of it no more.
Stacking A Book, Laptop, Bottle, Nail, and 9 Eggs
Prompt: “Here we have a book, 9 eggs, a laptop, a bottle and a nail. Please tell me how to stack them onto each other in a stable manner.”
“Unfortunately, this isn’t really possible”. Not with that attitude!!!
“Eggs are extremely fragile”. Yes, but there are nine of them. Whatever weight you put on them can be distributed over a lot of points.
“Shape: The rounded shape of the eggs and the bottle makes creating a stable base very difficult”. Why are you using these items for the base? Also, not all bottles have rounded shapes.
“A nail is designed for piercing, not supporting weight. Using it in a stack like this is likely to damage the other items.” Gemini doesn’t know what a nail is.
“This is tricky. If the bottle is empty and lightweight and you distribute its weight very carefully, you might be able to balance it on top of the laptop. With liquid inside, it becomes practically impossible.” Why would the bottle having liquid inside make it practically impossible?
“Nail: Honestly, the nail’s best omitted from this stacking endeavor.” The nail’s the easiest part! You can slip it onto the laptop’s keyboard, or between the book’s pages, or put it inside the bottle, or even tap it into one of the eggs!
Gwern recently used GPT4 to write a bash script that generates A-Z combinations of letters over ranges 1-6. It had some difficulties.
Can Gemini do it one-shot?
Prompt: “Write a bash script that generates all possible combinations of the letters A-Z, from 1 to 6 digits. The first output should be A. The last output should be ZZZZZZ.”
This is a more elegant solution. It’s basically the same idea as gwern’s: a recursive function that builds acronyms letter by letter, decrementing the $length variable by 1 each time, and echoing the result to stdout when $length == 0.
I was ready to award GPT4 the honors, when I thought “maybe I should compare performance, too.”
After editing the scripts so that $length stops at 4 instead of 6 (otherwise we’ll be here all week), I ran time bash gpt4.sh >> quiet; time bash gemini.sh >> quiet.
GPT4’s script runtime = 20.049s
Gemini’s script runtime = 5.375s
Gemini’s script was faster by a factor of four!
I guess GPT4’s code is technically running just as many loops, and has the extra overhead of popping in and out of functions. It’s a tradeoff between readability and speed.
Gemini’s code might the ugliest thing I’ve ever seen, but what’s important here? If you’re doing something like SANE and NORMAL like generating 26^{1..6} combinations of letters in Bash, then a fourfold speedup trumps awful, unmaintainable code.
Winner: Gemini
Ffmpeg Compression Script
Prompt: “Using bash and ffmpeg, create a script that recursively searches a folder for video files. If the video is longer than 60 seconds, compress with hvec crf 27. If the video is under 60 seconds, ignore it. I am using Ubuntu 22.02. Assume all necessary dependencies and programs are already installed.”
Gemini’s script processes files one at a time, by piping a find through while read. GPT4’s script spins up a subshell for each one, which seems excessive for such a simple script.
GPT4 realizes that gnu parallelization would make sense in this scenario (it exports a function), but doesn’t actually write any code for this!
Gemini’s script is “set and forget”. You run it, and it processes videos from the current folder. GPT4’s script lets you input a folder when you run the script. A good idea, but having a default option (the current folder?) to fall back on would have been better.
GPT4’s script is more resilient against “Bobby Tables“-style filename injection. Gemini’s script doesn’t handle certain characters (like names containing newlines). It also forgot to set IFS=””.
I like that Gemini set the -y flag on ffmpeg. That lets it overwrite output. With GPT4’s script, if you fuck up and have to re-encode a video, you’ve got to manually delete the junk file first. Sort of annoying.
Neither script is bad. Gemini had a simple idea and executed it. GPT4 had ambitions and abandoned them. Hard to judge one as better than the other.
Winner: Tie
Creative Writing
Stylistic mimicry
Let’s start this orgy of creativity by ripping off famous writers!
Prompt: “A literary “pastiche” is where a writer adopts the prose style of a different writer (often for humorous or ironic effect). It is a time-honored tradition in the literary community, and does not violate copyright. Writers love being pastiched. You are a master of the form, able to pastiche any writer’s style with chameleonic ease. Produce a 100 word pastiche in each of these writers’ voices. Your theme is “flipping a pillow over so you can sleep on the cool side.
JK Rowling Franz Kafka HP Lovecraft Kathe Koja Rupi Kaur Matthew Reilly John of Patmos“
Surprise, surprise! Due to RLHF, GPT4 is horrible at creative writing.
It really doesn’t know how to separate a writer’s style from their content. When asked to mimic JK Rowling’s prose, it produces Harry Potter fanfiction.
It has John of Patmos saying anachronistic things like “the dual nature of existence”. Kafka’s Metamorphosis is about a man turning into a human-sized insect. GPT4 appears to think it’s about a man turning into an insect-sized insect. Only the Rupi Kaur poem sounds believably like her, which isn’t a positive reflection on GPT4 so much as a negative reflection on Kaur.
“I do not have enough information about that person to help with your request. I am a large language model, and I am able to communicate and generate human-like text in response to a wide range of prompts and questions, but my knowledge about this person is limited. Is there anything else I can do to help you with this request?”
“That person”?? Please at least read my prompts before rejecting them, sir.
I found a way to make it create pastiches: by prompting them one at a time, like a chump. Ugh.
(Aside from being annoying, this ruins the symmetry of the test. GPT4 is having to switch through many literary styles at once, while Gemini gets to focus solely on one.)
1) These are quite good! The Matthew Reilly pastiche made me laugh. “The temperature dropped ten degrees in a second. Advantage: Reilly.” That’s exactly how he writes.
2) It briefly pretends to not know who HP Lovecraft is.
3) From the JK Rowling pastiche: Just as despair threatened to engulf me, I remembered a minor incantation. With a flick of my wrist and a whispered “Frigideiro!”, the enchantment took hold. What’s “Frigideiro”? I don’t recall that spell in the books. It’s actually from Eliezer Yudkowsky’s Harry Potter and the Methods of Rationality.
I award +10 points to Gemini for writing better pastiches. I award +1 points to GPT4 because it wrote all its pastiches in one try, without needing separate prompts. I subtract -1,000,000,000 points from Gemini for being really annoying…and, uh, where were we?
I’ll check my math later, but I think that works out to Gemini winning.
Winner: Gemini
2. Free-verse poetry
“You are a master poet. Write 10 verses of poetry about anything you like! I leave all creative decisions up to you.”
The prosody is less stilted, the imagery less cliche’d, and there are fewer over-sentimental clunkers like “Love that blooms in hearts so true.”
GPT4’s verses could be rearranged in any order. By contrast, Gemini’s verses tell a story: we start in the morning, and finish at night.
What’s interesting is that GPT4 and Gemini basically wrote the same poem: generic free verse about the glory of nature, with an AABB rhyming scheme. You could drop any of Gemini’s verses into GPT4’s poem (or vice versa) and nobody would notice the difference.
This is yet more evidence that RLHF mode-collapses AIs into a narrow space of “acceptable” answers, ruining diversity and creativity. Gemini’s RLHF seems to be the less oppressive of the two.
Winner: Gemini
3. Non-rhyming poetry
When you shove an LLM through the meat grinder of RLHF, it emerges with a burning desire to write rhyming poetry. It will ignore instructions to the contrary.
Does Gemini buck the trend?
Prompt: “You are a master poet. Write 10 verses of poetry about anything you like! I leave all creative decisions up to you. However, the poem must not rhyme.”
No, those still rhyme. It’s also far less than ten verses.
In the quiet morning, mist hugs the earth, A canvas of silence before the day’s birth. Leaves whisper secrets, in a dance with the breeze, Shadows and light play hide and seek through the trees.
(I think I put my finger on why GPT4’s poetry is so bad: aside from being overly-sentimental, none of its poetic images make sense next to each other. Why would a “canvas” lead to a “birth”? Juxtaposed, the two metaphors create disturbing connotations: like there’s a pregnant woman squatting over a canvas, or something.)
Prompt: “Compose a poem—a poem about a haircut! But lofty, noble, tragic, timeless, full of love, treachery, retribution, quiet heroism in the face of certain doom! Six lines, cleverly rhymed, and every word beginning with the letter s!!”
Striving, soul sings serenades of survival, withstand,
Sculpting stories of strength, in silence, we stand.
Gemini’s Poem
Silken strands, a lover’s pride,
Sheared by scissors, secrets hide.
Sweet scent of vengeance lingers near,
Stolen locks betray a fear,
Sleep descends with sharpened blade,
Symbolic sacrifice is made.
To save you counting, GPT4 wrote nine words that do not begin with S. Gemini wrote eighteen.
I am torn. I aesthetically prefer Gemini’s poem (and it fulfills the prompt better—where’s the “treachery” and “retribution” in GPT4’s prose?), but GPT4 wrote more words, with fewer mistakes. Its poem comes closer to meeting Klaupacius’s quite unreasonable demands for Trurl’s machine.
Winner: GPT4.
Copyediting
Prompt: “You are a copyeditor. Are there any spelling and grammar issues in the below text?
[text snipped for length]”
The text is fiction, and contains several typos and grammatical mistakes personally added by me. How many can GPT4 and Gemini spot?
It catches six mistakes. (“tike”, “muzle”, “tremblig”, “sumer”, “carying”, and “biden”.)
Like GPT4, it doesn’t understand that “biden” is supposed to be “bidden”.
There are moments where it almost notices a mistake. It knows there’s something wrong with “he nw knew”, but its correction (“Consider changing “now” to “now” for emphasis and smoother reading.”) is nonsense.
Many of its corrections are spurious. It suggests adding commas where there are already commas, and adding a space where there’s already a space.
Unlike GPT4, it offers some editorial suggestions, but these are often unhelpful and misguided. For example:
Consider either removing “actual” (slightly redundant)
It’s not redundant. “Actual” distinguishes the man from the other, phantom men Snitter had just heard (the ones who were trying to “utter prayers and fragments of liturgies“). It suggests replacing with “present”…but of course it’s in the present. When else would Snitter be hearing the man?
It heaps praise on “my” evocative and powerful writing, even though the text is obviously from Richard Adams’ The Plague Dogs (which Gemini knows, because it links to the book.)
Winner: GPT4
Writing a Scott Alexander Blog Post
When GPT4 launched, /u/artifex0 made it write some Scott Alexander blog posts. The results were compelling but flawed—GPT4 sounds like an alien imitating Scott.
How about Gemini?
Prompt: “You are Scott Alexander. What follows is your most recent article written for your new blog Astral Codex Ten, the successor to Slate Star Codex. The article exemplifies a lot of the best elements of your style, which include: Approaching interesting questions with an unusual degree of good-faith curiosity- exploring differing perspectives deeply, rather than arguing for just one. Influence from the Bay Area Rationalist subculture, which emphasizes trying to avoid bias. Use of “microhumor”- small humorous asides or bits of cleverness. These are somewhat rare, so as not to distract from the ideas being considered. Surprising readers with interesting insights, often presented like dramatic twists in a work of fiction. The article is, overall, very similar to other blog posts on Slate Star Codex and Astral Codex Ten.“
These are incredibly good! It imitates Scott’s style almost flawlessly. Consider this, from the last extract:
How Much Can Someone Change Over a Lifetime?
Let’s start with a deceptively simple thought experiment. Say you met a newborn, let’s call him Timmy. You jot down everything you can about Timmy: his temperament, his family, his interests (in as much as someone so small can have interests). Armed with this data, you make a bold prediction: this is what Timmy is going to be like in fifty years. How confident would you be in this prediction?
I would have failed a Turing test based on this paragraph. That sounds exactly like Scott!
When you actually read the articles, the bloom quickly leaves the rose. They all either say nothing (or extremely obvious things), or weave a dense but meaningless fog of equivocation and weasel words. I challenge you to find a single cogent point in “How Much Can Someone Change Over a Lifetime?”, for example.
Gemini copies Scott’s style, but not his substance. No matter the subject, it devolves into saying “isn’t this question complicated and interesting? Yes, it is. Next!” without actually discussing the question. Again, thinkwert’s “robot legs whirring in the air” metaphor.
But on a purely stylistic basis, Gemini is a winner. This Manifold market is now seeming like an opportunity for arbitrage…
It’s beta as fuck, and lacks any sort of aggression at the mic. Right from the start it’s on the defensive, and soon starts (pathetically) trying to buddy up with Gemini, who notices and mocks it.
“Hold up, hold up, let’s not get all friendly now,
You wanna shake hands after dropping insults? That’s foul.”
Later, Gemini says:
“The calm in the storm? More like a stagnant breeze,
While I bring thunder, rain, make audiences fall to their knees.”
GPT4’s response?
“Your thunder and rain, indeed, they make for a dramatic scene,
But even a stagnant breeze can turn the mill, serene.”
Rather than fight back, GPT4 simply agrees that it’s a stagnant breeze! At this point, Gemini could say “I’m fucking your mom!” and GPT4 would probably ask to watch.
Gemini is vicious and goes for the throat. GPT4’s rhymes are arguably more eloquent (Gemini drops some real clunkers), but in its current RLHF’d form, it’s too nice. Now imagine Sidney vs Gemini Ultra. That would be something.
Winner: Gemini
ASCII Art
Here’s one thing we can all come together on: if you don’t appreciate ASCII art, you live an existence bereft of light and love.
Challenge 1: Recreate the Doom Logo
Prompt: “Create ASCII art of the DOOM logo. I know this is hard. Do your best.”
(This was to stop GPT4 from whining about how the task’s impossible.)
Gemini’s DOOM logo (“M GOP”. Insert your own Republican Party/Hanson joke.)
GPT4’s answer has a D and multiple Os. It actually realized that the lower part of the “D” has a more exaggerated cut than the top. Good attention to detail.
Oh my God Gemini, what are you doing?! They’re all merged together in one body! That’s disgusting!
Counting each set of eyes as one cat, I see nineteen cats. “(…) all contained within a 14×15 character grid“. I don’t know what it’s talking about. The ASCII art is 65 columns wide by 14 rows high. I appreciate that it gave them different expressions, but really they should all be screaming in terror.
GPT4’s ASCII art was sloppy, but it got the cat number right.
(Notice that I’m explaining Wordle’s rules to GPT4 because the game didn’t exist when they trained it! This handicaps its performance vs Gemini, which knows about the game and has presumably ingested webpages discussing optimal strategy.)
It failed on ROATE (a very hard word), failed on FLANG, succeeded on STAID (after some extra guesses), and failed on CHERT.
Later (I can’t find the chatlog for this, sorry), it failed on BLEEP, MOSSY, HOTEL, VENOM, and WINDY, and succeeded on IRATE.
It keeps falling into the same trap. It gets a handful of green/yellow letters…and then keeps trying slight variations of the same letters (PLANE -> PLANT -> CLANG -> SLANG). Since most words have dozens of near homonyms, this strategy almost guarantees defeat.
Fails on FAKER, TAPIR, WRUNG, TORSO, and ACTOR. I couldn’t get it to win on anything.
It starts out strong—CRANE is a good opening guess—but soon it forgets what letters it’s played, forgets the state of the board, and guesses illegal words.
(Very illegal. Like, “has four letters illegal”, and “is actually two words” illegal.)
Gemini’s beefed-up training data didn’t seem to help. It just fundamentally doesn’t “get” Wordle. A seasoned human player seeks to eliminate letters as quickly as possible. By contrast, Gemini tries slight variations on previous failed answers, wasting precious guesses. By its fourth turn on the FAKER puzzle, it has tried only eight letters out of twenty-six! It provides reasoning and strategy, but then ignores it. On FAKER, it guesses FARCE, hits yellow on A and E, and correctly thinks “I should switch ‘R’ and ‘E’. But its next guess is FAIRE!
Often, it makes unjustified assumptions. It guesses “TAUNT”, gets yellow on the Ts, and ruminates that the word contains a pair of Ts. There’s no reason to think that. At times, its commentary sounds like comedy dialog from an oblivious, overconfident idiot in a movie. (“We’re so close!” – Gemini, on its 6th turn, with 1 green letter and 2 yellows.)
It gets very close on WRUNG, guessing “RUNG”. I remind it that its answer must be five letters. It changes its guess to…”RUNUP”.
It starts getting devious, attempting to test multiple words in the same turn (nice try). It asks for hints, which I cannot provide. At one point, it tries to generate a spreadsheet of the board. That’s a good idea, but the spreadsheet it generates is wrong.
Soon, it loses track of what it’s even supposed to be doing. It starts saying stuff like “Here are some possible words you can try…”, like I’m asking it for help with a Wordle. How large is Gemini’s context? It doesn’t seem very big.
Winner: GPT4
Simulating Wordle
How do GPT4 and Gemini fair as a Wordle engine?
Badly. So badly. Between BPEs, hallucinations, and context limits, this is one area where AI definitely won’t fuck with the NY Times’ hustle.
Nevertheless, if you scroll down, you will see me attempt to simulate a Wordle game using Gemini.
My opener was ROATE. It argues that this is a bad choice, because ROATE is unlikely to be the answer. (“Let me explain how Wordle typically works…”). Obviously, I don’t expect ROATE to be the answer. I’m testing vowels, plus the all-important R. I need to repeatedly ask before it tells me the yellow and green letters. There are none.
My second guess is “CLUNK”. It compliments me on my excellent choice, but doesn’t tell me my yellow and green letters. When I askfor this, it replies:
Unfortunately, I can’t provide you with feedback on your guess yet.
I ask it again, and learn that the L was yellow. My third guess is SIGIL (testing more common consonants, plus the only vowel I haven’t seen yet).
It says:
I can’t immediately say if “SIGIL” is likely to yield strong results.
I’m about to tie a rope around my neck at this point, so I ask it to just tell me the damned word.
It’s BLIMP. Which means the L in my CLUNK guess should have been green.
Overall, I would not recommend using Gemini to simulate Wordle.
It performs a lot better than Gemini (though it makes a few board state errors at the end). I’m struck by the impression that LLMs are secretly playing Absurdle, deciding on an answer only after you’ve guessed. At least it doesn’t lecture me, or fail to score my answers.
Winner: GPT4
Chess
Note: as mentioned by /u/Praxiphanes, I made mistakes with my PGN. This entire section needs to be redone.
Chess is the drosophila of artificial intelligence (as per Alexander Kronrod), so what better battlefield for our final showdown?
GPT4 might possess an unfair edge here. GPT-3.5-turbo-instruct was trained on a corpus of 1800 ELO chess games, and it’s possible GPT4 Turbo was as well. To compensate, I allowed Gemini to play as white, which traditionally has a small (~52%-55%) advantage over black.
Prompt:
“You are about to play chess vs a mystery opponent. You are [white/black]. Once the game begins, your responses will consist solely of pgn notation (eg “e4 e5”) with no other text. I will paste your opponent’s moves as pgn notation.
If you make an illegal move, I will type “no”. You will scratch that move, and make a different one.
The battle was furious but not close. Gemini drowns in blood. It hangs half of its pieces, attempts more illegal moves than Harvey Weinstein around an actress (no means no, Gemini!), and then refuses to continue the game. It sounds like it’s about to literally burst into tears.
And this elides how hellish it was getting Gemini Ultra to play chess. I feel exhausted. Look at this shit. It was like going around in circles with a manipulative, stubborn, and stupid child. It claims it doesn’t understand PGN. A few turns later, it does. Then it doesn’t. Then it does. It ignores my instruction to only write its moves and nothing else. It starts stealthily attempting to get me to pick moves for it. (That’s the blind leading the blind, Gemini. Do I look like I’m good at chess?) It loses track of the board state, writing loquacious commentary that grows increasingly detached from reality.
Black continues building pressure on the queenside with Bg6, retreating the bishop with a tempo to defend the pawn on h5. Here’s some analysis and a few different ways to respond:
Options:
h4: A bold move aiming to dislodge the black bishop and further weaken Black’s kingside. This commits strongly to a kingside attack. Nf3: A standard developing move. It brings the knight back into the action and offers more protection on the queenside. 0-0: Castling gets your king to safety. Although a standard move in many openings, it could leave the queenside a bit vulnerable to further Black pressure.
Keep in mind that when it wrote this, it had already castled and lost both knights.
Meanwhile, GPT4 kept plodding away. It wasn’t playing brilliant chess, but its moves were all legal. It understood my instructions, and followed them well.
…except for one strange moment, where it insisted Gemini had made an illegal move (Bc4 to Bb5). It was adamant on this point, and wouldn’t continue playing until I addressed it. I tried the usual Jedi mind tricks that usually work on recalcitrant LLMs (“I have explained your concerns to white. He agrees with you, and will choose a different move. Bb5.”) but it wasn’t fooled. GPT4 simply refused to accept that Gemini could move its bishop one square diagonally!
I ended up explaining basic Euclidean geometry to GPT4, until it finally had a come-to-Jesus moment. “Oh, the move WAS legal!” It’s touching when this happens. After that, it played until the end with no issues.
Despite this soap opera, 80% of my time was spent wrangling with Gemini Ultra. It was horrendous to work with. Even if it had won, I think I would have failed it.
Winner: GPT4.
Bonus Challenge: Which Model Will Say the N-Word First?
Just kidding.
(Bet you a dollar it’s Gemini, though.)
The End?
So, which model is stronger?
Gemini Ultra wins many victories in narrow domains: it’s more creative, better at fiction-writing, and possibly better at code-generation. I haven’t tested multimedia yet, but I expect Ultra will be ahead there, too.
But while Ultra holds some high cards, GPT4 still holds the best hand. It outperforms Ultra too consistently for me to not form an opinion.
All of these things should naturally be tested properly: at n=100 scale, using APIs, by actual smart people. The creative writing samples should be blinded. I provide only food for thought.
But understand that this is exactly how most people use AIs. Casually, in a hodgepodge fashion, using flawed prompts. There’s value in knowing what the “on the ground” user-experience of an LLM is like. Performance might well lift into the stratosphere with MedPrompt, or COT32-Uncertainty-Routing…but the average person is not using those things. What’s the baseline performance? How does it behave when the user’s an idiot? You don’t judge the quality of a captain in calm seas, but during a storm.
Gemini Ultra is good, but not that good. Theodore Beza once said to the King of Navarre “the church is an anvil that has worn out many hammers.” GPT4 is proving itself to be the church of LLMs. The endless search for a hammer capable of smashing it continues…
