Update: Find out more about the 50 Years of Text Games book and the revised final version of this article!
Screen
original version by Noah Wardrip-Fruin, Josh Carroll, Robert Coover, Shawn Greenlee, and Andrew McClain
First Prototyped: Fall 2002
Language: C, G3D, Max MSP
Platform: CAVE (SGI Irix)Opening Text:
In a world of illusions, we hold ourselves in place by memories.
Though they may be but dreams of a dream, they seem at times more there than the there we daily inhabit, fixed and meaningful texts in the indecipherable flux of the world's words, so vivid at times that we feel we can almost reach out and touch them.
But memories have a way of coming apart on us, losing their certainty, and when they start to peel away, we do what we can to push them, bit by bit, back in place, fearful of losing our very selves if we lose the stories of ourselves.
But these are only minds that hold them, fragile data, softly banked. Increasingly, they rip apart, blur and tangle with one another, and swarm mockingly about us, threatening us with absence.
The words appeared from darkness. Each phrase lit up in isolation, projected white letters read aloud by an unseen narrator. Once the final words were spoken, all went dark again. Then three short stories appeared, each on one eight-foot wall of the cube that wrapped around the reader. Each told of a half-forgotten memory:
In a sheetrock bedroom, under an acrylic blanket, a woman tosses, twists, turns over into another, oak-panelled room, a bed vast and cool and loose about her. She feels speckles of sun on her cheek, filtered through the trees that tower around this house. Her shoulders sore from splitting wood, swinging the big axe overhand, thud and snap, kindling clattering. Nana smiling, proud. She must have traveled all night to pull these blankets, heavy, onto her body, find this place again. She reaches for the lampswitch, to wake before sunrise, to bring Nana breakfast, but her fingernails tap the plastic face of her alarm clock, its boxy numbers telling her the time of another continent.
Each hundred-word story was read by a different narrator, while the listener, the reader—standing in darkness, wearing a visor that tracked their head and holding a pointer that tracked their hand—turned in place from one wall to the next.
Once all three stories finished, another pause. And then, one word at a time, the stories began to come apart.
The origin of Screen—an installation piece for the immersive VR environment known as the CAVE—can be traced to a decade before its genesis, with an influential article by the novelist Robert Coover for the New York Times Book Review: called, provocatively, “The End of Books.” It was announcement, warning, manifesto, and challenge, heralding to a conservative and establishment audience the arrival of new kinds of computer-enabled literature that could never have existed on the printed page. Tools like StorySpace and a new generation of computer-literate writers were threatening to disrupt centuries-old traditions, and Coover argued this was not something to be feared, but celebrated. The coming revolution would force all writers to challenge their assumptions about what the written word could do. “Much of the novel’s alleged power is embedded in the line,” he wrote in one example:
that compulsory author-directed movement from the beginning of a sentence to its period, from the top of the page to the bottom, from the first page to the last. Of course, through print’s long history, there have been countless strategies to counter the line’s power... But true freedom from the tyranny of the line is perceived as only really possible now at last with the advent of hypertext, written and read on the computer, where the line in fact does not exist unless one invents and implants it.
Literature, in short, had the potential to grow beyond the one-dimensional stream of linear words it had always been embedded in. But grow into what? It was an open question: one, Coover proselytized, that he and any other willing pioneers were eager to explore.
The previous year Coover had begun teaching a regular hypertext writing seminar at Brown University, which had a long tradition of embracing experimental literature. Through the 1990s he spearheaded many efforts to provide institutional support for young writers looking to incorporate computers into their practice. (Among other contributions, in 1999 he helped co-found the influential Electronic Literature Organization.) While much computer writing had focused on how animations, images, and sounds could intersect with the experience of reading, Coover encouraged his students to focus on the words:
I keep asking questions about text in this space. I don’t discourage multi-media efforts, but I don’t like the letters to disappear... Probably the most interactive thing that we do, in some ways the most human thing that we do, is to stare at little squiggles of ink on a white surface and out of those invent vast worlds, landscapes, characters almost more believable than the ones surrounding us, imaginary experiences that are so rich and complete and whole that they almost at times dwarf our ordinary experience.
...Perhaps graphic artists will help us to have deep imaginings in the future, not literary artists. That’s a possibility, but I’m not yet willing to throw in the towel.
Near the end of the ’90s, Brown had acquired an expensive new piece of hardware. The CAVE—in one of those recursive acronyms techies love so much, it stood for Cave Automatic Virtual Environment—had first been proposed and built in 1992 as a technological runaround to limitations with virtual reality. By projecting images on surfaces around a user, it avoided the need for the miniaturized head-mounted displays that were still prohibitively expensive and difficult to produce. Brown’s CAVE projected images on the floor and three walls of a cube (leaving the fourth open), with multiple speakers providing positional sound. The projectors alternated quickly between images for a left and right eye; by standing in the center wearing synchronized shutter glasses, the walls of the cube would fade away and the illusion of being surrounded by three-dimensional objects and environments could be created. (Similar tech had been part of Brenda Laurel’s proposed interactive theatre experiments at Atari.) Scanners tracked the position of the glasses and thus the user’s head, to adjust the scene for their perspective; and a held “wand” could also be tracked to allow for interacting with the environment. An advantage of the CAVE setup was that the user wasn’t cut off from their own body, as in traditional VR: they could still see their own hands, feet, and position within the space, and make eye contact with spectators or assistants outside of the cube, making for a less isolating and disorienting encounter with a virtual environment.
Brown’s CAVE had been funded under a promise of interdisciplinary collaboration, but at first this had mostly been between computer engineers and the sciences. Early projects included “3D weather visualization,” “graphical planning for brain surgery,” and “interactive modeling of biological macromolecules.” Coover was intrigued at the possibility of getting access for his writing students, but was at first rebuffed: “this is strictly a scientific instrument for scientists,” the system’s managers reportedly told him. But he persisted, and soon his lit students were tentatively experimenting with what it might mean to write words designed not for the two-dimensional surface of a page, but for a three-dimensional volume. In spring 2002, at age 70, Coover led Brown’s first Cave Writing Workshop, which had what might be one of the most exciting English department course catalogue descriptions of all time:
an advanced experimental electronic writing workshop, exploring the potential of text, sound, and narrative movement in immersive three-dimensional virtual reality. It brings together teams of undergraduate and graduate fiction writers, poets and playwrights, composers and sound engineers, graphic designers, visual artists, 3D modelers and programmers, to develop, within the environment of Brown’s “Cave” in the Technology Center for Advanced Scientific Computing and Visualization, projects that focus on the word.
By 2002, hypertext was in danger of becoming so normalized as to be invisible. The rise of the world wide web had made one particular flavor of interlinked text so commonplace it was getting hard for students to realize they could be experimental with it. Coover saw the CAVE as a way of keeping digital writing productively unfamiliar, a distinct space apart from the calcifying tropes and norms of the web. And while other VR labs were chasing 3D graphics still far from photorealism, Coover saw replicating reality as the least useful thing you could do with a CAVE. “In a way, the more realistic it gets in there,” he mused, “the less interesting it is.”
The projects that emerged from that first workshop were characterized by Coover as “finger exercises,” the technically complex but artistically flat patterns musicians play to warm up or get a feel for a new instrument. They were tech demos, more concerned with the praxis of three-dimensional words than what they might have to say. “In a demonstration of a text-painting program,” a profile of the class described, “the mouse in Mr. Coover’s hand acts like an airbrush nozzle, spraying strings of letters around the room as he gestures broadly. After the letters appear in space, they form words that literally hang in midair.” Another project mapped words in reverse onto a floating 3D cube, so that the reader had to move their body and head inside it to read them. Nearly a dozen projects were started that semester, but few progressed far beyond the concept stage: “It was just too hard,” Coover remembered. The group had to “figure out how the thing worked, and so came up with a lot more ideas that first semester than finished projects.”
But some of the finger exercises sparked more complex ideas. One of these was a demo by a student named Andrew McClain, studying both computer science and theatre: he’d prototyped code to make individual words of a passage seem to peel away and flock around the reader. The project inspired another member of the workshop who was a graduate student in Brown’s fiction writing MFA, Noah Wardrip-Fruin. Noah—who recalled playing Hunt the Wumpus on a parent’s university mainframe account as a kid, and later had his imagination sparked by Infocom—had come to the program having already created a handful of experimental digital writing projects, and was eager to find like-minded collaborators, beef up his technical skills, and work with Coover. (Coincidentally, one of his first digital text projects—a poetry generator written in high school—had mashed up words from sources including a short story of Coover’s found in a literary magazine; it would be years later when he’d come across the novelist again as a proponent of digital writing.) In collaboration with Shawn Greenlee, a sound artist also in the workshop, and Josh Carroll, a coder who joined the group that fall, they began to conceptualize a more elaborate project based on the peeling-words demo: a story about disintegrating memories that readers would try to keep together themselves.
The central idea evolved out of an article Carroll had read about “how human memory works... when you remember something you don’t just play it back... you re-experience it and then re-encode it, so every time you remember something, you’re actually changing the way you remember it.” In the completed version of the piece, after the stories of fading memories surrounding the reader have been read aloud, words begin peeling off of them. First one, then two, then dozens detach from their origin points and drift forward toward the center of the cube. The reader, with a position-tracked hand, could knock words back to their original place. Sometimes, they’d comply. But increasingly the words would return to different sentences than the ones they’d left, leading to more and more distorted texts. With stories surrounding the reader on three sides, turning to focus on words drifting free from one would inevitably mean neglecting another, which would continue to disintegrate behind you. Pushing words too hard could also shatter them in two, each fragment drifting into a different gap to form neologisms or nonsense. As each word detached it was spoken aloud, and as the speed of fragmentation increased, the narration began to sound like an abstract poem:
kindling... then... a... she... gather. Shoulder. By, way, to. Give his smoothing mouth. A then sunrise, a overhand again split old of these five warming napping the rhythm...
Eventually, no matter how dexterous the reader’s attempts to knock words back into place, a critical point would arrive where all the text from the memory stories drifted free from the walls to surround the reader in a dense cloud. Over a haunting sound like real and electronic whispers mingling together, a final text written by Coover (who also wrote the introduction) was spoken but not seen:
We stare into the white void of lost memories, a loose scatter about us of what fragments remain: no sense but nonsense to be found there. If memories define us, what defines us when they’re gone? An unbearable prospect. We retrieve what we can and try again.
The piece would conclude by returning the story text to the CAVE walls, but fragmented, with only the words that had been returned to the wall—whether to their original positions or not—still visible. The reader would spend their final moments surrounded by stories filled with gaps and errors, only shadows of the original vignettes remaining.
a under acrylic woman tosses, twists, over another, room, bed and and her. let speckles sun through that this Her sore splitting swinging overhand, and kindling Nana age She must all to heavy, her place car, reaches the to before bring breakfast, her of time tap plastic face her clock, boxy telling the do another
One young visitor, confused, had asked of the final shattered story text: “Is that my score?” It was a fair question. The relationship between games, artwork, and interactivity within the context of the CAVE was a complex one, deliberately challenged by Screen’s designers. Nearly all previous CAVE projects, for instance, had used its powerful hardware to render colorful 3D objects and immersive environments, not words. And while other CAVE works went to great lengths to make the walls around the user seem to disappear, Screen’s code instead worked to create the illusion that its words—which could have been positioned anywhere—were aligned precisely with the CAVE’s walls, no matter where the reader stood. One grumpy visitor proclaimed the piece wasn’t “using the medium,” and insisted on viewing the piece from a chair outside the CAVE (until words started floating off its walls). Screen’s creators, of course, had intended to challenge these kinds of assumptions, to create “an experience that doesn’t settle easily into the usual ways of thinking about gameplay or virtual reality.” Spearheaded by Wardrip-Fruin, the project evolved over further months of iteration to refine its unique “bodily interaction with text,” an experiment in exploring “the uncanny experience of touching words.” The question of whether you were meant to treat those words like parts of a literary fiction or the ball in Breakout was deliberately left to the audience to sort out.
Coover had noted that “one of the difficulties with virtual writing” was that animated text could be hard to focus on: “when you ask afterward, ‘what were the stories about?’ not many people noticed.” But some viewers praised Screen for so directly engaging with this existential question for an embryonic medium; for making the asking of where one’s focus was meant to land be precisely the point, both on a technical level and an artistic one.
You don’t want to read the words coming up to you—you want to win. You are trying to keep all those words from getting lost. You are trying to keep them on the wall, which represents our memory or rather: the external archive, external storage. The aim of our physical effort is to return the words to this archive. The more effectively we do this the less time we find to read the words we are saving, which means we don’t refresh the words in our internal archive... [we don’t keep] them alive in our mind.
The project continued to be refined in the run-up to a planned debut as part of the Boston Cyberarts Festival in April 2003. While access to the CAVE was shared between projects and departments via a sign-up sheet for time slots, the Screen team preferred to take the day’s last available shifts so they could keep working on the project late into the night when necessary. The Silicon Graphics workstations then running the CAVE proved not powerful enough to render the finale as originally planned—two thousand invisible boxes with letters painted on their sides, floating free in a 3D space—so compromises were needed to create the illusion of all three walls of text collapsing at once. (Later versions of the piece with more powerful hardware improved the effect.) Pacing and sound design continued to be tweaked up to the last minute. A CAVE piece, of course, was destined to have only the tiniest of audiences—imagine debuting a new VR game if there were only five Oculus headsets in the world, each built from scratch with different technical specs—but the reception was positive. The piece would be shown at a handful of art festivals and be updated and restaged several times during the rest of the decade.
Despite its small viewership, Screen’s creators saw its development as a useful experiment. “We’re discovering a lot of things about text, and unexpected ways of interacting with text in this environment,” Carroll wrote, “and that could have significant repercussions in other mediums and other applications.” Coover recalled: “We were asking the simple question: Does literature have a future in this space and if it does, how can we enhance it?” But the question, at least for the next twenty years, would remain largely unanswered: spatialized text has remained an uncommon oddity. Brown students and faculty kept experimenting with CAVE writing—among other outcomes, John Cayley’s influential essay “Writing on Complex Surfaces” was born from his experiences there—but even today when VR headsets have become affordable and methods of authoring and distribution much more widespread, only a handful of 3D text projects can be found. Searches for “text” or “poetry” on the Oculus Store in 2021 produced no relevant results. Perhaps this is okay. It takes a lot of sparks to find the ones that light new fires.
“Writing students are notoriously conservative creatures,” Coover had written in “The End Of Books”:
They write stubbornly and hopefully within the tradition of what they have read. ...But confronted with hyperspace, they have no choice: all the comforting structures have been erased. It’s improvise or go home. Some frantically rebuild those old structures, some just get lost and drift out of sight, most leap in fearlessly without even asking how deep it is (infinitely deep) and admit, even as they paddle for dear life, that this new arena is indeed an exciting, provocative if frequently frustrating medium for the creation of new narratives, a potentially revolutionary space, capable, exactly as advertised, of transforming the very art of fiction...
“It’s our chance to discover written language anew,” Wardrip-Fruin wrote at the time about the CAVE workshops: “it’s all still experiments.” Though its creators largely turned their attentions to other means and practices of computational writing, the experiments continue. As Coover’s closing words for Screen had said: We retrieve what we can, and try again.
Next week: a kingdom in a browser tab, filled with bad jokes and questionable puns, visited daily by millions of desk jockeys desperate for a taste of adventure.
There’s no way to play Screen today, but you can view video documentation or a full playthrough of the original experience. Extensive coverage from the Iowa Web Review was a major source and the origin of unattributed quotations inline. Thanks to Noah Wardrip-Fruin for sharing memories and materials from his time working on the project, and for his detailed write-up in the book Expressive Processing. Noah’s latest book is How Pac-Man Eats, and you can find him on Twitter at @noahwf.
Disclosure: I’m a former graduate student of Noah’s.
