1. BREAKAGES, SYSTEMS, INTERFACES
Is it possible to break something inside a virtual environment without causing an immediate rupture in the illusion and the immersion? This is, of course, something of a rhetorical question. A quick answer might be: of course, things can be broken; however, any possible fissures have to be defined and coded well in advance of the actual breakage.
I am wondering whether there can be moments in which a virtual breakage can have the status of a breakage in the physical world, with all its attendant consequences, including elements of unpredictability and unrepeatability. What is (or what could be) the relationship of this breakage to the status of the virtual world itself? More generally, what about the whole realm of unexpected and unprogrammed interactions in virtual worlds; might alternative strategies to the usual state of affairs exist, in which every possibility has to be accounted for and pre-programmed?
This realm would contain elements of human-machine interaction that do not fit neatly into practical, predictable and instrumental accounts of technology; it might include categories like misbehavior, mistakes, mischief, tampering (both willful and unintended), uncooperativeness and the general testing of limits for whatever reason.
I am asking these questions primarily about immersive simulation technologies, that is, Virtual Reality, although I think they might be applicable to cyberspace in general. The question might be more generally stated as: What happens to a system when confronted with something that cannot be systematized? (A more fundamental question, and a scarier one, is whether there is in fact anything that is ultimately resistant to systemization and assimilation.) I think this question strikes at the heart of why our Virtual Realities often seem, in fact, so unreal. Virtual technologies are usually touted as holding a vast liberating potential; but more often than not they seem to be exactly the opposite, restricting the participant's field of action to options that are tightly scripted by the designer.
Here are a few definitions of Virtual Reality. From Pimentel and Teixeira: "More than a 'fantasy machine,' virtual reality is - the creation of a universal metalanguage - (that) allows us to share ideas and thoughts - by communicating in - the human sensory language of reality." Or Marcos Novak: "a habitat for the imagination - a landscape of rational magic." Howard Rheingold: VR puts us "on the brink of having the power of creating any experience we desire." I have a working (and only slightly tongue-in-cheek) definition of my own. VR is: something that requires endless waiting for an interminable line, only to briefly enter a rudimentary world in which one is a solitary inhabitant with nothing to do.
Interface designers generally work under the assumption that the user will be willing and able to follow instructions; the art of good interface design, of user-friendliness, involves making tasks as sensible and logical (not to say obvious) as possible. For practical applications, ones in which real work gets done, this approach would appear to make perfect sense. However, these very assumptions might be antithetical to the task of creating artificial worlds that seem in any way real. It is exactly at the moment when the user stops following instructions that the effectiveness of the illusion is assessed.
2. BRITTLE RESTRICTIONS
Most interactive virtual worlds are notoriously brittle. The moment the user stops following instructions (whether these instructions are implicit or explicit), most VR stops functioning, at least to some degree. The user might wander (virtually) off of a predetermined path, generally ending up in a black void that could best be likened to the utter blankness of a movie screen after a projection bulb has blown. Or the user might wander (physically) out of range of a position sensing system, in which case the viewpoint might freeze, or even jump back to an arbitrary default position. Multiple choices in cyberspace rarely include "none of the above"; the closest one can usually get is to make no choice at all, in which case nothing very interesting happens. And generally, much interactive work is addressed to a fixed (or ideal number) of interactors, often one or two. When this limit is exceeded, the interactive work often begins to malfunction, sometimes ceasing to work altogether. Conversely, there are pieces of work that require a minimal number of players to function well, or at all; below this threshold, there is nothing to do.
So how much must we restrict our actions inside these artificial realities? And if these worlds are in fact as fragile as they appear, might there be some way to make them more pliable, more resilient? In my work Faraday's Garden, participants walk through a landscape of innumerable household and office appliances, power tools, projectors, radios, record players, and various other personal comfort devices. The floor of the room is carpeted with switch matting, a pressure-sensitive covering designed for security systems. When stepped on, the switch matting triggers the various machines and appliances, creating a kind of force field of noise and activity around each viewer. As the number of participants increases, the general level of cacophony rises, creating a wildly complex symphony of machines, sounds and projections.
In 1992, I presented the performance of a work called Runway based on Faraday's Garden at the Whitney Museum's midtown branch in New York. The audience for the work stood facing a stage area where four performers manipulated the various appliances and machines. The audience could step onto a hundred-foot-long path of switch matting, thus triggering off various machines. The idea was that the audience would have control over when the machines turned on and off, and the performers would determine what the machines would do when they were on. However, the audience that showed up was much larger than expected; and in the rush to get in, not a single member of the audience read the instructions printed in the program. As a result, the entire audience spent the entire performance standing on the switch matting, which kept every single machine on simultaneously and continuously. What was meant to be a complex interplay between audience, performers and machines instead became a single sustained machinic roar, punctuated by sudden moments of dramatic silence and stillness as the circuit breakers were repeatedly tripped and reset. This may well have been an improvement over my original concept.
3. ASSIMILATION AND CONTRADICTION
The formulation of virtual reality assumes an assimilation of the user's experience with the encompassing domain of the system. Whether we refer to the present state of existing VR apparatus, or to an "ideal" VR which has not yet been attained due to various limitations of today's technology, the objective is the same: any user data that can be monitored and tracked will be, and any sensory channel that can be addressed will be. The corollary to this is that any aspect, either input or output, that can't be accounted for is necessarily ignored. Thus, if we have no means of monitoring the "other" hand (the one that is not wearing the data glove), that hand doesn't exist in VR; and if we can't provide any sort of tactile feedback, then the virtual world will be made of substances that offer no resistance or solidity.
Of course everything that is ignored in the virtual world continues to exist in the actual one; we still have two hands, and our muscles and nerves continue to function. Because VR can't offer any account of the elements it can't assimilate, our experience is therefore doubled and schizophrenically split; rather than entering an ethereal dream world, a so-called "habitat of the imagination," our sensorium is split along an arbitrary fault line. Far from leaving our bodies behind, we are simply asked to ignore the unrepresented part, which has been "amputated" by the system.
This experience was described one hundred years ago in an early short story by H.G. Wells, The Story of Davidson's Eyes. Researcher Sidney Davidson who lives in England has his view suddenly and inexplicably replaced with what eventually turns out to be the view from a remote South Seas island. He sees a beach, a ship, penguins, the sea; he observes a sea battle, played out in eerie silence. His actual location, and his other senses, remain intact; at first, unable to reconcile the contradictory sensory fields, he panics. When his colleague Bellows speaks to him, Davidson hears a disembodied voice; when Bellows touches him, he recoils in terror. Bellows first thinks Davidson has been struck blind; Davidson is convinced that he is experiencing the afterlife when he says, "I suppose we're both dead. But the rummy part is I feel just as though I still had a body."
Slowly the mystery is elaborated (if not explained). Davidson's movements in his physical location are matched precisely by changes in his view in the remote one; when he goes upstairs to his bedroom, his view rises sickeningly high into the air above the island; when he descends from the hills of his home in Hampstead Village, his view descends correspondingly from a beach into the water and then deep into the sea. Day and night are reversed (since itturns out that he is perceiving a simultaneous view of the other side of the world). Eventually his sight returns to normal, the physical world gradually blotting out the remote one.
In the postscript to the story, which takes place two years later, Davidson (fully recovered) happens to be shown a photograph of the actual ship, and finally realizes that what he experienced was not a hallucination but an instance of remote vision. Wells, writing as the narrator Bellows says: "It sets one dreaming of the oddest possibilities of intercommunication of the future, of spending an intercalary five minutes on the other side of the world, or being watched in our most secret operations by unsuspected eyes." (If there exists a more prescient and detailed description of contemporary telematic systems, I would be amazed.)
Throughout Wells' story, what is striking is how Davidson's experience continually encompasses all his senses, which are put in radical contradiction to each other. The concept of remote vision, rather than being seen as something wonderful, is presented as a kind of horror story. One question raised by this story, then, would be: what is the relationship between these two realms of experience? And for designers and artists, the next question is: is there then any way to consider the whole of one's experience in the design of virtual worlds? Must we ignore what we can't control?
But once the VR apparatus is perfected - when VR encompasses sight, sound, smell, touch, proprioception, temperature, etc. - won't this whole problem be put to rest? At that point, wouldn't we really "leave our bodies behind"? I would propose that even when all the problems are worked out, and VR actually "works," even then, our experience will still remain doubled, and that there will always be disjunctions between the inside and outside of VR.
Perhaps the paradigmatic moment for us in Wells' tale is his description of the gradual return of Davidson's sight. "It's very dim and broken in places, but I see it all the same, like a faint specter of itself. (?) It's like a hole in this infernal phantom world. (?) It looks like the ghost of a bit of your hand sticking out of the darkening sky - " This condition of overlap, which is growing stronger here until Davidson's sight returns to normal, directly prefigures the development of "augmented reality," but this condition might also accurately capture our experience of all virtual worlds.
Here I'd like to go into yet another digression and talk for a moment about the debased art of karaoke. In karaoke, any distinctions between performer and audience breaks down; anyone who wants to can become the center of attention (for a moment). The performer/participant, who until a short while ago was just another member of the audience, immerse him- or herself in a media spectacle, a kind of primitive pop cultural proto-virtual reality. Once onstage, they are free to perform the text of the karaoke any way they desire. Karaoke has both an inside - the performer immersed in the image of the music video - and an outside, the hybrid live video performance witnessed by the audience. Karaoke isn't truly interactive (so far); but it is participatory, which is probably more important. Instead of holding up a tired notion of limited multiple-choice control of the media, karaoke allows the participant to mould an existing media spectacle into something of their very own.
I think karaoke, despite what might be called its primitive and debased character, embodies certain qualities and potentials that might point us in some provocative directions in thinking about interactive virtual worlds. In karaoke, the participant, rather than being given the task of navigation or choice, is encouraged to improvise, to "riff" off of a kind of score. This spontaneous performance is conjoined with the more calculated video to create a combined spectacle that is compelling enough to hold an audience's attention.
4. DREAMING AND SLEEPWALKING
An equivalence has been suggested, many times before, between virtual reality and dreams. However, in dreams the body is immobilized. In waking life, our brain passes neural instructions to our muscles to move, the muscles obey, and our various senses report back on this movement, all in one unbroken loop. In dreams, this loop is short-circuited; the impulse to move never reaches our muscles, and we dream that we have moved while our body remains relaxed and inert. This would seem to suggest that it is cinema rather than VR that is closer to the dream experience, where we really do "leave our body behind," immobilized and passive on a cushioned seat, and that if VR is to be compared to any aspect of sleep at all, it could much better be likened to sleepwalking.
The sleepwalker inhabits one realm while continuing to move in another, very much like the person encased in VR gear. This is clearly a precarious position, one where any sense of a unified experience is constantly under threat of interruption and disruption by an incommensurate physical reality that almost by definition takes precedence over the fragilely constructed world of the dream.
So perhaps we could then describe the challenge as this: we need to figure out how to keep sleepwalking even when we bump into something. This of course would be the moment where the sleepwalker would be rudely jolted awake. This dynamic is not likely to change, even when all the problems are solved, and we finally have "complete bodies" in VR. (Perhaps a useful phenomenon to consider here would be the not-uncommon nightmare of dreaming that you have woken up, only later realizing that you are still trapped inside of the dream.)
A first step in this direction would seem to be simply to acknowledge the continued presence of all facets of our existence during any immersion into VR, and to stop insisting on the rhetoric of a pure and unified out-of-body experience. As Davidson puts it in the H.G. Wells story, "We seem to have a sort of invisible bodies." This seems somewhat right; in cyberspace, our free hand exists just as much as the data-gloved one, it's just that it's become invisible, like a kind of phantom limb.
But I think it's more complicated than this. In what has become the 'standard" VR experience (that is, an HMD and dataglove) wherever, or whatever our "body" is, it is simultaneously multiplied and fragmented in ways that are extremely hard to describe. First, there is one's experience of one's body in the material world; this is based on the sense of proprioception, which is subject to the physical and physiological laws of gravity, inner ear balance, solid matter, and so on. Then there is one's imagined body, that is, one's self-image, which often, or perhaps even usually, diverges from the physical body in various specific ways. Next, there is the body as displayed in virtual space, which produces its own specific experience, based both on representation and also on invisibility or absence. Then there is the imagined VR body, which may overlap the original self-image, but has a different range of possibilities, based on the depicted VR body, which might be largely invisible, free from the pull of gravity, etc.
All four of these bodies - and my description here is already too schematic and incomplete - overlap and affect each other; they are certainly not experienced as distinct bodies - and are further complicated by the fact that in any viewer-centered VR, or in ordinary experience for that matter, our experience of our body is put into constant flux by our changing surroundings - encountering one's image in a mirror is only the most obvious example.
To add one last layer of complexity, we have only to realize how much the equation I just presented depends entirely on the specifics of VR as experienced in a standard HMD. I would have to come up with an entirely different formulation for a CAVE, for instance, where one's physical body is not masked from vision. We might also consider here just what is at stake in immersion, in our compulsion to 'stay inside" the simulation, in spite of what might be called the "actuality cues" which are constantly pulling at us. But, for now, assuming that we really do want to stay inside: how can limits be tested in this arena without causing the entire edifice to come crashing down?
Before dealing with this question, I would like to examine a few instances where the edifice does, in fact, come crashing down, and consider these as a few of the many possible methods of virtual world destruction.
I am going to use as examples certain episodes of the celebrated comic strip "Little Nemo in Slumberland," written and drawn by Windsor McKay in the first half of this century. Little Nemo inhabits a fantastic world that is somewhere between dream and nightmare, a world that is often subjected to powerful forces of dissolution that start small but grow inexorably, leading to utter destruction and disintegration, until finally (always in the last frame), Nemo awakes.
In a strip from November 5, 1911, we look out over a fantastic and grandiose city. Flip, the self-centered, unscrupulous and mischievous instigator of chaos in most of Nemo's dreams, has managed to get behind the wheel of a steamroller. Here, its effects are limitless and far-reaching, and eventually cause the entire city to collapse like a house of cards.
In A Trip to the Island in the Sky from April 21, 1912, a banquet is taking place on an airborne island, a kind of miniature flat earth. Flip wanders away to the edge of the island and, ignoring the prominent warning sign, strolls beyond the safe zone, managing to tip the entire world over, with the resulting anticipated chaos.
And then on January 21, 1912, Flip follows Nemo into Midget City (Nemo, the dreamer, is one of the midgets); Doctor Pill, trying to stop Flip, follows. The two end up battling in a scene that would later be replayed in countless Godzilla movies, ripping up buildings to be used as projectiles; of course, in the process, Midget City is utterly decimated.
In a strip from October 22, 1905, Nemo has to journey through a forest of gigantic mushrooms; exhausted, he bumps into one of them, which starts a horrific chain reaction, as mushroom after mushroom comes crashing down over the terrified boy.
Finally, on November 19, 1905, Nemo enters a crystal cave. He is introduced to Queen Crystallette, and, despite warnings, he is overcome by love and embraces her; she immediately shatters, beginning yet another chain reaction as every character other than Nemo fractures and splits into countless shards.
In each example, an event that would normally be contained in its effects, instead spreads like a powerful destructive virus until the world itself succumbs to uncontrollable internal pressures and Nemo wakes up. This destructive force is usually unintended; it is the result of carelessness or mischief, by either the dreamer or the dreamed.
This might be indicative of the kinds of things that could happen in multi-user virtual environments when a single process gets out of hand, or when users are granted the kind of absolute power posited by Jaron Lanier when he discusses "post-symbolic" communication. The questions raised are social: if two of us are inhabiting a common world, over which we both have absolute control, what happens to you when I sabotage that world, intentionally or inadvertently?
Despite the inflamed rhetoric of cyberterrorism, viruses and industrial sabotage that are often cited as the most immediate threats, I would guess that the real breakdowns are more likely to come from carelessness and poor design. As usual, it's the day-to-day stuff that gets you.
I am torn between two conflicting desires: on the one hand, I want to pinpoint the source of these disasters so that they can be contained; on the other, I want them to play themselves out, letting them articulate themselves in as much detail as possible so that they can be experienced fully. What could be more thrilling than collapse and destruction on such a vast scale; and where else could such events be experienced (and survived) outside of a virtual world?
5. STRATEGIES OF INTERACTIVITY
Are there any alternatives to the closed system of multiple-choice multimedia templates for designing virtual worlds? Does the designer have to anticipate every possible event in advance? The techniques of artificial life suggest an alternative. Here a process is started and allowed to develop according to evolutionary pressures, without attempting to predict any particular outcome. This would appear to have potential, but it has so far been used with a pretence of neo-scientific objectivity to develop simple organisms; I have seen few attempts to use it to open up a space of higher-level narrative or interaction. I suspect that it may be harder to do this than it appears at first sight; A-life generally involves a "bottom-up" approach, and the units it works with are tiny bits of code; A-life creatures so far focus mostly on elementary tasks like survival, eating, reproduction, etc. Then again, maybe the jump from their ant-like perspective to ours is not so huge as it seems.
Opposed to this are the "top-down" methods: what might be called the "brute force" approach to design; one tries to anticipate more and more branching possibilities, preparing the system for any possible action on the part of the user. The user simply follows one particular path through a totally precalculated narrative space. Eventually, as the world gets more articulated, this route would seem to be doomed to failure because it inevitably results in a combinatorial explosion. This approach also seems like less fun for the designer, and perhaps also for the user, since it precludes any possibility of surprise.
6. MACHINE LOGIC
As Walter Benjamin puts it, the only viewpoint where one doesn't see the equipment is that of the camera; and therefore cinematic language is predicated on a 'special" (and utterly artificial) "procedure." The audience is only able to look where the camera has already looked for them.
In Virtual Reality, the viewpoint is no longer predetermined by the camera - we can look anywhere we want. This is often presented as an vast improvement on the cinematic medium, so one might expect, following on from Benjamin's line of thought, that now, since we can look anywhere, we could, if we wanted to, see the equipment. But all of a sudden there isn't any equipment to be seen. Of course, this is because VR is the result of another 'special procedure"; namely, the element of immersion, which creates a division between inside and outside, and puts the user inside the image. The image has become opaque, and the equipment is hidden safely behind it. Even the virtual machinery - the virtual cameras, lighting equipment, and props - are invisible or unseen. Everything is adjusted so that no matter where you look, you see a unified image, without breaks or gaps.
Adolpho Bioy Casares presented the prototype for this aspect of VR in "The Invention of Morel" first published in 1940. The hapless and unnamed narrator escapes to a mysterious island inhabited by strange people who inexplicably ignore him. A series of fantastical events, including sudden changes in the weather and the tides, uncanny repetitions, and doubled suns and moons, lead the narrator to think that he has died, or else gone completely mad.
Eventually he learns the truth: the scientist Morel has invented the ultimate recording device, with a corresponding projector; these devices can reproduce a person "exactly as he/she is - with the sounds, tactile sensations, flavors, odors, temperatures, all synchronized perfectly. An observer (would) not realize that they are images - you would find it easier to think that - a group of actors, improbable doubles (had been engaged) - No screens or papers are needed; the projections can be received through space - "
But an unfortunate side effect of the recording device is that it steals the 'soul" of the living creatures photographed; the person dies, but the recording, an "image with a soul," lives on, "incorruptible." In fact, Morel brought his friends to the island under false pretences, and then turned the machine on them and himself, recording the entire group in orderto achieve a kind of immortality. Everything experienced by the narrator actually occurred much earlier; the island is inhabited only by projections, replaying over and over again.
By now, the narrator has fallen in love with one of the projections, the enigmatic Faustine; so he eventually turns the recording device on himself, suturing himself into the recording so that, even though he will die, his image can live on with her forever: "When I was ready, I turned on the receivers of simultaneous action. Seven days have been recorded. I performed well: a casual observer would not suspect that I am not a part of the original scene - I rehearsed every action tirelessly. I studied what Faustine says - I often insert an appropriate sentence, so she appears to be answering me - I hope that, generally, we give the impression of being inseparable, of understanding each other so well that we have no need of speaking."
Bioy Casares' story presents a kind of horrific but utterly compelling worst-case scenario, in which our assimilation to the feedback logic of the machine is taken to an ultimate conclusion.
Kevin Kelly asks: "What happens when we connect everything to everything?" The simple answer might be: you can't. That is, if you think you're connecting everything to everything, you're simply redefining "everything" to exclude and ignore whatever remains unconnected. The Invention of Morel shows the utter fallacy of Kelly's question: the narrator "connects" himself to Morel's invention, at the expense of his own life.
Of course, the narrator in Morel is rather witless. For an example of a more resourceful assimilation to machine logic, we have Buster Keaton's film 'sherlock Junior," in which Keaton, playing a projectionist, walks into the movie being projected. A "real" person now caught inside a film, Keaton is subjected to a series of arbitrary cuts. That is, the scene behind him switches, but he doesn't, and he is forced to deal in rapid succession with an ocean, a lion, a stone bench, and so on. Eventually, Keaton learns to manipulate cinematic logic to his own ends, solving the crime, winning the girl.
7. THE BREAKDOWN
How we deal with the possibility of breakdowns is perhaps an ethical question. In one sense, we don't really have a choice because everything is going to break down eventually, so it's simply a question of coming to terms with that fact. Certainly for media artists, there is a lot of pressure to insist that things can be made perfect and can be kept fully operational. But this requires repression of the knowledge that this is not true - eventually, your work is going malfunction. So the choice is whether or not to accelerate this process and make things fall apart or destroy themselves.
The challenge would be to make a work that would accept any sort of behavior. No matter how anarchic or destructive someone wanted to be, it would be part of the core functionality of the work. When you ask people to participate in your work, that work has to be able to take a reasonable amount of abuse. It's not fair to ask people to come and play in your playground and then tell them to be really careful because maybe they are going to break something.
In a sense I am quite happy with things going wrong. I want to find out what a given device is capable of outside of its original function. This is obviously related to how someone defines what a machine or device is supposed to be doing in the first place. Whether something is interpreted as a function, a malfunction, or an accident is completely dependent on your point of view. For something to be an authentic malfunction it would have to be something unplanned. Of course I hate it when my computer crashes. But at the same time these moments provide very fertile ground for thinking about how we work with machines and what they actually do. So although practically speaking such malfunctions are useless, philosophically and theoretically they are full of potential.
Moreover, many malfunctions are not malfunctions at all. They are just the way the technology works. Obviously we're not talking about cases of loss of life or serious injury or anything; that would mean giving up the game entirely. But most of the time what we are doing is not so important that we couldn't stand having something to shake us up a bit.
© 1998 Perry Hoberman / V2_