Lev Manovich
The Poetics of Augmented Space
[ 2002; revised 2004 ]
How is our experience of a spatial form is affected when the form is filled in with dynamic and rich multimedia information? (The examples of such environments are particular urban spaces such as shopping and entertainment areas of Tokyo, Hong Kong, and Seoul where the walls of the buildings are completely covered with electronic screens and signs; convention and trade shows halls; department stores; and at the same time, any human-constructed space where the subject can access various information wirelessly on her cell phone, PDA, or laptop.) Does the form become irrelevant, being reduced to functional and ultimately invisible support for information flows? Or do we end up with a new experience in which the spatial and information layers are equally important? In this case, do these layers add up to a single phenomenological gestalt or are they processed as separate layers?
Although historically built environments were almost always covered with ornament, texts (for instance, shop signs), and images (fresco paintings, icons, sculptures, etc. – think of churches in most cultures), the phenomenon of the dynamic multimedia information in these environments is new. Also new is the delivery of such information to a small personal device, which a space dweller can carry with her.
Therefore, this essay will discuss how the general dynamic between spatial form and information which has been with us for a long time and which I outlined above functions differently in computer culture of today. Since the kinds of environments I offered, as examples above do not have a recognizable name yet, I will give me a new name - an augmented space. The term will be explained in more detail below, but here is the brief definition: augmented space is the physical space overlaid with dynamically changing information. This information is likely to be in multimedia form and it is often localized for each user.
I want to focus on the experience of the human subject in augmented space as opposed to particular electronic, computer and network technologies through which the augmentation is achieved. I also want to re-conceptualize augmentation as an idea and cultural and aesthetic practice rather than as technology. To do this, I will discuss how various practices in professional and vernacular architecture and build environments, cinema, 20th century art, and media art can be understood in terms of augmentation. I hope that this will firmly position the concept of augmented space in historical and cultural as opposed to purely technological sphere.
Augmentation and Monitoring
The 1990s were about the virtual. We were fascinated by the new virtual spaces made possible by computer technologies. Images of an escape into a virtual space that leaves -physical space useless, and of cyberspace – a virtual world that exists in parallel to our world – dominated the decade. This phenomenon started with the media obsession with Virtual Reality (VR). In the middle of the decade graphical browsers for the World Wide Web made cyberspace a reality for millions of users. During the second part of the 1990s, yet another virtual phenomenon – dot coms – rose to prominence, only to crash in the real-world laws of economics. By the end of the decade, the daily dose of cyberspace (using the Internet to make plane reservations, check e-mail using a Hotmail account, or download MP3 files) became so much the norm that the original wonder of cyberspace so present in the early cyberpunk fiction of the 1980s and still evident in the original manifestos of VRML evangelists of the early 1990s - was almost completely lost.[1] The virtual became domesticated. Filled with advertisements and controlled by big brands, it was rendered harmless. In short, to use Norman Klein’s expression , it became an “electronic suburb.”
At the begnning of the twenty first century the research agendas, media attention, and practical applications have come to focus on a new agenda – the physical – that is, physical space filled with electronic and visual information. The previous icon of the computer era – a VR user traveling in virtual space – has been replaced by a new image: a person checking her e-mail or making a phone call using her PDA/cell phone combo while at the airport, on the street, in a car, or any other actually existing space. But this is just one example of what I see as a larger trend. Here are a few more examples of the technological applications that dynamically deliver dynamic data to, or extract data from, physical space – and which already are widely employed at the time of this writing:[2]
1. Video surveillance is becoming ubiquitous. No longer employed only by governments, the military and businesses but also by individuals; cheap, tiny, wireless, and Net-enabled, video cameras can now be placed almost anywhere. (For instance, by 2002, many taxis already had video cameras continuously recording the inside of the cab).
2. If video and other types of surveillance technologies translate the physical space and its dwellers into data, cellspace technologies (also refered to as mobile media, wireless media, or location-based media) work in the opposite direction: delivering data to the mobile physical space dwellers. Cellspace is physical space that is “filled” with data, which can be retrieved by a user via a personal communication device.[3] Some data may come from global networks such as the Internet; some may be embedded in objects located in the space around the user. Moreover, while some data may be available regardless of where the user is in the space, it can also be location-specific. The examples of the cellspace applications which are not localised is using GPS to determine your coordinates, or surfing and checkling email using a cell phone. The examples of location specific applications are using a cel phone to check in at the airport, pay for a road toll, or retrieve information about a product in a store.[4]
3. While we can think of cellspace as the invisible layer of information that is laid over physical space and is customized by an individual user, publicly located computer / video displays present the same visible information to passersby. These displays are gradually becoming larger and thenner; they are no longer confined to flat surfaces; they no longer require darkness to be visible. In the short term, we may expect large thin displays to become more pervasive in both private and public spaces (perhaps using technology such as e-ink). In the longer term, every object may become a screen connected to the Net with the whole of built space eventually becoming a set of display surfaces.[5] Of course, physical space has long been augmented by images, graphics, and type; but replacing all of these with electronic displays makes it possible to present dynamic images, to mix images, graphics, and type, and to change the content at any time.
If we consider the effect of these three technological applications (surveillance, cellspace, electronic displays) on our concept of space and, consequently, on our lives as far as they are lived in various spaces, I believe that they very much belong together. They make physical space into a dataspace: extracting data from it (surveillance) or augmenting it with data (cellspace, computer displays).
It also makes sense to conceptually connect the surveillance/monitoring of physical space and its dwellers, and the augmentation of this space with additional data, because technologicaly these two applications are in a symbiotic relationship. For instance, if you know the location of a person equipped with a cell phone, you can send them particular information relevant to that specific location via their cell phone. A similar relationship exists in the case of software agents, affective computing, and similar interfaces, which take a more active role in assisting the user than the standard Graphical User Interface (GUI). By tracking the user – her mood, her pattern of work, her focus of attention, her interests, and so on – these interfaces acquire information about the user, which they then use to automatically perform the tasks for her.
The close connection between surveillance/monitoring and assistance/augmentation is one of the key characteristics of the high-tech society. This is how such technologies are made to work, and this is why I am discussing data flows from physical space (surveillance, monitoring, tracking) and into physical space (cellspace applications, computer screens, and other examples below) together.
Panopticum and Information Theory
Let us now add to these three examples of the technologies that are already at work by citing a number of the research paradigms which are being actively conducted in University and industry labs. Note that many of them overlap, mining the same territory but with a somewhat different emphasis:
- Ubiquitous Computing: the shift which away from computing centered in desktop machines and towards smaller multiple devices distributed throughout the space.[6]
- Augmented Reality: a paradigm that originated around the same time as ubiquitous computing (1990)– the laying of dynamic and context-specific information over the visual field of a user (see below for more details).[7]
- Tangible Interfaces: treating the whole of physical space around the user as part of a human-computer interface (HCI) by employing physical objects as carriers of information.[8]
- Wearable Computers: embedding computing and telecommunication devices into clothing.
- Intelligent Buildings (or Intelligent Architecture): buildings wired to provide cellspace applications.
- Intelligent Spaces: spaces that monitor user’s interaction with them via multiple channels and provide assistance for information retrieval, collaboration, and other tasks (think of Hal in 2001).[9]
- Context-aware Computing: an umbrella term used to refer to all or some of the developments above, signaling a new paradigm in the computer science and HCI fields.[10]
- Ambient Intelligence: alternative term, which also refers to all or some of the paradigms, summarized above.
- Smart Objects: objects connected to the Net; objects that can sense their users and display “smart” behavior.
- Wireless Location Services: delivery of location-specific data and services to portable wireless devices such as cell phones (i.e., similar to cellspace).
- Sensor Networks: networks of small sensors that can be used for surveillance and enviromental monitoring, to create intelligent spaces, and similar applications.
- E-paper (or e-ink): a very thin electronic display on a sheet of plastic, which can be flexed in to different shapes and which displays information that is received wirelessly.[11]
While the technologies imagined by these research paradigms accomplish their intentions in a number of different ways, the end result is the same: overlaying dynamic data over the physical space. I will use the term “augmented space” to refer to this new kind of physical space. As I have already mentioned, this overlaying is often made possible by the tracking and monitoring of users. In other words, the delivery of information to users in space, and the extraction of information about those users, are closely connected. Thus, augmented space is also monitored space.
Augemented space is the physical space which is “data dense,” as every point now potentially contains various information which is being delivered to it from elsewhere. At the same time, video surveillance, monitoring, and various sensors can also extract information from any point in space, recording the face movements, gestures and other human activity, temperature, light levels, and so on. Thus we can say that various augmentation and monitoring technologies add new dimensions to a 3D physical space, making it multi-dimensional. As a result, the physical space now contains many more dimensions than before, and while from the phenomenological perspective of the human subject, the “old” geometric dimensions may still have the priority, from the perspective of technology and its social, political, and economic uses, they are no longer more important than any other dimension.
This demise in importance of geometry as seen in augmented spaces can be understood as a part of a larger paradigm shift. If modern society as summed up in Michel Foucault’s metaphor of Panopticum was organized around the strait lines of human sight, i.e. the geometry of the visible, this is no longer the case for our society. While some technologies such as video surveillance and infrared communication still require a line of sight, most do not. The examples are cellular and Bluetooth communication, radar, and environmental sensors. Instead of the binary logic of visible/invisible, the new spatial logic can be described using such terms as functions or fields, since from the point of view of these new technologies, every point in space has a particular value on a possible continuum. (Think for instance of a strength of your cellular signal which varies depending how close you are to a cell or whether you outside or inside.) In the case of information delivery into space, these values determine how much, how quickly and how successfully this information can be delivered – in other words, it corresponds to communication bandwidth. In the case of monitoring or surveillance, these values similarly affect how much and how successfully information can be extracted from a point, or region in space. In either case, if the old binary logic of visible/invisible (or present/absent) had still applied in this case, we would either register a signal or not. Instead we witness a new logic, which is described by the key intellectual paradigm of information society - mathematical theory of communication developed by Claude Shannon and others in the 1940s. According to this theory, communication is always accompanied by noise, and therefore a received signal always has some noise mixed in.[12] In practical terms, this means that any information delivered to or extracted from augmented space always occupies some position on the continuous dimension whose poles is a perfect signal and complete noise. In a typical situation, we are usually somewhere in between: our cell phone conversation is accompanied by some background noise; a surveillance system delivers blurry or low-res images, which needs to be interpreted, i.e. a decision needs to be made by somebody what is the signal being present. Thus along with providing a theoretical framework to describe all electronic communication, mathematical theory of communication by Shannon turns to also perfectly capture the practical reality of our communications, at least up until now. That is, in the majority of cases, the signals we receive are accompanied by noise visible to us.
Augmentation and Immersion
I derived the term “augmented space” from the already established term “augmented reality” (AR).[13] Coined around 1990, the concept of “augmented reality” is normally opposed to “virtual reality” (VR).[14] in the case of VR, the user works on a virtual simulation, in the case of AR, she works on actual things in actual space. Because of this, a typical VR system presents a user with a virtual space that has nothing to do with that user’s immediate physical space; while, in contrast, a typical AR system adds information that is directly related to the user’s immediate physical space.
But we don't necessarily have to think of immersion in the virtual and augmentation of the physical as opposites. On one level, whether we think of a particular situation as immersion or augmentation is simply a matter of scale - i.e. the relative size of a display. When you are watching a movie in a movie theatre or on big TV monitor, or when you are playing a computer game on a game console that is connected to the TV, you are hardly aware of your physical surroundings. Practically speaking, you are immersed in virtual reality. But when you watch the same movie, or play the same game, on the small display of a cell phone or PDA that fits in your hand, then the experience is different. You are still largely present in physical space, and while the display adds to your overall phenomenological experience, it does not take over. So, whether we should understand a particular situation in terms of immersion or augmentation depends on how we understand the idea of addition: we may add new information to our experience – or we may add an altogether different experience.
“Augmented space” may bring associations with one of the founding ideas of computer culture: Douglas Engelbardt’s concept of a computer augmenting human intellect that wasarticulated 40 years ago.[15] The association is appropriate, but we also need to be aware of the differences . For the vision of Engelbardt, and the related visions of Vannevar Bush and J.C.R. Licklider, assumed a stationary user – a scientist or engineer at work in his office. Revolutionary for the time, these ideas anticipated the paradigm of desktop computing. Today, however, we are gradually moving into the next paradigm, one in which computing and telecommunication capacities are delivered to a mobile user.[16] Thus, augmenting the human also comes to mean augmenting the whole space in which she lives, or through which she passes.
Augmentation as an Idea
Having analyzed at some length the concept of augmented space, we are now ready to move to the key questions of this essay. What is the phenomenological experience of being in a new augmented space? What can be the new cultural applications of new computer and network enabled augmented spaces? What are possible poetics and aesthetics of an augmented space?
One way to begin thinking about these questions is to approach the design of augmented space as an architectural problem. Augmented space provides a challenge and an opportunity for many architects to rethink their practice, since architecture will have to take into account the fact that virtual layers of contextual information will overlay the built space.
But is this a completely new challenge for architecture? If we assume that the overlaying of different spaces is a conceptual problem that is not connected to any particular technology, we may start to think about which architects and artists have already been working on this problem. To put it another way, the layering of dynamic and contextual data over physical space is a particular case of a general aesthetic paradigm: how to combine different spaces together. Of course, electronically augmented space is unique - since the information is personalized for every user, it can change dynamically over time, and it is delivered through an interactive multimedia interface, etc. Yet it is crucial to see this as a conceptual rather than just a technological issue – and therefore as something that in part has already - been a part of other architectural and artistic paradigms.
Augmented space research gives us new terms with which to think about previous spatial practices. If before we would think of an architect, a fresco painter, or a display designer working to combine architecture and images, or architecture and text, or to incorporate different symbolic systems in one spatial construction, we can now say that all of them were working on the problem of augmented space. The problem, that is, of how to overlay physical space with layers of data.. Therefore, in order to imagine what can be done culturally with augmented spaces, we may begin by combing cultural history for useful precedents.
To make my argument more accessible, I have chosen two well-known contemporary figures as my examples. Janet Cardiff is a Canadian artist who became famous for her ‘audio walks’. She creates her pieces by following a trajectory through a space and narrating an audio track that combines instructions to the user (“go down the stairs”; “look in the window”; “go through the door on the right”) with narrative fragments, sound effects, and other aural ‘data’. To experience the piece, the user dons earphones connected to a CD player and follows Cardiff’s instructions.[17] In my view - even though Cardiff does not use any sophisticated computer, networking, or projection technologies - her ’walks’ represent the best realization of the augmented space paradigm so far. . They demonstrate the aesthetic potential of laying new information over a physical space. Their power lies in the interactions between the two spaces - between vision and hearing (what the user is seeing and what she is hearing), and between present and past (the time of the user’s walk versus the audio narration, which, like any media recording, belongs to some undefined time in the past).
The Jewish Museum Berlin by Daniel Libeskind can be thought of as another example of augmented space research. For, if Cardiff lays a new dataspace over the existing architecture and/or landscape, then Libeskind uses the existing dataspace to drive the new architecture that he constructs. After putting together a map that showed the addresses of Jews who were living in the neighborhood of the museum site before World War II, the architect connected different points on the map and then projected the resulting net onto the surfaces of the building. The intersections of the projected net and the Museum walls gave rise to multiple irregular windows. Cutting through the walls and the ceilings at different angles, these windows evoke many visual references: the narrow eyepiece of a tank; the windows of a medieval cathedral; the exploded forms of the cubist/abstract/suprematist paintings of the 1910s-1920s. Just as in the case of Cardiff’s audio walks, here the virtual becomes a powerful force that re-shapes the physical. In the Jewish Museum Berlin the past literally cuts into the present. Rather than something ephemeral, an immaterial layer over the real space, here dataspace is materialized to become a sort of monumental sculpture.
White Cube as Cellspace
While we may interpret the practices of selected architects and artists as having particular relevance to thinking about the ways in which augmented space can be used culturally and artistically, there is another way to link the augmented space paradigm with modern culture. Here is how it works.
One trajectory that can be traced in 20th century art runs from the dominance of a two-dimensional object placed on a wall, towards the use of the whole 3-D space of a gallery. (Like all other cultural trajectories in the 20th century, this one is not a linear development; rather, it consists of steps forward and steps back that occur in rhythm with the general cultural and political rhythm of the century: the highest peak of creativity took place in the 1910s-1920s, followed by a second peak in the 1960s). Already in the 1910s, Tatlin’s reliefs broke the two-dimensional picture plane and exploded a painting into the third dimension. In the 1920s, Lissitzky, Rodchenko, and other pioneering exhibition designers moved further away from an individual painting or sculpture towards using all surfaces of an exhibition space – yet their exhibitions activate only the walls rather than the whole space.
In the mid-1950s, assemblage legitimized the idea of an art object as a three-dimensional construction ( The Art of Assemblage, MOMA, 1961). In the 1960s, minimalist sculptors (Carl Andre, Donald Judd, Robert Morris) and other artists (Eva Hesse, Arte Povera) finally started to deal with the whole of the 3-D space of a white cube. Beginning in the 1970s, installation (Dan Graham, Bruce Nauman) grew in importance to become, in the 1980s, the most common form of artistic practice of our times – and the only thing that all installations share is that they engage with 3-D space. Finally, the white cube becomes a cube – rather than just a collection of 2-D surfaces.
If we follow this logic, augmented space can be thought of as the next step in the trajectory from a flat wall to a 3-D space which has animated modern art for the last hundred years. For a few decades now, artists have already dealt with the entire space of a gallery: rather than creating an object that a viewer would look at, they placed the viewer inside the object. Now the artists have a new challenge: placing a user inside a space filled with dynamic, contextual data with which the user can interact. Alternatively, if we want to be more modest, we can say that the arrival of augmented space in the 1980s and 1990s as deployed in urban sphere was paralleled by the development of a similar concept of space by installation artists. If before 3D space was in practice reduced to a set of surfaces – walls in the case of the built environment; flat paintings or gallery walls in an art environment – now it is finally used as 3D space.
White Cube Versus Black Box
Before we rush to conclude that the new technologies do not add anything substantially new to the old aesthetic paradigm of overlaying different spaces together, let me note that -– in addition to their ability to deliver dynamic and interactive information -– the new technologically implemented augmented spaces also differ in one important aspect from Cardiff’s walks, Libeskind’s Jewish Museum, and other similar works.. Rather than -– laying a new 3-D virtual dataspace over the physical space, Cardiff and Libeskind overlay only a 2-D plane, or a 3-D path, at best. Indeed, Cardiff’s walks are new 3-D paths placed over an existing space, rather than complete spaces. Similarly, in the Jewish Museum Berlin, Libeskind projects 2-D maps onto the 3-D shapes of his architecture.[18]
In contrast, GPS, wireless location services, surveillance technologies, and other augmented space technologies all define dataspace – if not in practice, than at least in theory - as a continuous field that completely extends over, and fills in, all of physical space. Every point in space has a GPS coordinate that can be obtained using a GPS receiver. Similarly, in the cellspace paradigm, every point in physical space can be said to contain some information that can be retrieved using a PDA or similar device. With surveillance;while in practice video cameras, satellites, Echelon (the set of monitoring stations that are used by the U.S. to monitor all kinds of electronic communications globally), and other technologies, can so far only reach some regions and layers of data but not others; the ultimate goal of the modern surveillance paradigm is to able to observe every point at every time. To use the terms of Borges’ famous story, all of these technologies want to make the map equal to the territory. And if, in accord with Foucault’s famous argument in Discipline and Punish, the modern subject internalizes surveillance and thereby removes the need for anybody to be actually present in the center of the Panopticum to watch him/her, modern institutions of surveillance insist that s/he should be watched and tracked everywhere all the time.
It is important, however, that, in practice, dataspaces are almost never continuous: surveillance cameras’ look at some spaces but not at others, wireless signals are stronger in some areas and non-existent in others, and so on. As Matt Locke eloquently describes this,
Mobile networks have to negotiate the architecture of spaces that they attempt to inhabit. Although the interfaces have removed themselves from physical architectures, the radio waves that connect cell spaces are refracted and reflected by the same obstacles, creating not a seamless network but a series of ebbs and flows. The supposedly flat space of the network is in fact flat, pulled into troughs and peaks by the gravity of architecture and the users themselves.[19]
The contrast between the continuity of cellspace in theory and its discontinuity in practice should not be dismissed. Rather, it itself can be the source of interesting aesthetics strategies.
My third example of already existing augmented space – electronic displays mounted in shops, streets, lobbies, train stations, and apartments – follows a different logic. Rather than overlaying all of the physical space, here dataspace occupies a well-defined part of the physical space. This is the tradition of Alberti’s window, and, consequently, of post-Renaissance painting, the cinema screen, the TV screen, and the computer monitor. However, if the screen has, until recently, most usually acted as a window into a virtual 3-D space; in the last two decades of the 20th century it has turned into a shallow surface in which 3-D images co-exist with 2-D design and typography. Live-action footage shares space with motion graphics (animated type), scrolling data (for instance, stock prices or weather), and 2-D design elements. In short, the Renaissance painting became an animated Medieval illustrated book.
My starting point for the discussion of the poetics of this type of augmented space is the current practice of video installation, which came to dominate the art world in the 1990s. Typically, these installations use video or data projectors. They turn a whole wall or even a whole room into a display or a set of displays, thus previewing and investigating (willingly or not) the soon-to-come future of our apartments and cities when large and thin displays covering most surfaces may become the norm. At the same time, these laboratories of the future are rooted in the past: in the different traditions of “image within a space” of 20th century culture.
What are these traditions? Among the different oppositions that have structured the culture of the 20th century, and which we have inherited, has been the opposition between the art gallery and the movie theatre. One was high-culture; the other was low-culture. One was a white cube; the other was a black box.
Given the economy of art production – one-of-a-kind objects created by individual artists – 20th century artists expended lots of energy experimenting with what could be placed inside the neutral setting of a white cube: by breaking away from a flat and rectangular frame and going into the third dimension; covering a whole floor; suspending objects from the ceiling; and so on. In other words, if we are to make an analogy between an art object and a digital computer, we can say that, in modern art, both the ‘physical interface’ and the ‘software interface’ of an art object were not fixed but open for experimentation. Put diffirently, both the physical appearance of an object and the proposed mode of interaction with an object were open for experimentation. Artists also experimented with the identity of a gallery: from a traditional space of aesthetic contemplation to a place for play, performance, public discussion, lectures, and so on.
In contrast, since cinema was an industrial system of mass production and mass distribution, the physical interface of a movie theatre and the software interface of a film itself were pretty much fixed: a 35-mm image of fixed dimensions projected on a screen with the same frame ratio, dark space where viewers were positioned in rows, and the fixed time of a movie itself. Not accidentally, when the experimental filmmakers of the 1960s started to systematically attack the conventions of traditional cinema, these attacks were aimed at both its physical and its software interfaces. Robert Breer, for example, projected his movies on a board that he would hold above his head as he walked through a movie theatre towards the projector; Stan VanderBeck constructed semi-circular tents for the projection of his films; etc.
The gallery was the space of refined high taste while the cinema served to provide entertainment for the masses, and this difference was also signified by what was deemed to be acceptable in the two kinds of spaces. Despite all the experimentation with its “interface,” until recently the gallery space was primarily reserved for static images; to see moving images, the public had to go a movie theatre. Thus, until at least the 1980s, moving images in a gallery were indeed an exception (Duchamp’s rotoscopes, Acconci’s masturbating performance, which can be thought as a kind of animation within the gallery).
Given this history, the 1990s’ phenomena of omni-present video installations taking over the gallery space goes against the whole paradigm of modern art – and not only because installations bring moving images into the gallery. Most video installations adopt the same physical interface: a dark enclosed or semi-enclosed rectangular space with a video projector at one end and the projected image appearing on the opposite wall. Therefore, from a space of constant innovation in relation to the physical and software interface of an art object, a gallery space has turned into what was, for almost a century, its ideological enemy – a movie theatre that ischaracterized by the rigidity of its interface.
Since the early days of computer culture in the 1960s, many software designers and software artists – from Ted Nelson and Alan Kay to Perry Hoberman and IOD – have revolted against the hegemony of mainstream computer interfaces, such as the keyboard and mouse, GUI, or commercial Web browsers. Similarly, the best of video or, more generally, moving image installation artists, go beyond the standard video installation interface – a dark room with an image on one wall. Examples of such artists include Diana Thater, Gary Hill, and Doug Aitken, as well as the very first ‘video artist’ – Nam Juke Paik. The founding moment of what would come to be called ‘video art’ was Paik’s attack on the physical interface of a commercial moving image – his first show consisted of televisions with magnets attached to them, and TV monitors ripped out of their enclosures.
The Electronic Vernacular
When we look at what visual artists are doing with a moving image in a gallery setting in comparison with other contemporary fields, we can see that the white gallery box still functions as a space of contemplation – quite different from the aggressive, surprising, overwhelming spaces of a boutique, trade show floor, airport, or retail/entertainment area of a major metropolis.[20] While a number of video artists continue the explorations of the 1960s ‘expanded cinema’ movement by pushing moving image interfaces in many interesting directions, outside of a gallery space we can find much richer field of experimentation. I can single out four areas. First, contemporary urban architecture - in particular, many proposals of the last decade that incorporate large projection screens into architecture and project the activity inside onto these screens. Example include Rem Koolhaas’ unrealized 1992 project for the new ZKM building in Karlsruhe; a number of projects again so far mostly unrealized by Robert Venturi to create what he calls “architecture as communication” (buildings covered with electronic displays); realized architectural/media installations by Diller + Scofilio such as Jump Cuts and Facsimile[21]; the highly concentrated use of video screens and information displays in certain cities such as Seoul, Hong Kong and Tokyo, or in Times Square, NYC; and, finally, imaginary future architecture as seen in movies from Blade Runner (1982) to Minority Report (2002), which use electronic screens on a scale that is not yet possible. Second is the use of video displays in certain kind of contemporary spaces where communication of information to public is the key functions: trade show design, such as the annual SIGGRAPH and E3 conventions; company showrooms; airports and train stations. The third is the best of retail environments. These range from small high-end boutiques (I will discuss this type of space in more detail shortly) to mega-size shopping centers / eating/ entertainment complexes which incorporate projection screens, dynamic lighting systems, mirrors, transparent and translucent surfaces to create an experience of an animated and dynamic space. The forth is the multi-media design of music performances, from the concerts of the brand name pop starts, to the numerous VJs performing nightly in clubs in most major cities on earth, to ‘hybrid’ groups which situate themselves between club and art culture, such as brilliant collective Light Surgeons based in London.
While at this moment they are still imagioned and implemented by the practioners from diffirent fields, we start slowly seeing the diffirent species of augmented spaces being combined into one. A shopping complex leads to an interior shopping street which leads to a multiplex; or an airport complex combines information displays about airline departures and ariival and shopping areas with their own promotions playing on LCD screens, and so on. Although at present the small electronic screens are usually distributed throughout these spaces (for instance, small LCD monitors mounted in elevators of new hi-rize buildings in Hong Kong and China such as CITIC Plaza in Guangzhou), the single larger screen (or other method for large image creation) has a potential to unite them all, offering a a kind of symbolic unity to a typically heterogeneous urban program: a shopping center + entertainment center + hotel + residential units. As as an example, consider Langham Place (Mongkok, Hong Kong, opened November 2004 ) developed by The Jerde Partnership, the pionners of the ubran version of ‘experience design’ they refer as ‘placemaking.’ An entertainment complex with an area of 1.8 million square feet, it combines a 15-storey shopping mall with 300 shops, a 59-level Grade A office tower and the 5-star Langham Place Hotel. The focal point of the complex is Digital Sky which is spanning the entire roof of the mall. Showing continuos visuals, this giant ‘screen’ is made possible by 200 projectors, PCs, speakers, and special effects lights.[22] No longer a square superimposed on a façade or a wall, here an image envelops the whole space as an ambient “elevator music” sky to shop under.
To discuss the use of electronic images in architecture further, let us turn to Robert Venturi. His projects and theories deserve special consideration here since, for him, an electronic display is not an optional addition but the very center of architecture in the information age. Since the 1960s, Venturi continuously argued that architecture should learn from vernacular and commercial culture (billboards, Las Vegas, strip malls, architecture of the past). Appropriately, his books Complexity and Contradiction in Architecture and Learning from Las Vegas are often referred to as the founding documents of post-modern aesthetics. Venturi proposed that we should refuse the modernist desire to impose minimalist ornament-free spaces, and instead embrace complexity, contradiction, heterogeneity, and iconography in our built environments.[23] In the 1990s, he articulated the new vision of “architecture as communication for the Information Age (rather than as space for the Industrial Age).”[24] Venturi wants us to think of “architecture as an iconographic representation emitting electronic imagery from its surfaces day and night.” Pointing to some of the already mentioned examples of the aggressive incorporation of electronic displays in contemporary environments, such as Times Square in NYC, and arguing that traditional architecture always included ornament, iconography, and visual narratives (for instance, a Medieval cathedral with its narrative window mosaics, narrative sculpture covering the façade, and narrative paintings), Venturi proposed that architecture should return to its traditional definition as iconography, i.e. as information surface.[25] Of course, if the messages communicated by traditional architecture were static and reflected the dominant ideology, today’s electronic dynamic interactive displays make it possible for these messages to change continuously ; making the information surface a potential space of contestation and dialog, which functions as the material manifestation of the often invisible public sphere.
Although this has not been a part of Venturi’s core vision, it is relevant to mention here a growing number of projects in which the large publicly mounted screen is open for programming by the public who can send images via Internet or information being displayed via their cell phones. Even more suggestive is the project Vectorial Elevation, Relational Architecture #4 by artist Raffael Lozano-Hemmer[26]. This project made it possible for people from all over the world to control a mutant electronic architecture made from search lights in Mexico City’s Zócalo Square. To quote from the statement of the 2002 Prix Ars Electronica jury, which awarded this project the Golden Nica in the Interactive Art category:
Vectorial Elevation was a large scale interactive installation that transformed Mexico City’s historic centre using robotic searchlights controlled over the Internet. Visitors to the project web site at
The website featured a 3D-java interface that allowed participants to make a vectorial design over the city and see it virtually from any point of view. When the project server in Mexico received a submission, it was numbered and entered into a queue. Every six seconds the searchlights would orient themselves automatically and three webcams would take pictures to document a participant’s design.[27]
Venturi’s vision of “architecture as iconographic representation” is not without its problems. If we focus completely on the idea of architecture as information surface, we may forget that traditional architecture communicated messages and narratives not only through flat narrative surfaces but also through the particular articulation of space. To use the same example of a medieval cathedral, it communicated Christian narratives not only through the images covering its surfaces but also through its whole spatial structure. In the case of modernist architecture, it similarly communicated its own narratives (the themes of progress, technology, efficiency, and rationality) through new spaces constructed from simple geometric forms – and also through its bare, industrial-looking surfaces. (Thus, the absence of information from the surface, articulated in the famous “ornament is crime” slogan of Adolf Loos, itself became a powerful communication technique of modern architecture.)
An important design problem of our own time is how to combine the new functioning of a surface as an electronic display with the new kind of spaces and forms being imagioned by contemporary architects.[28] While Venturi fits electronic displays on to his buildings, which closely follow traditional vernacular architecture, this is obviously not the only possible strategy. The well-known Freshwater Pavilion by NOX/Lars Spuybroek (Netherlands, 1996) follows a much more radical approach. To emphasize that the interior of the space constantly mutates, Spuybroek eliminates all straight surfaces and straight angles; he makes the shapes defining the space appear to move; and he introduces computer-controlled lights that change the illumination of the interior.[29] As described by Ineke Schwartz, “There is no distinction between horizontal and vertical, between floors, walls and ceilings. Building and exhibition have fused: mist blows around your ears, a geyser erupts, water gleams and splatters all around you, projections fall directly onto the building and its visitors, the air is filled with waves of electronic sound.”[30]
I think that Spuybroek’s building is a successful symbol for the Information Age. Its continuously changing surfaces illustrate the key effect of the computer revolution: the substitution of every constant by a variable. In other words, the space that symbolizes the Information Age is not the symmetrical and ornamental space of traditional architecture, the rectangular volumes of modernism, nor the broken and blown up volumes of deconstruction.Rather, it is space whose shapes are inherently mutable and whose soft contours act as a metaphor for the key quality of computer-driven representations and systems: variability.
Learning from Prada
Venturi wants to put rich electronic ornamentation and iconography on traditional buildings. In contrast, in his Freshwater Pavilion Lars Spuybroek constructs a new kind of space which he then fills with information – but information reduced to abstract color fields and sound. In other words, in the Freshwater Pavilion, the information surface functions in a very particular way, displaying color fields rather than text, images, or numbers. Where can we find today interesting architectural spaces combined with electronic displays that show the whole range of information, from ambient color fields to figurative images and numerical data?
Beginning in the mid 1990s, the avant-garde wing of the retail industry began to produce rich and intriguing spaces, many of which incorporate moving images. Leading architects and designers such as Droog/NL, Marc Newson, Herzog & de Meuron, Renzo Piano, and Rem Koolhaas created stores for Prada, Mandarina Duck, Hermes, Comme des Garcons, and other high-end brands; while architect Richard Glucksman collaborated with artist Jenny Holzer to create Helmut Lang’s stunning New York parfumerie, which incorporates Holzer’s signature use of LCD displays. A store featuring dramatic architecture and design, and the mixing of a restaurant, fashion, design, and art gallery became a new paradigm for high-end brands. Otto Riewoldt describes this paradigm using the term “brandscaping” – promoting the brand by creating unique spaces. According to Riewoldt: “Brandscaping is the hot issue. The site at which goods are promoted and sold has to reinvent itself by developing unique and unmistakable qualities.”[31]
OMA / Rem Koolhaas’ Prada store in New York (2002) pushes brandscaping to a new level. Koolhaus seems to achieve the impossible by creating a flagship store for the Prada brand – and at the same time an ironic statement about the functioning of brands as new religions.[32] The imaginative use of electronic displays designed by Reed Kram of Kramdesign is an important part of this statement. On entering the store, the visitor discovers glass cages hanging from the ceiling throughout the space. Just as a church would present the relics of saints in special displays, here the glass cages contain the new objects of worship – Prada clothes. The special status of Prada clothing is further enhanced by the placement of small flat electronic screens throughout the store on horizontal shelves right alongside the merchandize. The clothes are equated with the ephemeral images playing on the screens, and, vice versa, the images acquire a certain materiality, as though they are themselves objects . By positioning screens showing moving images right next to the clothes, the designers ironically refer to what everybody today already knows: we buy objects not for themselves but in order to emulate the specific images and narratives that are presented by the advertisements of these objects. Finally, on the basement level of the store, you discover a screen displaying the Prada Atlas. Designed by Kram, the Atlas maybe be mistaken for an interactive multimedia presentation of OMA (Office for Metropolitan Architecture, the name of Koolhaus’ studio) research for its Prada commission. It looks like the kind of information that brands normally communicate to their investors but not to their consumers. In designing the Atlas, as well as the whole media of the store, Kram’s goal was to make “Prada reveal itself, make it completely transparent to the visitors.”[33] The Atlas lets you list all of the Prada stores throughout the world by square footage, look at an analysis of optimal locations for store placement, and study other data sets that underlie Prada’s brandscaping. This ‘unveiling’ of Prada does not break our emotional attachment with the brand; on the contrary, it seems to have the opposite result. Koolhaus and Kram masterfully engage the ‘I know it is an illusion but nevertheless I believe it’ effect: we know that Prada is a business that is governed by economic rationality and yet we still feel that we are not simply in a store but in a modem church.
It is symbolic that Prada NYC has opened in the space that was previously occupied by a branch of the Guggenheim Museum. The strategies of brandscaping are directly relevant to museums and galleries that, like all other physical spaces, now have to compete against that new information, entertainment, and retail space: a computer or a cell phone screen connected to the Net. Although museums in the 1990s have similarly expanded their functionality, often combining galleries, a store, film series, lectures, and concerts, design-wise they can learn from retail design, which, as Riewoldt points out, “has learnt two lessons from the entertainment industry. First: forget the goods, sell thrilling experience to the people. And secondly: beat the computer screen at its own game by staging real objects of desire – and by adding some spice to the space with maybe some audio-visual interactive gadgetry.” [34]
In a high-tech society, cultural institutions usually follow the technology industry. A new technology is developed for military, business, or consumer use, and after a while cultural institutions notice that some artists are experimenting with that technology and so they start to incorporate it in their programming. Because they have the function of collecting and preserving artworks, the art museums today often look like historical collections of media technologies from previous decades. Thus one may well mistake a contemporary art museum for a museum of obsolete technology. Today, while outside one finds LCDs and PDAs, data projectors, and HDTV cameras, inside a museum we may expect to find slide projectors, 16-mm film equipment, and 3/4-inch video decks.
Can this situation be reversed? Can cultural institutions play an active, even a leading, role, acting as laboratories where alternative futures are tested? Augmented space – which is slowly becoming a reality – is one opportunity for these institutions to take a more active role. While many video installations already function as laboratories for developing new configurations of images within space, museums and galleries as a whole could use their own unique asset – a physical space – to encourage the development of distinct new spatial forms of art and new spatial forms of the moving image. In this way, they can take a lead in testing outone part of the augmented space future.
Having stepped outside the picture frame into the white cube walls, floor, and the whole space, artists and curators should feel at home taking yet another step: treating this space as layers of data. This does not mean that the physical space becomes irrelevant; on the contrary, as the practice of Cardiff and Libeskind shows, it is through the interaction of the physical space and the data that some of the most amazing art of our time is being created.
Augmented space also represents an important challenge and an opportunity for contemporary architecture. As the examples discussed in this essay demonstrate, while many architects and interior designers have actively embraced electronic media, they typically think of it in a limited way: as a screen, i.e., as something that is attached to the ‘real’ stuff of architecture, i.e. surfaces defining volumes. Venturi’s concept of architecture as ‘information surface’ is only the most extreme expression of this general paradigm. While Venturi logically connects the idea of surface as electronic screen to the traditional use of ornament in architecture and to such features of vernacular architecture as billboards and window product displays, this historical analogy also limits our visions of how architecture can use new media. For, in this analogy, an electronic screen becomes simply a moving billboard or a moving ornament.
Going beyond the ‘surface as electronic screen paradigm’, architects now have the opportunity to think of the material architecture that most usually preoccupies them and the new immaterial architecture of information flows within the physical structure as a whole. In short, I suggest that the design of electronically augmented space can be approached as an architectural problem. In other words, architects along with artists can take the next logical step to consider the ‘invisible’ space of electronic data flows as substance rather than just as void – something that needs a structure, a politics, and a poetics.
[1] VRML stands for the Virtual Reality Modeling Language. In the first part of the 1990s, the inventors of this language designed it to model and access 3-D interactive virtual worlds over the Internet, and promoted it as the material realization of the idea of cyberspace. (See, for instance, Mark Pesce, "Ontos, Eros, Noos, Logos," the keynote address for ISEA (International Symposium on Electronic Arts) 1995, http://www.xs4all.nl/~mpesce/iseakey.html.) As of this writing (May 2002), Internet-based 3-D virtual worlds have failed to become popular.
[2] This text was originally written in early 2002; current edit was done in September 2004.
[3] Coined in 1998 by David S. Bennahum, the term “cellspace” originally referred to the then new ability to access e-mail or the Internet wirelessly. Here I am using the term in a broader sense.
[4] It is interesting to think of GPS (Global Positioning System) as a particular case of cellspace. Rather than being tied to an object or a building,, here the information is a property of the Earth as a whole. A user equipped with a GPS receiver can retrieve a particular type of information relative to their location – the coordinates of this location. GPS systems are gradually is being integrated into various telecommunication and transportation technologies, from cell phones, to PDAs, to cars.
[5] Recall the opening scene of Blade Runner (1982) in which the whole side of a high-rise building acts as a screen.
[6] M. Weiser, “The Computer for the Twenty-first Century,” Scientific American, 265(3):94–104, September 1991.
[7] W. MacKay, G. Velay, K. Carter, C. Ma, and D. Pagani, “Augmenting Reality: Adding Computational Dimensions to Paper,” Communications
of the ACM, 36(7):96–97, 1993. Kevin Bonsor, “How Augmented Reality Will Work,” http://www.howstuffworks.com/augmented-reality.htm.
[8] See the ‘Tangible Bits’ project at the MIT Media Lab, http://tangible.media.mit.edu/projects/Tangible_Bits/projects.htm.
[9] Guido Appenzeller, Intelligent Space Project (http://gunpowder.Stanford.EDU/~appenz/ISpace/); Intelligent Room Projects, AI Lab, MIT. (http://www.ai.mit.edu/projects/iroom/projects.shtml).
[10] Tom Moran and Paul Dourish, “Introduction to the Special Issue on Context-aware Computing,”Human Computer Interaction, 16:108, 2001.
[11] Ivan Noble, “E-paper Moves a Step Nearer,” BBC News Online, 23 April, 2001. (http://news.bbc.co.uk/hi/english/sci/tech/newsid_1292000/1292852.stm).
[12] If the noise falls below a certain threshold, we are able to reconstruct the send signal perfectly; conversely, if noise is above a particular threshold, the signal disappears. These thresholds are never absolute; they are specific to a particular communication situation, influenced by the bandwidth of a communication channel and also the content of a message.
[13] For AR research sites and conferences, see http://www.augmented-reality.org.
[14] With a typical VR system, all work is done in a virtual space; physical space becomes unnecessary, and it’s the user’s visual perception of physical space is completely blocked. In contrast, an AR system helps the user to work in a physical space by augmenting that space with additional information. This end is achieved by laying information over the user’s visual field. An early scenario of a possible AR application that was developed at Xerox PARC involved a wearable display for copier repairman, which overlaid a wireframe image of the copier’s insides over the actual copier as it was being repaired. .
Today, additional scenarios for everyday use can be imagined: for instance, AR glasses for a tourist that layer dynamically changing information about the sites in a city over her visual field. Military and artistic applications are also being developed, as presented for instance in the exhibition showcasing AR projects developed by Ars Electronica FutureLab (Ars Electronica Festival 2003). In this new iteration, AR becomes conceptually similar to wireless location services. The idea shared by both is that when the user is in the vicinity of particular objects, buildings, or people, then information about them is delivered to the user.But while this information is displayed, in cellspace, on a cell phone or PDA; in AR the information is laid over the user’s visual field.
The decrease in the popularity of VR in mass media and a slow but steady rise in AR-related research in the last five years is one example of the ways in which the augmented space paradigm is now overtaking the virtual space paradigm. Interestingly, this reversal can be said to be anticipated in the very origins of VR. In the late 1960s, Ivan Sutherland developed what we came to know as the first VR system. The user of the system saw a simple wireframe cube whose perspectival view would change as the user moved his head. The wireframe cube appeared overlaid over whatever the user was seeing. Because the idea of a 3-D computer graphics display whose perspective changes in real time according to the position of the user became associated with subsequent virtual reality systems, Sutherland is credited with inventing the first VR system. But it can be also argued that this was not a VR but rather an AR system because the virtual display was overlaid over the user’s field of vision without blocking it. In other words, in Sutherland’s system, new information was added to the physical environment: a virtual cube.
[15] Vannevar Bush, “As We May Think” (1945); Douglas Engelbart, “Augmenting Human Intellect: A Conceptual Framework” (1962). Both in Noah Wardrip-Fruin and Nick Montfort, eds., The New Media Reader (MIT Press, forthcoming 2002).
[16] And while it it may still be more efficient to run, say, CAD, 3-D modeling, or Web design software while sitting comfortably in front of a 30-inch LCD display, there are many other types of computing and telecommunication activities that do not require or encourage stationary use.
[17] I only experienced one of her “walks” that she created for P.S. 1 in New York in 2001.
[18] For whose readers familiar with these concepts, the artistic augmented spaces I have evoked can be thought of as 2-D texture maps, while technologically augmented spaces can be compared to a solid texture.
[19] Matt Locke, in Mobile Minded, eds. Geert Lovink and Mieke Gerritzen (Corte Madera, CA: Ginko Press, 2002), 111.
[20] This passive and melancholic quality of video art was brilliantly staged in a recent exhibition design by LO/TEK, Making Time: Considering Time as a Material in Contemporary Video & Film, in the Hammer Museum in Los Angeles (February 4 - April 29, 2001). As Norman Klein pointed out to me, LO/TEK designed a kind of collective tomb - a cemetery for video art.
[21] Overview of Diller + Scofilio projects can be found at http://www.labiennaledivenezia.net/it/archi/7mostra/architetti/diller/open.htm.
[22] Raymond Wang, “Langham Place offices to roll next month,” The Standard (Greater China’s Business Newspaper), 19 June 2004 (www.thestandard.com.hk/thestandard/news_detail_frame.cfm?articleid=48588&intcatid=1).
[23] Robert Venturi, Complexity and Contradiction in Architecture (New York: Museum of Modern Art, 1966); Robert Venturi, Denise Scott Brown, and Steven Izenour, Learning from Las Vegas (Cambridge, Mass.: MIT Press,1972.)
[24] Robert Venturi, Iconography and Electronics upon a Generic Architecture: A View from the Drafting Room (MIT Press, 1996).
[25] Robert Venturi in a dialog with George Legrady at the Entertainment and Value Conference, University of California, Santa Barbara, May 4, 2002. The term I ‘information surface’ is mine.
[26] See http://prixars.aec.at/history/interactive/2000/E00int_01.htm.
[27] Ibid.
[28] See http://www.manovich.net/IA.
[29] See Ineke Schwartz, “Testing Ground for Interactivity: The Water Pavilions by Lars Spuybroek and Kas Oosterhuis,” http://synworld.t0.or.at/level3/text_archive/testing_ground.htm.
[30] Ibid.
[31] Otto Riewoldt, qtd. in Mark Hooper, “Sex and Shopping,” ID, The DNA Issue (2001), 94.
[32] For an insightful analysis of the branding phenomenon, see Naomi Klein, No Logo (New York : Picador, 2000).
[33] Reed Kram, personal communication with the author, June 5, 2002. For more Kram projects, see www.kramdesign.com/.
[34] Riewoldt, qtd. in Hooper, 2000.
