Next month, when media arts professor Phillip Baldwin and his creative technology classes performTimon of Athens at an off-off Broadway theater, all the world will be an interactive stage — and all the men, women, cameras, computers, and controller-free gaming devices will be players.
As student actors tread the boards in Manhattan's hip performance venue The Tank, a wheeling robot will follow them. Projected images will glide across the walls while Shakespeare's verses mingle with soundscapes produced by computer- or SmartPhone-controlled devices. It's 17th-century drama, newly recharged by the boundless imaginative possibilities of 21st-century technology.
“The devices add to the atmosphere,” says Baldwin, who uses the Manhattan space for his students’ final exam. “We want to make the performance more hybrid so that the audience will be more engaged. The challenge is that all these elements can upstage you. But I want to throw them together and see what works.”
The graduate students in Baldwin’s Art and Technology, Visual Interpretation, and Play Analysis classes are studying everything from pre-med to music, from nursing to theater. The one subject they have in common is the study of interface.
Using Kinect for Xbox 360 — a consumer-level device that allows users to move models, film, sound, and other elements simply by calibrating their bodies to its infrared camera — they are creating prototypes of devices that can be used not just for theater, but also for teaching, medical therapy, martial arts, and dozens of other applications. The Kinect’s camera actually reads the depth and position of a person’s arms and legs in 3D space using XYZ coordinates. Coupled with a brain sensor cap, its potential is astonishing.
"My students are hacking the Kinect so that human gesture will trigger film and sound," Baldwin says. "Two of them are using it for physical therapy for people with spine and head injuries. One student started the air conditioner in his room with his brain.”
At a recent lab night held in the Wang Center chapel, students demonstrated devices and applications that in other cirumstances might be used for patient education, yoga, even Twitter feed.
Tonight, however, is all about the Bard. Using the Ableton Launch Pad, a combination of software and hardware that allows him to input sounds and then recreate them improvisationally in a sound montage, Jay Loomis is preparing the background “music” for the play.
“There’s a part where Timon, who lives in a city surrounded by people, becomes disenchanted with humanity and goes off into the wilderness,” says Jay, who earned a bachelor’s degree in modern and ancient languages from Wheaton College and is pursuing a second degree in music in order to enter a Ph.D program in ethnomusicology.
“I wanted to reflect both aspects of his life. For his time in the city, I mixed in voices from audio books and NPR interviews and layered them with Charlie Parker jazz, rhythm and blues, and African drumming and singing. For the wilderness scenes, I used the sounds of crickets, thunder, rain, and wind.”
Senior Ryan Walsh, an Information Systems major, is working on three projects that will all be used in the performance. “One is a musical step-sequencer using the Kinect that lets the performer activate sounds within a grid by toggling the squares with his hands and feet,” Ryan explains. “Another is a projector on a rotating platform that I can control with the brain cap device, which is calibrated to my thoughts. And the third is a project similar to the step-sequencer, only the performer is controlling thumbnails of films." (See video embedded below.)
The practical uses of the Kinect are just being discovered. “We can develop formatted software so that patients can perform exercises where there are no nurses,” says Baldwin. “In the future, if you have some kind of injury, and you’ve got this Kinect, the hospital or insurance company can send you home with our software and ask you to go through certain motions.”
For Baldwin, one of the best parts of inventing creative technology is the collaboration among students with many different interests and talents. “I’ve got serious code students, pre-med students, and art students working together to make this magic happen,” he says with pride. The icing on the cake: last year his creative technology classes garnered glowing critiques from the judges at the Kennedy Center after being showcased in the regional American College Theater Festival award competition.
As for the Bard, he may not have imagined a Java-speaking device with omni-directional wheels and multiple camera positions playing the part of Timon. But he well might affirm the ongoing relevance of Hamlet's speech to the players: "Suit the action to the word, the word to the action."
- Toby Speed
