Creative Kinect hacking

When Microsoft’s new motion-capture sensor for the Xbox 360 brought Minority Report-style control to the living rooms of delighted gamers in late 2010, it took things up a notch. Once confined to big-budget studios, academia and the military, motion capture technology was finally at everyone’s fingertips.

“The Kinect is a very special device,” enthuses Memo Akten, a Turkish-born, London-based visual artist, director and musician whose clients include Coca-Cola, Google and Sony PlayStation. “What it brings to the world of computer vision isn’t technically new, but it’s an awful lot cheaper than its predecessors, with comparable – if not better – quality.”

Akten has years of experience at the cross-section of art and technology. “Now kids all over the world can play with cheap DIY 3D scanning, point cloud extraction – even rough motion capture,” he continues, picking out several applications that have caught his eye – from gesture-controlled quadcopters to abstract visual interpretations of dance. “By reducing the technical entry point, the Kinect has levelled the playing field and allowed a lot more creative experiments to surface.”

Crucially, the affordability and relative ease of use enables designers to concentrate on the creative process from the start, rather than grappling with the technical functionality. “The possibilities are less about technology, and more about imagination,” confirms Daniel Soltis, technical lead at Moving Brands, who first gave in to his inner hacker when experimenting with Arduino and Processing back in 2006. “I find hacking exciting because people define their own possibilities, rather than simply accepting what has been packaged up,” he enthuses. Although Microsoft could have tried to use its global clout to clamp down on hacking, Soltis celebrates the company’s open-minded attitude to people adapting the device to suit their own ends.

The availability and accessibility of open-source drivers got French-born, London-based designer Paul Ferragut hooked from the outset: “Everyone was excited about the new field of possibilities,” he recalls. “And soon, new motion-sensing devices with higher resolution will be released that will enable us to track smaller-scale movements, such as those of lips or fingers.”

As Ferragut points out, being able to track the position of a body in a space opens up possibilities for interactive design. “Coordinates can be used to trigger content,” he observes. “The geometry of a space can also be used to improve new techniques such as augmented reality or projection mapping.”

As an example, he draws attention to Project Eden by Patricio Gonzalez Vivo, which uses a Kinect to map the relief of sand in a sandbox. When an image of islands surrounded by blue sea is projected onto the sand, it becomes a miniature landscape that you can shape and reshape with your hands.

While the barriers to designers experimenting with this kind of abstract application have dropped away, major studios are also making time to explore the potential of the Kinect. Nick Hardeman is a new media artist, designer and experimenter in the R&D lab at Arnold Worldwide, whose clients include Dell and Jack Daniel’s. Hardeman’s department is tasked with feeding research back into other Arnold projects. Among the Kinect projects that have caught his eye are SwimBrowser, a web browser that enables you to “swim” through the internet; and ‘The Treachery of Sanctuary’ by Chris Milk, an interactive installation that transforms your arms into wings, before a flock of birds attempts to tear away your shadow .

Moving Brands’ design director Campbell Orme has worked with the Kinect since its inception, but has been involved with gesture-based projects since 2004, then using JesterTek and Director. “It was great at the time, but a Kinect is miles ahead in terms of cost, flexibility and community contribution,” he says.

Daniel Soltis says his early Arduino work was similar: “I spent a lot of time working with wireless communication, GPS and accelerometers,” he remembers. “Nowadays, all that’s built into smartphones, and it’s easier to build an app than to try to replicate that hardware in prototype form.”

James Patten, founder of graphic and interactive design outfit Patten Studio, was also an early adopter: “Leaving work one Friday, a friend suggested that instead of taking it easy over the weekend, I should go buy a Kinect and see what I could do with it,” chuckles the MIT Media Lab graduate, who has exhibited and performed in venues across the globe, including MoMA and the Guggenheim. “At this point, it had just gone on sale and the open-source drivers were literally hours old and barely worked,” he continues. “I spent the weekend hacking and built a software tool that would enable you to track touch points on any flat surface the Kinect could see.”