SixthSense: A Wearable Gestural Interface

This prototype is comprised of a pocket projector, a mirror, and a camera. The hardware components are contained in a pendant-like wearable device. Both the projector and the camera are connected to the mobile computing device in the user’s pocket. The projector projects visual information on walls and other physical objects, which become interfaces, while the camera recognizes and tracks the user's hand gestures and physical objects using computer-vision techniques.

SixthSense implements several applications that demonstrate the usefulness, viability, and flexibility of the system. The map application allows users to navigate a map (zoom in, zoom out or pan) displayed on a nearby surface with intuitive hand gestures. The drawing application lets the user draw on any surface by tracking the fingertip movements of the user’s index finger. SixthSense also recognizes user’s freehand gestures (postures). For example, the system implements a gestural camera that takes photos of the scene the user is looking at by detecting the "framing" gesture. For example, users can display and flick through photos on any surface or wall, draw icons or symbols in the air with their index fingers, select a magnifying-glass symbol to go to the map application, or draw an @ symbol to check their mail. The SixthSense system also augments physical objects by projecting more information about them as users interact with them.  For example, a newspaper can show live video news or dynamic information can be provided on a regular piece of paper.

Pranav Mistry 
Pattie Maes
MIT Media Lab