BiDi Screen: A Thin, Depth-Sensing LCD for 3D Interaction Using Light Fields

An LCD screen is transformed into a BiDi (bidirectional) screen to support 2D multi-touch and walk-up 3D gesture interaction. An image sensor placed a small distance behind an LCD forms a mask-based light-field camera, allowing passive depth estimation. The BiDi screen also supports novel mixed-reality rendering with external light-emitting widgets that light a virtual scene.
 
The BiDi Screen, inspired by emerging LCDs that use embedded optical sensors to detect multiple points of contact, is capable of both image capture and display. The project explores the spatial light-modulation capability of LCDs to allow dynamic mask-based scene capture without interfering with display functionality.
 
The system alternately switches between a display mode showing traditional graphics and a capture mode in which the backlight is disabled, and the LCD displays a pinhole array or an equivalent tiled-broadband code. The BiDi screen captures an array of images equivalent to that produced by an array of cameras spanning the display surface. The recovered multi-view orthographic imagery is used to passively estimate scene depth, supporting real-time 3D gesture interaction.

Matthew Hirsch 
MIT Media Lab

Douglas Lanman 
Brown University

Henry Holtzman
Ramesh Raskar 
MIT Media Lab