SIGGRAPH Emerging Tech.: Special Interest Group on Computer Graphics and Interactive Techniques Conference Emerging Technologies

Single-photon sensors are a novel class of imaging sensors that are sensitive to the individual arrival of photons. In recent years, single-photon sensors have witnessed rapid growth in key sensor characteristics such as array resolution, which, for the first time, has made computer vision and imaging with these emerging sensors possible. This rapid growth has also spurred the development of algorithms that can harness the rich spatio-temporal scene information captured by single-photon sensors, which has led to several new imaging capabilities. These capabilities include imaging in challenging SNR conditions, high-dynamic range imaging, compensating extreme camera motion extents, and even providing the functionality of numerous imaging systems via post-capture processing. Our demo will showcase these exciting capabilities to a wide computer vision and graphics audience, and in doing so, make a case for the mainstream adoption of single-photon technology.

As spatial computers move towards glasses, where constraints on size, weight and power are critical, a shift in how we use sensors is needed. Ultraleap has developed the first hand tracking pipeline using event cameras on an AR headset. Event cameras allow to break the frame-imaging paradigm of "low power equates to lo sample rate" and enhance user experience while maintaining low power budget.

We have developed an innovative lighting system that changes specific target colors while keeping the lights appearing naturally white. By precisely controlling the spectral power distribution (SPD) of illumination and harnessing the unique phenomenon of metamerism, our system achieves unique color variations in ways you’ve never seen before. Our system calculates the optimal SPDs of illumination for given materials to intensively induce metamerism, and then synthesizes the illumination using various colors of LEDs. We successfully demonstrated the system’s implementation at Paris Fashion Week 2024. As models step onto the stage, their dresses initiate a captivating transformation. Our system altering the colors of the dresses, showcasing an impressive transition from one stunning color to another.

Real-time and slow-motion are basically incompatible properties in the technologies of recording and playback. We propose a framework to realize the coexistence of both using parallel video presentation based on the integrative capacity and the temporal nature of cognition. In this framework, camera input is divided into short time durations, distributed into layered timelines, and then stretched out. And then, all layers are composited and presented in a display simultaneously. In our demonstration, users can experience their surroundings through see-through goggles or digital mirrors in which this framework is embedded. Under certain conditions, the perception of the layers is integrated into the cognitive process, resulting in the coexistence of real-time and slow-motion. This research suggests a new field of “Temporal Editing to Humans.”

We aims to computationally replicate the dynamic caustics observed in nature, such as the captivating light patterns found at the pool or within flowing rivers. Traditional computer science approaches have only been able to reproduce these light patterns through static fabrication. We introduce “Dynamic Acousto-Caustics” a method that merges acoustofluidics with optics to dynamically manipulate light caustics by controlling the shape of liquid surfaces. Using computer-controlled acoustic fields to deform the surface of a liquid medium, we generate dynamic light behaviors in fluid patterns unachievable with static refractive surfaces. This research not only extends the understanding and application of controlled caustics across various technical and creative domains but also exemplifies the intersections made possible at the convergence of multidisciplinary research. While devices exist to generate waves by vibrating water surfaces, they could not produce continuous caustics at specific locations and timings. Employing ultrasonics to deform the water surface allows us to directly manipulate the surface to create more continuous animated patterns. We shape caustics through acoustic field manipulation and optimize the visual outcome emanating from the geometry of these manipulated objects within a dynamic system. Our approach leverages the Digital Twin methodology as an optimization strategy to fine-tune the interplay between sound and light. This enhances the understanding and application of controlled caustics in numerous technical, demonstrating the crossing points enabled by the convergence of interdisciplinary research streams.

This study introduces EmBelt, an innovative haptic device integrated with VR storytelling that creates an extended reality (XR) experience designed to the enhance understanding of eating disorders (EDs) related to body image concern (BIC) among individuals without such anxieties. The system dynamically adjusts compression levels in response to user interactions, providing a portrayal of the pressures linked to the thin ideal and extending related topics. Future directions include potential platform expansion to reach a broader audience and applications in medical treatment, underscoring EmBelt’s potential as a tool for various intervention uses.

FEEL TECH Wear is a system that facilitates haptic interactions while keeping most of the palm free, by presenting directional force through rotational skin-stretch distribution feedback to the wrist and providing texture sensation through vibration feedback to the fingertips. With advancements in hand tracking and passthrough technologies, hand interactions in Mixed Reality (MR) environments have become more accessible, necessitating palm-free haptic feedback methods that do not hinder interactions with real objects or impair vision-based hand tracking. The hardware of FEEL TECH Wear primarily consists of two components: a hand-mounted device for each hand and a control unit located at the back of the head. The hand-mounted device is equipped with four channels of rotational skin-stretch tactors at the wrist and vibration tactors at the thumb and index finger. Using FEEL TECH Wear, three applications have been realized: haptic feedback for virtual objects, haptic augmentation for real objects, and haptic guidance towards objects.

We demonstrate an ultrasound haptic-based mid-air thermo-tactile display system, designed as a proof-of-concept with an open-top chamber, heat modules, and an ultrasound haptic display. Our method involves directing heated airflow toward the focused pressure point produced by the ultrasound display, delivering thermal and tactile cues in mid-air simultaneously. We showcase our system across four different VR environments—campfire, water fountain, kitchen, and candle—to illustrate the rich user experiences that result from integrating thermal and tactile feedback.

Vibrotactile interaction is an essential source of HCI and should be on a large scale, such as room scale, whereas in the past many devices have been small. However, there is a practical difficulties for implementing vibrotactile actuators and sensors to every furnitures in the room. To address this problem, our approach, HaptoRoom, archives reconfigurable ubiquitous haptic interaction to existing furniture without additional equipment by installing actuators only in the floor. A user can design detailed spatial distribution of vibrotactile pattern, leveraging with multi-point individual control of actuators array inside floor interfaces.

Cutting-edge computer-generated holography (CGH) algorithms have realized significant advancements in reconstructing the desired light field, enabling the creation of 4D light field holograms. These holograms offer view-dependent effects, such as motion parallax, occlusion, and specular reflections, which are crucial for enhancing the perceptual realism of 3D scenes. Our holographic display prototype leverages sophisticated CGH algorithms in conjunction with a high-speed binary spatial light modulator (SLM) to deliver an immersive experience of 4D light field holograms. This demonstration highlights the latest progress in CGH technology and emphasizes the critical role of parallax cues in providing perceptually realistic 3D contents.

Advancements of multimodal video content, interactive gaming, and virtual reality experiences frequently employ motion platforms to authentically replicate the dynamics of ground swaying and tilting. However, they are expensive to install and require significant space. In response to these constraints, we propose a method that utilizes a kinesthetic illusion. This technique involves the selective application of vibratory stimuli to the ventral and dorsal tendons of the ankle, thus engendering a perceptual sensation of ground sway. Additionally, integration of these vibratory stimuli with continuous visual inputs facilitates the simulation of ground tilt. The generation of these sensations is exclusively through vibratory means without any actual ground movement. This feature significantly enhances the safety and ease of installation, thereby presenting a viable and efficient alternative to conventional motion platform setup.

We present a system that enables multiple users to share the viewing experience of mirror-transcending aerial imaging, in which virtual objects continuously move between mirrored and physical spaces. The system consists of an aerial-imaging optical system with a wide viewing range and a control system that synchronizes multiple display devices’ position and image rendering. We implemented a prototype to display the mirror-transcending aerial image for multiple users. The prototype looks like a three-sided mirror, allowing four to five users to participate in the viewing experience and share their impressions.

Did you already imagine how would it be to watch a sport match without sounds? You would miss all this specific sport related sounds but also mostly miss a big part of the atmosphere present in the stadium, that is particular to live events. This is what happens to most Deaf and Hard of Hearing persons. Towards Tokyo 2025 Deaflympics, we developed an AI-based system able to recognize sounds and players motion to render in real time sound related Onomatopoeia over the match video as one could see in Comics or Manga.

We present the latest development of Nukabot, a human-computer interaction mediation system that connects the fermented food microbiome with its human carer through voice communication. A nukadoko is a traditional Japanese fermentation technique that involves rice bran mixed with salt and water (nuka) put in a wooden, enamel, or plastic container (doko). The current version of Nukabot is a porcelain nukadoko equipped with chemical sensors to track the fermentation status, voice recognition, and vocal synthesis engines that enable humans to talk and ask questions about the ferment.

Experience Project Starline, the first photorealistic telepresence system that demonstrably outperforms 2D videoconferencing systems, as measured by participant ratings, meeting recall, and non-verbal behaviors [Lawrence et al. 2021]. Our prototype communication endstations combine state-of-the-art face tracking, real-time neural view synthesis, spatial audio, multi-stream compression, and a high-resolution auto-stereoscopic display to produce a striking and natural sense of co-presence between two meeting participants that is not possible with traditional videoconferencing.

Robots consisting of many articulated parts performing complex movements are challenging to design. We showcase an interactive system for exploring shapes and structures of robots through 3D sketching, generating plausible movements of robots through AI, and reviewing and refining them in VR. Such immersive prototyping in the early stages can help reduce the time and cost associated with trial and error in later stages, contributing to shortening and streamlining of the robot development process.

Tweezers are used to carry out crucial functions in various research fields, most notably biology. Thus, proficiency in using tweezers impacts experimental outcomes. However, becoming proficient in using tweezers under expert guidance is not always easy for beginners because the force applied to the tip of a tweezer is virtually imperceptible. To help learners, we propose SkillPicker, which provides the user with images taken at the tip of their tweezers and measures the pinch force simultaneously. Thus, experts can record their technique using SkillPicker, enabling beginners to practice autonomously. SkillPicker is equipped with a small magnifying camera at the fulcrum of the tweezers and a pressure sensor to record the pinch force applied by the user’s thumb. We establish an empirical correlation between the pressure applied and the pinch force at the tip, thereby enabling the estimation of the pinch force applied at the tip of the tweezers. SkillPicker also enables pressure values to be visualized or converted into musical notes on a scale to promote instant comprehension. This system facilitates the recording and sharing of tweezer operation, enabling beginners to learn dexterity both visually and physically.

We demonstrate an interactive 3D data visualization tool called TangibleData. TangibleData adapts hand gestures and mid-air haptics to provide 3D data exploration and interaction in VR using hand gestures and ultrasound mid-air haptic feedback. We showcase different types of 3D visualization datasets with different data encoding methods that convert data into visual and haptic representations for data interaction. We focus on an intuitive approach for each dataset, using mid-air haptics to improve the user’s understanding.

The Malleable-Self Experience comprises the integration of the visual element of virtual reality (VR) with the whole-body haptic sensations of the Synesthesia X1 haptic chair. The goal is to induce a provocative experience that expands one’s understanding of the self by creating a malleable perception of the body image. We explore the effects of visual and whole-body haptic integration on augmenting body image during dynamic transformations of visual representations of the body in VR. We design the plausibility of these perceptual augmentations using a specific sequence of multisensory events: (1) establishing body ownership of a virtual body anchored in the same self-located space as the participant, (2) separating the virtual body to hover above the participant’s physical body, enhanced by accompanying haptic stimuli to increase proprioceptive uncertainty, and (3) transforming the virtual body with integrated visuo-haptic stimuli to sustain perceptual congruency.

We have developed a display system that achieves to render palm-sized volumetric contents in real space. The system consisted of two optical paths for laser rendering, in which femtosecond-laser-excited emission points were scanned by a liquid-crystal spatial light modulator and a 3D laser scanner. A single volume was formed by the cooperative dual-path laser rendering based on the optimal design of the optical system and scanning path. Volumetric contents such as volumetric graphics and animations with well-filled voxels were demonstrated in a centimeter-sized volume.