SIGGRAPH '23: ACM SIGGRAPH 2023 Emerging Technologies

We explored continuous changes in self-other identity by designing an interpersonal facial morphing experience where the facial images of two users are blended and then swapped over time. To explore this with diverse social relationships, we conducted qualitative and quantitative investigations through public exhibitions. We found that there is a window of self-identification as well as a variety of interpersonal experiences in the facial morphing process.

From these insights, we synthesized a Self-Other Continuum represented by a sense of agency and facial identity. This continuum has implications in terms of the social and subjective aspects of interpersonal communication, which enables further scenario design and could complement findings from research on interactive devices for remote communication.

We present RCSketch, the award-winning interactive system that lets anyone sketch their dream vehicles in 3D, build moving structures of those vehicles, and control them from multiple viewpoints. Visitors to this interactive showcase are able to use our system and design vehicles of their own and perform a wide variety of realistic movements across the vast digital landscape onboard their vehicles.

Origami offers an innovative way to implement haptic interaction with minimum actuation, particularly in immersive encountered-type haptics and robotics. This paper presents two novel action-origami-inspired haptic devices for Virtual Reality (VR). The Zipper Flower Tube is a rigid-foldable origami structure that can provide different stiffness sensations to simulate the elastic response of a material. The Shiftly is a shape-shifting haptic display that employs origami to enable a real-time experience of different shapes and edges of virtual objects or the softness of materials. The modular approach of our action origami haptic devices provides a high-fidelity, energy-efficient and low-cost solution for interacting with virtual materials and objects in VR.

We present an AI-mediated 3D video conferencing system that can reconstruct and autostereoscopically display a life-sized talking head using consumer-grade compute resources and minimal capture equipment. Our 3D capture uses a novel 3D lifting method that encodes a given 2D input into an efficient triplanar neural representation of the user, which can be rendered from novel viewpoints in real-time. Our AI-based techniques drastically reduce the cost for 3D capture, while providing a high-fidelity 3D representation on the receiver’s end at the cost of traditional 2D video streaming. Additional advantages of our AI-based approach include the ability to accommodate both photorealistic and stylized avatars, and the ability to enable mutual eye contact in multi-directional video conferencing. We demonstrate our system using a tracked stereo display for a personal viewing experience as well as a lightfield display for a room-scale multi-viewer experience.

We designed a VR controller to integrate experimental haptic technology into a practical controller. The device consists of two independent controllers, each with a weight-shifting module that can provide vibration, impact, and shape perception yet is sufficiently compact to be handled as a conventional commodity controller. Combining two controllers allows the device to be held differently for various applications.

Although there is no distinctive header, this is the abstract. This submission template allows authors to submit their papers for review to an ACM Conference or Journal without any output design specifications incorporated at this point in the process. The ACM manuscript template is a single column document that allows authors to type their content into the pre-existing set of paragraph formatting styles applied to the sample placeholder text here. Throughout the document you will find further instructions on how to format your text. If your conference's review process will be double-blind: The submitted document should not include author information and should not include acknowledgments, citations or discussion of related work that would make the authorship apparent. Submissions containing author identifying information may be subject to rejection without review. Upon acceptance, the author and affiliation information must be added to your paper.

Vertical force-feedback is extremely rare in mainstream interactive experiences. This happens because existing haptic devices capable of sufficiently strong forces that would modify a user's jump require grounding (e.g., motion platforms or pulleys) or cumbersome actuators (e.g., large propellers attached or held by the user). To enable interactive experiences to feature jump-based haptics without sacrificing wearability, we propose JumpMod, an untethered backpack that modifies one's sense of jumping. JumpMod achieves this by moving a weight up/down along the user's back, which modifies perceived jump momentum—creating accelerated & decelerated jump sensations. Our device can render five distinct effects: jump higher, land harder/softer, pulled higher/lower, which we demonstrate at SIGGRAPH 2023 Emerging Technologies in two jump-based VR experiences.

Untethered VR/AR HMDs can only last 2-3 hours on a single charge. Toward resolving this issue, we develop a real-time gaze-contingent power saving filter which modulates peripheral pixel color while preserving visual fidelity. At SIGGRAPH 2023, participants will be able to view a short panoramic video within a VR HMD with our perceptually-aware power saving filter turned on. Participants will also have the opportunity to view the power output of scenes through our power measurement setup.

A large percentage of people with autism or developmental disorders, which are mental disabilities, have sensory hypersensitivity. Therefore, the spread of “quiet rooms” in which they can feel at ease in social life is a necessary element in realizing a symbiotic society. However, the high cost of installing quiet rooms, which require highly soundproof rooms isolated from the outside, is an obstacle to their widespread use. The Inclusive Quiet Room is a new concept of portable quiet rooms that combines an easy-to-construct instant house, immersive videos, and relaxing sounds. In addition to enabling many people to experience the benefits of the room, the work proposes an image of the future quiet rooms that can be easily constructed anywhere. In this paper, we analyze the effectiveness of the Inclusive Quiet Room, exhibited in France, based on survey data from 372 respondents. Through the analysis, the relaxation effects and the demands for quiet rooms are substantiated. The room gives the feeling of being warmly embraced and secured. If all people including those without mental disorders could experience this embraced feeling, they would understand the need and benefits of relaxing environments for the people with sensory hypersensitivities.

We present LivEdge, a novel method for live stream interaction on smartphones utilizing electro-tactile sensation through the edges. Conventional interactions between users and a streamer on a smartphone are restricted to the streamer’s response through user comments or effects. Our goal is to provide a more immersive interaction through the use of haptic technology. LivEdge can convey spatial tactile sensations through electrical stimulations from electrode arrays affixed to both edges of the smartphone. This spatial tactile stimulus represents the streamer’s physical presence and movements in contact with the edge of the screen. Preliminary experiment showed LivEdge enhances the live stream experience.

We propose Material Texture Design, a material texture representation system. This system presents a pseudo-attraction force sensation in response to the user’s motion, and displays a shear sensation at the fingertips. The user perceives a change in the center of gravity from the shear sensation and feels the artificial material texture. Experimental results showed that the perceived texture could be changed by adjusting the frequency. Through demonstration, users can distinguish different textures such as water, jelly, or a rubber ball, depending on the frequency and latency. We propose this system as a small, lightweight, and simple implementation system for texture representation.

By manipulating light as a wavefront, holographic displays have the potential to revolutionize virtual reality (VR) and augmented reality (AR) systems. These displays support 3D focus cues for visual comfort, vision correcting capabilities, and high light efficiency. However, despite their incredible promise, holographic displays have consistently been hampered by poor image quality. Recently, artificial intelligence–driven computer-generated holography (CGH) algorithms have emerged as a solution to this obstacle. On a prototype holographic display, we demonstrate how the progress of recent state-of-the-art Neural Holography algorithms can produce high-quality dynamic 3D holograms with accurate focus cues. The advances demonstrated in this work aim to provide a glimpse into a future where our displays can fully reproduce three-dimensional virtual content.

This paper introduces a system that focuses on physics-based manipulation and haptic rendering to achieve realistic dexterous manipulation of virtual objects in VR environments. The system uses a coreless motor with wire as the haptic actuator and physics engine in the software to create a virtual hand that provides haptic feedback through multi-channel audio signals. The device simulates contact collision, pressure, and friction, including stick-slip, to provide users with a realistic and immersive experience. Our device is lightweight and does not interfere with real-world operations or the performance of vision-based hand-tracking technology.

Virtual reality (VR) passthrough uses external cameras on the front of a headset to allow the user to see their environment. However, passthrough cameras cannot physically be co-located with the user’s eyes, so the passthrough images have a different perspective than what the user would see without the headset. Although the images can be computationally reprojected into the desired view, errors in depth estimation and missing information at occlusion boundaries can lead to undesirable artifacts.

We propose a novel computational camera that directly samples the rays that would have gone into the user’s eye, several centimeters behind the sensor. Our design contains an array of lenses with an aperture behind each lens, and the apertures are strategically placed to allow through only the desired rays. The resulting thin, flat architecture has suitable form factor for VR, and the image reconstruction is computationally lightweight, enabling low-latency passthrough. We demonstrate our approach experimentally in a fully functional binocular passthrough prototype with practical calibration and real-time image reconstruction.

We develop a virtual reality (VR) head-mounted display (HMD) that achieves near retinal resolution with an angular pixel density up to 56 pixels per degree (PPD), supporting a wide range of eye accommodation from 0 to 4 diopter (i.e. infinity to 25 cm), and matching the dynamics of eye accommodation with at least 10 diopter/s peak velocity and 100 diopter/s2 acceleration. This system includes a high-resolution optical design, a mechanically actuated, eye-tracked varifocal display that follows the user’s vergence point, and a closed-loop display distortion rendering pipeline that ensures VR content remains correct in perspective despite the varying display magnification. To our knowledge, this work is the first VR HMD prototype that approaches retinal resolution and fully supports human eye accommodation in range and dynamics. We present this installation to exhibit the visual benefits of varifocal displays, particularly for high-resolution, near-field interaction tasks, such as reading text and working with 3D models in VR.

The physical world has contents at varying depths, allowing our eye to squish or relax to focus at different distances; this is commonly referred to as the accommodation cue for human eyes. To allow a realistic 3D viewing experience, it is crucial to support the accommodation cue—the 3D display needs to show contents at different depths. However, supporting the native focusing of the eye has been an immense challenge to 3D displays. Commercial near-eye VR displays, which use binocular disparity as the primary cue for inducing depth perception, fail this challenge since all contents they show arise from a fixed depth—ignoring the focusing of the eye. Many research prototypes of VR displays do account for the accommodation cue; however, supporting accommodation cues invariably comes with performance loss among other typically assessed criteria for 3D displays. To tackle these challenges, we present a novel kind of near-eye 3D display that can create 3D scenes supporting realistic accommodation cues in a single shot, i.e., without using time multiplexing or eye tracking. This display, which we present in our demo, can stream 3D content over a large depth range, at 4K spatial resolution, and in real-time. Our display offers an exciting step forward towards a truly immersive real-time 3D experience. Participants will get to enjoy 3D movies and play interactive games in their demo experience.

This paper proposes a wearable haptic device that utilizes control moment gyroscopes and a motion sensor to achieve somatomotor reconfiguration, altering the user’s somatic perception of their body. The device can manipulate sensations, making body parts feel heavier or lighter, and modify the ease of movement during interactions with objects. Given its potential applications in avatar technology, sports, and assistive technology, this proposed device represents a promising avenue for enriching the user’s bodily experiences.

Enhancing human capabilities through the use of multiple bodies has been a significant research agenda. When multiple bodies are synchronously operated in different environments, the differences in environment placement make it difficult to interact with objects simultaneously. In contrast, if automatic control is performed to complement the differences and to perform a parallel task, the mismatch between the user and robotic arm movements generates visuomotor incongruence, leading to a decline in embodiment across the body. This can lead to difficulty completing tasks or achieving goals, and may even cause frustration or anxiety. To address this issue, we have developed a system that allows a parallel operation of synchronized multiple robotic arms by assisting the arm towards which the user’s gaze is not directed while maintaining the sense of embodiment over the robotic arms.

We propose Transtiff, a stick-shaped device that can display various stiffness for stick-based haptic interaction. The device has a stiffness-changing joint replicating an artificial muscle mechanism in the relay portion of the stick to change its stiffness. Transtiff can be applied to touch interaction of the screen, augmenting the haptic experience of operating with a stylus pen, which is usually felt uniform. As applications, users can experience the sensation of pen and brush writing on a single device. In addition, it is possible to change the stiffness of the device for each object on the screen to reproduce the tactile feel of that object.