SA '24: SIGGRAPH Asia 2024 XR

Traditional design processes for physical prototypes can be time-consuming and costly due to iterations of prototyping, testing, and refinement. Extended Reality (XR) technology with video passthrough offers unique benefits for alleviating these issues by providing instant visual feedback. We have developed an XR system with multimodal input capability that provides annotations and enables interactive visual modifications by superimposing and aligning visual counterparts to physical objects. This system can help designers to quickly experiment with and visualize a wide range of design options, keep track of design iterations, and explore innovative solutions without the constraints of physical prototyping. As a result, it can significantly speed up the iterative design process, while requiring fewer physical modifications in each iteration.

Previous studies have explored various forms of virtual companions, including displaying companions on flat screens, headsets, or the embodiment of their physical bodies. Although most studies use additional devices to enrich sensory feedback to embody the virtual companion in a virtual reality (VR) headset, it requires electricity to activate. We present “Petroller,” an attachable mechanism that embodies a virtual companion by altering the built-in VR controller with the prop-based haptic, allowing users to hug, pat, and exercise together. Additionally, we designed two usage scenarios: pet companionship and rehabilitation exercises. We aim for users to interact with the Petroller by pressing and stroking, providing relaxation, emotional healing, and even promoting physical exercise.

Many studies have explored weight feedback in virtual reality(VR), but few can provide sufficient full-body weight sensations. We present Dragravity, a dynamic weight-shifting haptic proxy using wearable, reconfigurable waterbags. Our waterbags can be worn on arms, lower body, or legs using elastic bands to distribute weight across different body parts. This enhances the user’s full-body weight sensation, enabling application possibilities with the haptic proxy. The waterbags can simulate weight sensation in VR using water with a maximum additional weight of 10.4 kg. To demonstrate, we designed two applications. The first is planetary exploration, where users can experience gravitational sensations on planets with the dynamic full-body weight sensation. The second is fitness weight training. By leveraging dynamic weight, users can experience various types of weight training on different muscles.

EchoVision is an interactive mixed reality art experience that simulates bat echolocation using custom-designed, handheld, bat-shaped masks based on the open-source HoloKit headset. As users initiate sounds, they experience echolocation—the remarkable navigation method bats employ in darkness—through visual representations of propagating echoes mapped onto the real-world 3D shape of the surrounding environment, scanned in real-time by the LiDAR sensor of a smartphone mounted on the masks. Unlike traditional indoor mixed reality headsets that often require time-consuming per-person setup and calibration, EchoVision’s ergonomic handlebar design allows for easy passing and sharing among numerous participants, enabling quick engagement with the experience. This feature is especially valuable for in-the-wild, large-scale, pop-up exhibitions in bat habitats, such as Austin’s Congress Avenue Bridge, enhancing high-traffic scientific education through deeper empathetic engagement. By providing a unique glimpse into how non-human creatures perceive their sensory world, EchoVision promotes an ecocentric design perspective, fostering interspecies understanding and appreciation.

VR collaboration enables users to express their emotions through visual cues. Although various techniques for representing emotions have been proposed, the impact of different representation locations on their effectiveness remains unclear. This research investigates how reprojecting a collaborator’s emotions to various locations can enhance the reception of affective cues in VR collaboration. We introduce ExpressiveWorld, a customizable collaborative space that allows VR users to configure both the type and location of their partner’s emotion representations. Our system provides four representations methods: 3D head models, emojis, visual effects, and vibrotactile feedback, representing five distinct emotions: Neutral, Happy, Angry, Sad, and Surprised. These representations can be reprojected onto the user’s or partner’s hand and task objects within the VR environment. With the ability to customize their emotional representations and positions on or off independently, users can create up to 84 different configurations. This flexibility helps VR users maintain focus on tasks while achieving emotional understanding and consensus with their partners.

This study explores the multi-dynamic gestural interaction system using VR headsets and wearable devices, for enhancing VR painting with pressure control, gesture recognition, and haptic feedback. Testing demonstrated novel visual expressions through hand gestures, suggesting the potential for more nuanced and dynamic interactions in VR art and beyond.

Terrain generation and authoring in Virtual Reality (VR) offers unique benefits for terrain authoring including stereo display, immersive and intuitive design experience, and natural input modalities. We present this VR-based terrain fast prototyping system to integrate natural input modalities, preserve artistic controls and lower the effort of landscape prototyping. The system utilizes freehand interfaces and a generative model to help users quickly prototype different types of natural landscapes, such as mountains, mesas, canyons, and volcanoes. With the freehand interfaces, users can use their hands to draw mid-air strokes as the 3D contours of the desired landscapes. Then, a Conditional Generative Adversarial Network (CGAN) generates realistic landscapes based on the 3D contour. The freehand interface detects users’ gestures to control landscape editing. By incorporating CGAN as the terrain synthesizer, our system allows users to immersively, rapidly, and easily prototype terrains using intuitive interactive hand controls.

GravField (short for "Gravitational Field") is a live-coding intercorporeal improvisation system within a collocated Mixed Reality (MR) environment. It explores the dynamic interplay between bodies, mediated through live-coded MR audiovisual affordances. Inspired by contact improvisation techniques, the system offers several MR audiovisual affordances that drive different modes of intercorporeal interactions. For example, "Spring" shapes spatial relations through proximity, "Rope" fosters bodily synchronization, and "Magnetic Field" captures emotional responses of attraction or repulsion. The landscape of these affordances, dynamically adjusted by live coders, actively influences participants’ collective movements, which in turn alter the live music generation, creating a feedback loop. Drawing on Merleau-Ponty’s phenomenology, GravField illuminates the potential for future somaesthetic interplay systems in holistic interactive MR environments.

“Into the Womb” is an XR content with a simulated experience of being born again. The author who has a developmental disability expresses her hope that if she could be reborn without her disorder, she would feel universal love. The work raises social issues about women with developmental disabilities, who have lost love, which is the most important human emotion, due to trauma.

The protagonist, who suffers from a developmental disability at the beginning, is verbally tricked by people and expresses her difficulty in living, having been traumatized by love and other things. The story begins with a scene of the world from the perspective of a girl with developmental disabilities in a night city called Kabuki-cho in Shinjuku, Tokyo. Participants will see the transition from the dark night city back into the womb of their mothers, experiencing a simulated escape from a society filled with contradictions. They can hear the mother’s words and the encouraging words around them, which confirm their love. Then, they realize that they are being born again by moving their hands inside the womb to be born again. In the end, they assimilate with the fetus and are enveloped in light, and they are born again and ’alive’ in the world. When they finish the experience and remove the headset, they feel a world full of hope after they have been born again.

This work aims to help people feel free from trauma and have heart-warming feelings by going through a simulated experience of being born again.

Our project establishes digital 3D models of artifacts through digital scanning technology and develops proportionate interactive devices that can interact with projection images using spatial location technology. The project aims to commemorate the Southward Evacuation of the artifacts led by the Palace Museum in China during World War II. By allowing visitors to physically touch the artifact installations for experiential purposes, it breaks the traditional exhibition limitation of only being able to view artifacts from a distance. This approach enables audiences to gain a more comprehensive understanding of cultural heritage through a multi-sensory experience, thereby encouraging reflection on the value and significance of artifact preservation.

“MeiMeiRoRo" is an interactive experience in a nested maze, and its concept is to present two consciousnesses, active and passive. In this experience, operations to a miniature avatar, such as tilting, are returned to the user through a nested structure. Operations on the miniature avatar are an active awareness, and when the resulting operations are returned to the user, a passive awareness occurs. These two consciousnesses are the concept, which works because the nested structure implies that the operations on the miniature avatar are an operation on him/herself.

NatureBlendVR is a hybrid interactive experience designed to promote wellbeing by leveraging the benefits of forest bathing. The system combines XR technology with interactive bio-responsive physical elements. Throughout the experience, users are immersed in a hybrid space where elements of a physical garden are placed in the immediate surroundings of the user, including a flower sphere that provides haptic and visual biofeedback reacting to the person’s heartbeat. A gradual transition from the view of the real space to a meticulously crafted virtual forest where digital elements match the position of physical ones promotes a sense of continuity, allowing the person to feel present in the natural space as embodied sensations match the expectation created by virtual interactions.

The sense of ownership over non-human body parts, such as tails and third arms, has been explored using Virtual Reality (VR) technology. These studies are grounded in evolutionary backgrounds and anatomical associations. However, whether the body model can be extended to imaginary body parts without such evolutionary and anatomical associations remains unclear. In this demonstration, we developed a device that creates a sense of ownership over imaginary body parts, specifically cat ears. We achieve this through haptic feedback to the head via hair. Using soft actuators made of shape memory alloys, we move the hair to reproduce various haptic sensations and the comfort associated with stroking cat ears, including the range and direction of the touch. Participants transform into an avatar with cat ears and experience having their imaginary cat ears stroked.

Autobiographical memory recall can be triggered through sensory stimuli, often creating a positive impact on emotional well-being. However, there are few systems that can adapt to users’ emotional state and provide personalized stimuli to better support this process. Re-Touch enhances memory recall through the integration of Virtual Reality (VR), haptic feedback, and affective computing. It allows users to revisit their personal memories in a controlled VR space. By monitoring physiological responses (photoplethysmography (PPG) and electrodermal activity (EDA)) the system adjusts the haptic feedback intensity and the VR content to match the user’s emotional needs, enhancing sensory and emotional immersion.

Cinematic virtual reality (CVR) brings viewers an immersive movie-watching experience. Different from storytelling of other media, CVR directors will use different interaction techniques and visual cues to guide the viewers through the immersive experience. In this paper, we present Reyeal, an implicit gaze-based interaction for creating personal landscape painting in cinematic virtual reality, in which the users will have their own experience of immersive landscape painting through their eyes. While most of the visual cues rely on visible objects, with Reyeal (Reveal by Eye), the CVR directors do not need to design interactions to guide the users looking at specific target objects. To demonstrate our technique, different elements of a Chinese landscape painting such as clouds and mountains will be drawn to the virtual environment by the users’ gaze accordingly, which will result in a compelling experience for each user.

This project aims to build a sense of connection in a remote two-player VR game by offering proximity-based interaction with audio and video connectivity. We develop a co-dependent game system where virtual proximity, or the distance between two players are assumed to reflect emotional closeness of the two individuals. The VR metaphorically reflect and promote players’ mutual engagement. Positioned at opposite ends of the virtual environment, the two players exchange audio and visual signals to locate each other until they come close. As they get nearer, visual and audio elements, such as color saturation and diverse sounds, are designed to enhance their connectivity, reflecting their emotional closeness and awareness of each other’s presence. After gameplay, the players’ locations are recorded and displayed on a map.

Slime Hand XR is an interactive head-mounted display (HMD) system that induces illusory skin deformation in VR. In the proposed system, participants wearing the HMD place their hands on a tray where a chunk of slime is visible within the HMD’s view. Participants can instantly and boldly feel illusory skin stretching by looking at the deformed slime while the experimenter pinches and pulls their hand’s skin at the same time. In addition, the tip of the deformed slime can be clipped with a clothespin in VR by the experimenter. Then, this setup can maintain the form of illusory skin deformation. This system adopts the principle of the slime hand illusion, which can induce a strong sense of skin deformation. By using computer-generated slime, we achieved a unique skin deformation illusion that is hard to replicate in reality, such as suspended deformation. In testing the illusion effect of this system at an open laboratory exhibition for children, 56 out of 63 participants reported a strong reaction to the sensation of skin deformation in the questionnaire.

Chakra, originating in Hinduism, is described as luminous wheels or auras representing different elements within the body that reflect a person’s self-awareness. Chrakra-based meditation techniques can support individual wellbeing, but the provision of Chakra feedback in a visual and tangible manner to facilitate mindfulness is largely unexplored. Transcendental Chakra is a virtual reality (VR) multi-sensory experience using guided audio, visual effects, and vibrotactile feedback to aid beginners in chakra meditation by visualizing their astral avatar. The goal of this work is to foster both spiritual and physical self-awareness.

There are many studies on hand tracking and interaction with virtual puppets in the immersive experience. This research presents an advanced gesture recognition system called Theater of Future Puppetry. Unlike systems that only use hand tracking and hand skeleton manipulation of puppets, this system can recognize specific gestures used in traditional Budaixi (Taiwanese glove puppetry) and interact with them. The gestures were designed based on discussions with traditional Budaixi masters. Through this system, users can learn and understand the knowledge and cultural background of Budaixi puppet manipulation.

This research proposes a wireless foot vibration device designed to enhance virtual walking experiences in virtual environments. The device, constructed from snowboard bindings and 3D-printed parts, features vibrators and pressure sensors to provide tactile feedback and control walking direction. Users can navigate virtual environments, created in Unity, by shifting their weight without moving their legs. This system provides a realistic walking sensation through synchronized foot vibrations corresponding to various ground textures. The compact design offers high immersion and freedom, suitable for use in standing, seated, or lying down postures, overcoming physical constraints of traditional systems.

We present the first showcase of Zenbu Koko, a XR meditation platform, developed by All Here and designed in collaboration with Kengo Kuma and Associates. This platform presents a 5 minute immersive experience that guides participants on a journey focused on self-awareness and inner sensations. It features immersive visuals and audio, meditation guidance, haptic feedback, and a tool for reconstructing and displaying participants’ bodies in virtual environments using depth sensor cameras. Originally conceived as a project bridging neuroscience and meditation, this platform has been accepted for a talk presentation at SIGGRAPH Denver in 2024. We aim to provide a first public demonstration here.

This paper introduces "MONOLITH", a non-linear, multi-threaded narrative film based on VR headset technology. Viewers immerse themselves in the VR experience from a first-person perspective, navigating through time and space to witness stories triggered by a monolith amidst the evolution of civilization. The film uses a multi-threaded structure. The overt thread highlights key events and character interactions with the monolith across different periods, while the covert thread reveals the protagonist’s background and identity metaphors. In this science fiction story, viewers explore the complex relationships between time, consciousness, and human nature.

In A Vocal Landscape our aim has been to create a VR experience that feels familiar and relatable—both visually and in the way the story's narrative unfolds. Our approach in storytelling has been minimalistic, instead of using big theatrical dialogues or trying to save the world with a VR film. Instead, we focus on a more niche and complex subject matter: human communication. We have kept the dialogue naturalistic and sounding like everyday words— something we rarely see in VR, and the story has a simple structure: it's a late-night conversation between two people and each person's association arising from what's said and what's heard.

East Beijing Road is an asymmetric VR installation that reimagines the stories of a building with a history spanning over 100 years in Shanghai, China, whose residents were compelled to move out due to an ongoing gentrification policy. The project addresses the perceptions of the past and present, the fluidity of archives, and the meanings of ruins. A synthetic space was reconstructed based on 3D photogrammetry models captured in this building. The audience, both inside and outside VR, must use distinct interfaces and experience different visual presentations to collaboratively unfold the story. Old and new, digital and analog, real and fictional, and virtual and physical elements interweave. The asymmetrical design in interactive storytelling blends different times and spaces, allowing the re-enactment of the past through audience collaborative actions, raising questions about the relationship between interactivity and the visibility of the past. Asymmetry VR can be defined as “co-located users access the same virtual environment using different kinds of technology” [Ouverson and Gilbert 2021]. This project deploys the characteristic of asymmetric VR and provides an experience of unstable, unbalanced, and incomplete information among audiences inside and outside VR. The imbalances and incompleteness in media design are a metaphor for the fragmented time and space inherent in the archival practice of unveiling the past in the digital epoch. No complete information is visible without collaboration between the audiences inside and outside VR. It is an embodied experience that the past can only be revealed through collaborative interaction. The story of the building on East Beijing Road becomes a form of repertoire because of the collaborative engagements of the audiences.

Empereur is an evocative and interactive virtual reality narrative that immerses users in the inner world of a father suffering from aphasia, a condition that severely impairs verbal communication. The story is told from the perspective of his daughter, who seeks to reconnect with him by piecing together his fragmented language and memories. Through hand-tracked interactions, users are invited to participate in the father's attempts to communicate, creating a narrative experience that transcends traditional storytelling formats.

Virtual reality plays a crucial role in enabling a fully immersive exploration of memory and language, offering a level of engagement that other formats cannot achieve. The experience is heightened by its poetic, black-and-white aesthetic and innovative hand-tracking system, which fosters a deep emotional connection between the user and the characters.

This essay explores the interactive elements and design choices that make Empereur a pioneering achievement in immersive storytelling. It also examines how the project leverages VR's unique capabilities to evoke empathy and engage audiences on a deeper emotional level, contributing significantly to the field of interactive media.

At 2:46 PM the lives of Hidenori, Mao, Kinue and thousands of others changed, irreversibly. The Fukushima Daiichi accident on the 11th of March 2011 is the worst nuclear power plant accident after Chernobyl.

Fukushima – The Home That Once Was is a free-movement virtual reality documentary. The viewer can move freely inside the realistically captured 3D-houses and down the town streets inside the Fukushima Daiichi's off-limits Exclusion Zone using their VR-headset.

The viewer will meet six former residents, who share their memories about losing their home, about the importance of community and of their special connection to the land. In the film there is around 1,5 hours of content, but the viewer can stay in the experience only up to 20 minutes. Thus each viewing will be different and each viewer will see a different version of the film.

"Magic You" is a virtual reality interactive narrative experience, a coming-of-age personal story about ADHD, with joys, anxieties, bitter memories, and growing pains, told in a hand-drawn, colored-pencil aesthetic as a magical journey with surreal overtones, providing a more in-depth look into the inner world of people with ADHD. The artwork hopes to present the experiences and imagination of ADHD patients to the audience in poetic and romantic way, make audience spiritual healing and empathizing.

Bandai Namco Filmworks is pleased to announce that it will produce "Mobile Suit Gundam: Silver Phantom" the latest pioneering VR film based on the "Mobile Suit Gundam" series, with Atlas V, a VR production company that has created numerous immersive experiences, including "Gloomy Eyes" and "Battlescar. "Mobile Suit Gundam: Silver Phantom" is a visual work that offers fans the opportunity to immerse themselves in the Gundam world in a new way using VR and invites them into an unprecedented interactive story. This work will be distributed by Astrea, a company that has developed numerous VR titles, and will be available on Meta's VR headset, Meta Quest. Through this new VR experience of the Gundam world, we will deliver the appeal of Gundam to fans around the world.

No Exit in Hell is an innovative recreation of Jean-Paul Sartre’s classic play No Exit in VR theatre. It immerses viewers in a novel narrative experience by not only bringing realistic theatre acting into virtual reality but also allowing them to take on different roles of the play. In this role-switching experience, gamification designs are integrated to guide viewers through each role transformation and maintain their focus, ensuring that every shift in character unveils a new facet of the story.

Once a Glacier tells the story of a relationship between a girl and a glacier. As the girl grows older, the existence of the ice is threatened, and the viewer is taken on a journey through her seemingly futile efforts to protect what was once an entire glacier. The story is inspired by Jiabao Li's own experience from two years that she lived in Alaska, where she was the young girl protecting a piece of ice in the freezer. In Inupiaq tradition, a tribe of indigenous people from northwestern Alaska, glaciers carry memories from the past and communicate them in song. The climate crisis has become a terrifying reality that includes seeing the end of glaciers—the end of these sung histories—happen before our eyes. Once a Glacier shows a gesture of nurturing, and whether the girl's efforts are successful or not, the work suggests a poignant level of grace and humility for moving forward into the future.