SA '23: SIGGRAPH Asia 2023 XR

A Safer Place represents a ground-breaking fusion of art, technology, and research that offers viewers a deeply immersive and empathetic experience within a hospital Emergency Department and beyond.

Eight experts with lived experience guide us through a virtual reality experience of a hospital Emergency Department [ED]. They describe the scenarios of mental distress that led to their admissions, along with their experiences of waiting rooms, ED beds, and hospital procedures, and their insights into what can be done to address the shortcomings of emergency care.

A Safer Place serves as a powerful tool for fostering empathy and understanding among healthcare professionals, policymakers, and the general public. By allowing viewers to step into the shoes of those who have faced mental distress and sought care, the project aims to drive meaningful change in emergency care practices. The immersive VR experience offers a unique opportunity to dismantle barriers, spark conversations, and inspire compassionate, patient-centered approaches to mental health care in emergency settings.

The foundation of this creative endeavor is built upon extensive research conducted by Renata Kokanović and her team (RMIT Melbourne), who collaborated with 17 individuals accessing the ED for mental health care. The primary objective of the initial research project was to foster understanding among medical staff by gaining insights into ED processes from the unique perspective of those who have lived experience.

Using virtual reality technology, the project takes viewers on a compelling journey, providing an immersive and empathetic encounter with the experiences of the eight guides. Through their narratives, viewers gain a deeper understanding of the complex and nuanced realities faced by individuals impacted by mental health distress. The project aims to bridge the gap between medical staff and patients, fostering empathy, compassion, and ultimately leading to improved patient-centered care.

Moreover, it offers insights and recommendations for addressing the limitations and improving emergency care. By amplifying the voices of those with lived experience, the project aims to drive meaningful change in the healthcare system, encouraging more compassionate and effective practices.

Please be aware that whilst the piece does not depict people in distress, the audio includes frank discussion of experiences of distress, including of suicidal feelings and psychosis.

Anxiety and stress have gradually become commonplace mental health problems afflicting many people worldwide. We propose asmVR, a novel approach to enhance the autonomous sensory meridian response (ASMR) experience by combining multimodal triggers including visual, auditory, tactile, and emotional stimuli. asmVR helps users enhance ASMR tingling sensations through online and offline modes, provides realistic VR environments and remote avatar ASMRtist for users, showcasing its potential for stress relief, emotion regulation, and customization. Additionally, it reveals new possibilities for the future application of VR in the field of psychotherapy.

Body Cosmos is an immersive and interactive artwork that connects the human body with the cosmic environment through real-time bio-data. Using volumetric rendering techniques and particle system, we create a surreal virtual reality that reflects the intricate structures of human anatomy and celestial nebulae. We integrate a heart rate sensor and an EEG headband to capture and process bio-data into emotion indicators, which influence the visualization of particles in the artwork. This creates an intimate, personal connection with the cosmos, transcending immediate presence and nurturing a perpetual presence within the cosmic expanse. We also provide two engagement modes: Roller Coaster Mode and Free Explore Mode, allowing different levels of exploration. Body Cosmos is more than a visual spectacle; it sparks curiosity, expands the imagination, and enhances awareness of our embodied and cosmic identity.

"Cubic Gymnastics" is an XR system that enables users to engage in a wide range of exercises specifically designed for geometric shapes, which are visualized within the HMD space. By wearing the HMD, users can mentally immerse themselves in a geometrical body viewed in the mirror referred as the "cubic body," comprised of multiple scalable cuboids and/or curved solid shapes. The physical restriction imposed on the user’s body by a long rod, typically limiting hand motion, creates a perception of being transformed into a flat form, thereby enhancing the illusion of owning the cubic body. These geometric exercises involve unique movements that are physically impossible for the human body, including the separation of the upper and lower body, adhesion, stretching, deformation, and rotation-on-the-ground. Results from a survey conducted during a laboratory exhibition confirmed that a significant number of participants strongly experienced the sensation of performing these unconventional object-like movements specific to the transformed geometric shapes.

How is the world perceived by someone with depression? The animated immersive film uses virtual reality to address depression and the ways to cope with it. Director and protagonist Ondřej guides us through diverse landscapes associating the story of his struggle with depression since puberty. We share his feelings during the first depressive episodes at a family trip in his childhood, at university when striving for perfect results, at work in his everyday fights with the depressive ‘darkening’. Through animation, combining a stylised form of Ondřej’s environment and abstract images of his emotions, the viewers will experience and understand what it is like to live with this illness, how to tackle it and what mechanisms are used by people with depression to feel better. Most of the interactions are voice controlled. The main character Ondřej finds out that his tool to get the depression under control is his voice. He uses humming, singing and even shouting as a calming and relieving technique.

The "Dongba Script Character Construction Space" is an interactive science VR experience based on the Dongba script of the Naxi people in Yunnan Province, China, which provides an immersive virtual reality experience based on the traditional Yiyin script theory. [Selmanović et al., 2020] Specifically, this work combines the character construction characteristics of "pictogram", "simple ideographs" and "compound ideographs" to explore the meaning and visual symbolic characteristics of Naxi pictograms through three interesting forms of interaction, and provides more possibilities for the inheritance and development of Dongba in the future by using various forms of interaction.

The intangible cultural heritage(ICH) of China contains a wealth of oral literature. Their narrative characteristics tell the historical memory of a region or a nation. Miao ancient songs are one of them. "Dream Songs" is an immersive virtual narrative experience integrated with body interaction. Based on the singing content and antiphonal form of "Maple Song" in Miao ancient songs, our work explored the design methods of digital narrative expression of oral literature of ICH from three aspects: antiphonal interactive narrative, metaphorical body interaction, and gamification of knowledge.

The Dynamic Ocean Explorer XR experience allows users to explore oceanographic volumes in mixed reality; combining standard 2D ocean science visualisation approaches such as flow fields with approaches only possible in dynamic 3D such as arbitrary 6-axis clipping, as well as the embodied interaction techniques of natural gestural interaction and direct manipulation. The system is instantiated as mixed reality with pass-through video in a VR headset, and also using Web3D in a standard browser.

The widespread adoption and application of MR technology have opened up new possibilities in the field of education. Integrating physical objects and virtual contents in the physical interactive interface provides a larger practical platform for astronomy science education. However, very few attempts are currently at this form of science education. Therefore, we have summarized the design principles for the spherical TUI regarding both interactive interfaces and behaviors. Based on this, we have developed a spherical TUI-based astronomical knowledge system. It delivers users with a more diverse knowledge presentation by a fusion of virtual and reality and mapping interaction of spherical TUI in MR.

Composed of light, movement, and AR, <Flow> is an AR immersive performance in which the virtual actor and the audience create movement together. The virtual actor appears on stage through AR technology, can predict and flexibly respond to the audience's behavior based on AI. The audience interacts with the virtual actor through hand interaction designed based on ‘Contact Improvisation’. The movements they form together are visualized on stage and tell a story of memory, while implicitly capturing the flow of relationships between two different worlds: virtual and reality, past and present, existence and non-existent.

It’s all about money! Money, from bills to bitcoins, has no intrinsic value beyond what we’ve collectively agreed to grant it. However, there’s no denying that money governs our lives. FORTUNE offers a modern & pop exploration of the power of money, its psychology and the arbitrary and unequal dimensions of our economic system.

FORTUNE is a series of 8 animated documentary shorts in augmented reality, 8 filters for Snapchat & Instagram. 8 people tell how money dictates their lives, how they have to rely on it to survive, or how they think it will bring happiness or freedom. From counterfeiting, to dumpster diving, from selling everything for bitcoin to retiring at age 30: what are we willing to do and give to make a little more money?

The series is developed on Snap Lens AR Studio and Meta Spark AR to bring narrative, animated and interactive AR stories to life on social media.

Bombed houses, schools, hospitals, streets and many others. War in Ukraine from its beginning has taken away thousands of households and lives as well. Kharkiv was one of the towns which was hit in a massive way. In this immersive experience in virtual reality inspired by the famous Marina Abramovich performance “The Artist Is Present” you look into the eyes of those who lost their homes or places closely connected with them. What can this simple look awake in us? Will it bring despair, grief or hope?

Augmented reality (AR) task support systems have not yet been widely spread in actual companies despite the fact that many researchers have worked tirelessly to validate their effectiveness for various industrial tasks. One of the reasons for this is the excessive development cost of a practical AR system. Development still strongly depends on outstanding developers’ skills and trial-and-error. To solve this problem, we propose to separate an AR task support system into a reusable software part (including basic functions and its user interfaces) and task support contents that differ depending on each task. We have developed a general software platform and an original file format for describing task support contents based on analyzing various tasks. In addition, we have also proposed a web tool that enables easy design of AR task support content even for those without AR development experience. The output of the tool, an XML file written in the format, is loaded into the software platform to enable operators to use the AR task support system. We demonstrate a series of steps from the creation of task support content to the use of the AR task support system.

Haptic Hongi – Reiterated uses Augmented Reality (AR) to recreate meeting an indigenous person on their own terms, and thereby reimagine first encounters that took place during the pre-colonisation era in Aotearoa New Zealand. These encounters were often a-symmetrical affairs, freighted with troubled power dynamics and violence, so there is a need to rebalance, restore and reconnect in a playful way. In the experience, AR technology is used to place a virtual woman in front of the user, and then haptic actuators are used to enable the user to feel a hongi, the traditional Māori greeting involving pressing of noses. In this way, AR technologies enable people to communicate across time and space so that they may feel as though they have met.

Healing Horizons presents a novel, bio-adaptive Virtual Reality (VR) approach to Traumatic Brain Injury (TBI) rehabilitation. Traditional therapies often fall short, prompting the need for a more personalised, dynamic intervention. Healing Horizons meets this demand by combining real-time physiological data with VR technology, facilitating effective recovery. The system uses the novel Galea VR head-mounted display (HMD) with integrated physiological sensors to continuously monitor brain activity, skin responses, muscle movements, and heart rate variability. These real-time bio-signals predict the user’s cognitive fatigue, a common challenge in TBI patients. The VR environment then adapts dynamically to these physiological cues, ensuring the therapeutic experience is optimised for the individual’s current cognitive state. This personalised approach helps develop resilience and social skills while addressing cognitive fatigue within a realistic yet controlled setting. Our demo showcases the potential of VR HMDs with physiological sensors to be used for bio-responsive virtual rehabilitation.

Memes vs. algorithm in metaverse inspired by real rebellion stories. Kristine is Not Well is a 20-minute animated VR social media simulation that highlights the resistance of online activists against algorithmic surveillance and censorship.

"Limit Situation" is a virtual reality artwork that explores the universal existential dilemma of human existence through immersive and narrative-driven virtual spaces. The artwork is divided into six chapters of virtual spatial experiences, inspired by six books that provide background and context. These virtual experiences create a realm for emotions and contemplation. From conceptual to tangible, from real to virtual, the artwork represents the artist's understanding of existential dilemmas.

We present Secret Mapping, an interactive tabletop installation coupled with a handheld Augmented Reality (AR) visualisation. The installation showcases five selected locations which are the human-scale twins of actual, real-word exhibits all located throughout former Yugoslavia. AR visualisation illuminates these miniature 3D replicas and encompasses them with an extended version of the archived visuals created by the artists, specifically to each spot. The installation invites visitors to join the artists on their trip and reenact their journey with an collaborative tabletop interface, while offering the possibility for backwards or forward exploration.

M10H (Missing 10 Hours) is a narrative VR project directed by Fanni Fazakas. In the multi-ending experience, the user is embodied as the bystander (the buddy of the perpetrator), who can influence the narrative. By consulting with psychologists and experts, we made sure that the experience is ethically correct and has a measurable impact. In the experience, users can explore a similar night to that that of a real-life victim of GHB, the ’date rape drug from the Bystander’s perspective. Multiple interactive gameplay elements will allow the user to influence how the story evolves and gain a multi-perspective understanding of the issue. After onboarding, the bystander can explore real-life scenarios where a “pal” called Greg puts GHB into a drink that belongs to their common friend, a young woman called Mara. Since GHB makes it impossible for Mara to defend herself, or even to ask for help, the only possibility is that the user will recognize what happens and save her from Greg and his questionable intentions. During the multi-ending storyline, the user has to think, react and challenge themselves.

The MR BLS Rescue Trainer is a training system that allows users to experience realistic simulation with physical feeling in lifesaving procedures, that is BLS (basic life support), including CPR (cardiopulmonary resuscitation) and AED (Automated External Defibrillator) by applying physical mixed reality incorporated with partially tangible components

Virtual parallel experience enables a person in Virtual Reality (VR) to receive multiple and concurrent perspectives. It also enables the person to synchronously participate in activities or solve multiple tasks at once. This research focuses on utilizing parallel views for asynchronous tasks by providing the user with intuitively extended view perspectives driven by gaze. In this demonstration, we present NinjaHeads, a parallel view system that generates four additional perspectives "parallel views", based on the user’s gaze point. We introduce a technique where a user can manipulate parallel views and their behaviors through eye, head, and hand movements. This allows the user to concurrently examine an object or a point of interest from different perspectives, as if the user has more than a pair of eyes.

This project reconstructs and reproduces the ecological context of “NVSHU” culture in an immersive (Virtual Reality) VR experience. We explore interactive approaches of communication for this unique intangible heritage context. "NVSHU" amplifies the effectiveness of intangible cultural heritage (ICH) preservation through narrative interaction and virtual character popularization techniques, aiming to enhance audiences’ experiential learning, and augment their sense of presence and immersion.

Perinatal Dreaming is a ground-breaking virtual reality experience developed by fEEL felt Experience and Empathy Lab (University of New South Wales Sydney) and led by midwife, nurse, artist and trauma support worker, Marianne Wobcke.

As one of thousands of Indigenous children forcibly removed from her mother at birth, Marianne has spent her career researching and supporting perinatal and intergenerational trauma, focusing especially on work with mothers and babies.

The VR artwork presents a visually stunning immersive audio-visual experience evoking early life in the womb and entry into the world, taking us through experiences the ‘good’ and ‘toxic’ womb and first encounters with breast, skin and the world. Designed as a unique art experience, the piece can also be used in conjunction with therapeutic work.

Prison X is an innovative VR experience transporting audiences into a mythological Neo-Andean underworld. As a foundational part of the Neo-Andean Metaverse, it translates Andean perspectives on time, space, philosophy, and mythology into an immersive 3D realm. This collaborative project by indigenous artists reclaims indigenous aesthetics using pioneering technology. The prison environment was created through photogrammetry and hand-painted in Tilt Brush then imported into Unity. Characters represent Andean futurism, designed based on pop culture and folklore. Animation utilizes motion capture while spatial audio and interactivity enhance immersion. Premiering at Sundance, Prison X combines storytelling, immersion and interactivity to innovate emerging film futurism. It explores anti-colonial narratives, challenging notions of identity and reality. The diversity of perspectives expands possibilities for XR as a tool of empowerment. Prison X points to an inclusive creative future, pioneering new visual language and storytelling.

SensoryScape presents a novel approach to context-aware empathic interactions (CAEIxs) using a VR head-mounted display (HMD) integrated with various biosensors. This innovative system addresses the need for personal, emotionally resonant, and empathic digital experiences by dynamically responding to users’ real-time emotional and situational context. SensoryScape harnesses the Galea VR HMD, a device with integrated physiological sensors that track bio-signals such as brain activity, skin responses, muscle movements, and heart rate variability. These real-time biosignals, along with situational context, help estimate the user’s emotional state within its context. The VR environment and empathic companion adapt to these cues, ensuring an optimized experience for the user’s current emotional state. The demonstration of SensoryScape highlights the potential of VR HMDs with integrated physiological sensors for crafting empathic digital experiences. Our research contributes to immersive, adaptive, and user-focused empathic interactions, marking a promising shift in human-computer interaction.

In this demo, we present FusionNexus, an asymmetric collaboration experience where a virtual-world player needs to guide a real-world player to make certain actions. A machine learning algorithm was used to do the pose detection of the real-world player. We collect the virtual-world player’s heartbeat to control drum beats in background music played to the real-world player to convey emotional status of the virtual-world player. In this demonstration, two players will collaborate to accomplish a time task.

Transcale is a VR experience where the users can change the body scale smoothly while walking. By continuously modifying their perceived body scale, this interaction method provides the users to explore the virtual world from various points of view from a dwarf to a giant.

444.2 is a volumetric 6-DoF VR fashion film presented for the category of XR theatre. It is the outcome of transdisciplinary practice-based research at the nexus of fashion, expanded imaging and digital media. The researcher's praxis emerges as rhizomatic, within a network of human and non-human agents. The project interrogates the notion of digital fashion bodies from a critical speculative and posthumanist [Braidotti, 2013] lens using Deleuzian concepts and diffractive methods. 444.2 stages fashion bodies as morphing figurations unfolding to a cinematic Afro-diasporic score. The volumetric scenography of black female bodies as celestial bodies in the virtual environment of the Southern African Large Telescope becomes an enactment of African cosmology. Posthumanistic performance emerges as an experience of pluriversal and decolonial [Escobar, 2018] performativity in VR.