SA Real-Time Live! '24: SIGGRAPH Asia 2024 Real-Time Live!

This live performance piece explores a dancers interaction with her 3D captured virtual self. The arts-led work investigates notions of authenticity by combine multiple pre-captured 3D volumetric videos and live performance. Using an interactive XR rig and movment generate sound the dancer eventually erases all 3D volumetric representations of self.

BEAM brings 3D graphics and interactivity to the world of music-making. BEAM is a real-time audio effects plugin made by Lunacy Inc, which runs inside any digital audio workstation, like Ableton, Logic, and Pro Tools. It lets music producers craft complex chains of audio effects by manipulating a rich, audio-reactive scene of 3D objects.

Users feed a sound source into BEAM, which is visualized as an audio-reactive beam of light. Audio effects can be dragged onto the beam to alter the sound, and each effect is visualized with an object that procedurally changes its appearance based on the effect parameters. The audio signal can be split into multiple parallel paths to allow flexible routing and control.

We propose a "Debate System in Japanese Rap Battle Format." Our goal is to elevate discussions into entertainment, encouraging people of all generations to form their own opinions.

This system generates two distinct opinions (lyrics) when presented with a discussion topic, and based on these opinions, agents engage in a debate in the form of a rap battle. The system's technical composition incorporates several innovative elements:

(1) Lyric (Opinion) Generation: The system creates lyrics that represent opinions on the given topic. A unique feature of this process is its consideration of rhymes specific to the Japanese language. This ensures that the generated content not only conveys meaningful opinions but also adheres to the rhythmic and phonetic patterns characteristic of Japanese rap.

(2) Contextual Awareness: The system is designed to understand and maintain the context of the ongoing debate. This contextual awareness enables coherent and relevant exchanges between the agents, simulating the flow of a real debate or conversation.

(3) Responsive Utterances: Building on its contextual awareness, the system generates responses that directly address the opponent's previous statements. This feature creates a dynamic, interactive debate environment, where each agent's contribution is influenced by and responds to the other's arguments.

(4) Beat-Adaptive Audio Synthesis: To enhance the rap battle experience, the system incorporates audio synthesis that adapts to the underlying beat. This synchronization between the generated lyrics and the musical rhythm results in a more authentic and engaging rap performance.

(5) Integration into Audio-Visual Performance: All these elements---generated lyrics, contextual responses, and beatsyn-chronized audio---are combined to create a comprehensive audio-visual performance. This integration transforms the debate from a mere exchange of words into an entertaining and immersive experience. The culmination of these technical components results in a unique performance where complex debates are conducted through the medium of a Japanese rap battle, offering an innovative blend of technology, language, and music.

We introduce Digital Salon, a novel approach to 3D hair grooming and simulation by integrating advanced AI and physics-based algorithms. This tool enables users to create detailed hairstyles through natural language descriptions, seamlessly blending text-driven hair generation, interactive editing, and high-fidelity rendering within a cohesive workflow. With its innovative real-time simulation capabilities, Digital Salon supports dynamic hair interactions, accommodating 10,000 to 80,000 strands, thus making sophisticated hair design accessible to a wide range of users. This tool significantly enhances the creative process in digital media by providing an intuitive, versatile, and efficient solution for hair modeling and animation.

We demonstrate a combination of neuroscience experiment and visual interaction over the internet. Activity of cultured neurons is obtained by measurement instrument, transmitted to WebGL visualization in real-time, and broadcasted online as light and sound effects. The system reveals the beauty of the dynamics within a living neural network and enables the audience to interact with the culture by sending stimulation commands via the chat interface.

Most 3D games leverage animations to breathe life into characters. Even at the early stages of game development, prototyping with working animations helps developers quickly understand and adapt the game design. Controllable characters require animations that are constrained in multiple ways in order to blend together and avoid artifacts such as feet sliding. Authoring, modifying and iterating on such animations is a costly process that often requires high level expertise or expensive motion capture.

We present a novel AI-powered graph tool that allows users to leverage generative machine learning models, presented as nodes in the graph, and create animations from only a few high-level inputs. This greatly reduces the time and expertise required to author animations from scratch, while offering unseen levels of control with generative motion creation.

The combination of the many relevant nodes that the graph supports and the flexibility of our state-of-the-art diffusion models allows for going beyond text-to-motion and in-betweening, common in the literature. We can generate motions that can be synchronized on multiple aspects, such as duration, speed, feet contact timings, while being constrained by signals such as text-prompts and trajectories. Our diffusion models also allow for stylizing existing motions, and generating looping animations. This makes it possible, for the first time to our knowledge, to generate complete working locomotion sets from high level inputs such as text and duration, directly usable with common character controllers.

Our system being implemented in Unity, we showcase the power of this novel approach by constructing a prototype animated character with working locomotion and attack motions without preexisting animations, in a few minutes.

Viewtify® fully leverages a game engine and GPU to generate and render high-quality 3DCG from CT and MRI images in real time. By using stereoscopic displays, it presents 3D visuals with depth information. Its high-speed processing also enables the real-time display of 4D CT and 4D MRI images as animated 3DCG.

We introduce a novel real-time virtual try-on system powered by generative AI. Our demonstration highlights key features, including real-time virtual try-on, realistic wrinkle generation, and human-garment interaction. We showcase the system's ability to produce highly plausible results across diverse poses and perspectives, offering a seamless and interactive experience for users.

Soon, many robots equipped with AI capable of providing valuable services will appear in people's lives, much like the Cambrian explosion, but for robots. However, until now, robots have been developed primarily from a technical perspective, such as sensing, locomotion, and manipulation, but to become truly successful products, they will require a proper product design process, similar to that of the automotive industry.

With RobotSketch [Lee et al. 2024] (Figure 1), we show that such is possible for the robotics industry and highlight the growing importance of future-oriented robot design tools. These new tools can help robot designers explore and develop a wide range of alternative shapes, structures, and movements of robots in a short period during the early stages of design, and enhance the services and experiences that the robots can offer.

RobotSketch has been made possible by the recent development of three key technologies. First, a 3D sketching technology that enables the quick and easy creation of shapes and structures of robots using intuitive pen and multi-touch gestures [Lee et al. 2022]. Second, a VR technology that enables the use of a tablet device as a transparent window for 3D sketching in an immersive workspace [Lee et al. 2023]. Third, an AI technology that enables the fluent locomotion skills of robots through reinforcement learning in a physics simulation [Hwangbo et al. 2019].

Creating 3D digital content has been a tough challenge, especially when dealing with scenes packed with objects or characters performing complex motions. With Tripo Doodle, we can now rapidly prototype entire scenes and fully animatable characters, with nothing more than simple doodles and text prompts. The AI-generated assets are high- quality and ready to be used across a range of applications---from gaming and animation to immersive virtual experiences and beyond. This technology opens the door to new creative possibilities, making 3D content creation faster and more accessible than ever before.

This presentation features three distinct technologies that together will demonstrate a unique animation experience to the SIGGRAPH Asia audience. Our ML Posing technology is the realization of the SIGGRAPH Asia 2023 paper "Pose and Skeleton-aware Neural IK for Pose and Motion Editing" running live in a production animation setting. We'll be demonstrating on Buzz Lightyear, and as we manipulate the ML Posing handles, the character does feel "alive" in a way that feels unique compared to traditional workflows. Less flashy, but actually far more involved, is the underlying "Invertible Rig" technology that ML Posing utilizes. This technology is many years in the making and represents a fundamentally new approach to rig construction: each rig component can run bi-drectionally, thus making the animation controls and rig joints fully invertible at a fundamental level. Finally, interactive motion blur in Presto's viewport realizes a longtime animation request of visualizing motion blur as they animate.