SA '21: SIGGRAPH Asia 2021 Real-Time Live!

While VTubing has grown in popularity over the last few years, CodeMiko stands out by offering high fidelity motion and facial capture and a higher degree of interactivity due to Twitch chat's ability to affect her streams. CodeMiko has created a new form of entertainment inside of Twitch by mixing gaming, live interactivity, and traditional scripted content. New interactions are created all the time, making it an incredibly exciting experience for the viewers and makes them want to keep watching. To bring CodeMiko to life, all you need is motion capture through an Xsens suit and facial capture through an iPhone live link. Her fingers are captured using Manus VR. Most of her expressivity is done through her animation blueprint and I've added blendshapes with squash and stretch to make her face feel more alive. Heightened interactivity with chat is one of the most unique aspects of the CodeMiko show. When I connect to Twitch chat, tools that I created allow the viewers to play with the character morph screen, while others boast arcade-like features. Chat can make her explode, throw balls and try to hit her, or make her fart. Chat can also visit Miko's world as a little frog, which Miko can then grab and throw around if she wants to. As an example, I have connected to chat and they will play with the character morphs live to change CodeMiko's body¹.

Another interesting technical aspect of CodeMiko is that Miko is a mixture of motion capture and pre-recorded game animations. I've blended both so that she can switch from a fully free range of real time motion to being able to walk and run like a third person game character. Chat can then come into her world and CodeMiko can shoot them as if it were a live video game! The blending with a game mode also allows me to create interactions and reactions to things happening in her environment caused by chat. For example, when chat throws a ball at her face and it hits her, she reacts in real time with an over-the-top, pre-recorded hit animation. This opens a whole new world where I can create miniature games with mocapped characters. I've created trivia levels, a dodgeball minigame, and can't wait to make a mocapped platformer.

Live-streaming on Twitch means long hours performing on a regular schedule almost every day of the week, but the audience always wants new and exciting content. I had to get very good at quickly evaluating the 3rd party content available from the Epic store and other 3rd party providers for its quality, ability to be quickly integrated into the stream, and ability to enhance Miko's world in general. This has led to all kinds of crazy new features being added like dragon eggs that need to be hatched by the audience or a car that Miko tries to drive. Miko also has an array of traditionally expensive content like entire new artified scenes and full character costume replacements, which are all enabled by building an efficient pathway in for 3rd party content.

The interactivity that I've created for chat allows for them to become a part of each individual show, but also the content loop itself. Members of chat can use Twitch's built in monetization system to affect the stream in various ways. These interactions become clippable moments that viewers can then take joy in posting on social media, and oftentimes, clips from these streams will go viral. Viral clips bring in new viewers, who will then test interactivity features and create interesting moments, and these moments will go viral again. Since her debut on Twitch, CodeMiko has become more than just a streamer but an evolution of real time entertainment as we know it. This project has fully and successfully integrated a fast moving pipeline of both 3rd party and original content in order to appeal to the need for fresh content on a daily basis.

We developed the "Fencing tracking and visualization system." It detects the tips of sabers (fencing swords) to visualize the trajectory of the sabers in real time, which doesn't require any markers but works only with the input of the images from cameras. This is the only fencing visualization technology that has been used in actual international fencing matches, such as the H.I.H. Prince Takamado Trophy JAL Presents Fencing World Cup 2019.

Fencing sabre, especially its tip, moves quite fast, and its flexibility results in a large distortion in its shape. Additionally the tip is the size of only a few pixels when captured even by a 4K camera so that it is too small to detect with image recognition techniques. We developed a multi-stage deep learning network for general object detection based on YOLO v3 [Redmon and Farhadi 2017, 2018], starting from the hardware selection of a camera for analysis. Since a single camera can only cover about 8 meters, we eventually installed 24 4K cameras on the both sides of the piste to cover the entire match area and improved the robustness of the sabre tip detection. We also developed a system to estimate the 3D position of the tips from the detection results of multiple cameras. (Rhizomatiks) Planning, Creative Direction : Daito Manabe (Rhizomatiks) Planning, Technical Direction, Hardware Engineering : Motoi Ishibashi (Rhizomatiks) Software Engineering : Kyle Mc-Donald (IYOIYO), anno lab (Kisaku Tanaka, Sadam Fujioka, Nariaki Iwatani, Fumiya Funatsu), Kye Shimizu Dataset System Engineering: Tatsuya Ishii (Rhizomatiks), ZIKU Technologies, Inc. (Yoshihisa Hashimoto, Hideyuki Kasuga, Seiji Nanase, Daisetsu Ido) Dataset System Engineering : Ignis Imageworks Corp. (Tetsuya Kobayashi, Katsunori Kiuchi, Kanako Saito, Hayato Abe, Ryosuke Akazawa, Yuya Nagura, Shigeru Ohata, Ayano Takimoto, Kanami Kawamura, Yoko Konno) Visual Programming : Satoshi Horii, Futa Kera (Rhizomatiks) Videographer : Muryo Homma (Rhizomatiks) Hardware Engineering & Videographer Support : Toshitaka Mochizuki (Rhizomatiks) Hardware Engineering : Yuta Asai, Kyohei Mouri, Saki Ishikawa (Rhizomatiks) Technical Support : Shintaro Kamijyo (Rhizomatiks) Project Management : Kahori Takemura (Rhizomatiks) Project Management, Produce : Takao Inoue (Rhizomatiks) This work was conducted with assistance from Dentsu Lab Tokyo.

Glisp is an open-source project to develop a design tool that aims to combine the flexibility of computational art with traditional human-centered graphic design. It started from the author's awareness of the problems with existing proprietary software as a visual artist who works on design and video production every day. The core idea is quite simple; by using the source code itself written in Lisp as a graphic data file, it brings the same extensibility to design as a programming language. With the Lisp's power of "boot-strapping", artists including non-programmers can program unique ways to draw graphics to fit their purpose, and become to be able to speculate unexplored styles far beyond existing cliché.

Pixar's Presto Crowd Framework (Pcf) has taken interactive crowd scalability to new heights by harnessing the power of Presto's scalable execution framework for procedurally rigged crowds. Pcf is a vectorized and highly parallelized crowd system that represents an entire crowd as an aggregate model. Prior to Pcf, Presto crowds used a model per agent, via the Mf (model framework), and only scaled to around thousand agents, even with fast GPU based linear blend skinning. Pcf CrowdPrims by contrast can scale to tens and even hundreds of thousands of agents using the same underlying data while maintaining interactive frame rates. Pcf was also designed from the ground up to import from and export to USD to make pipeline deployment seamless. Pcf Crowd Prims use procedural implementations for crowd techniques that are traditionally simulated, like Finite State Machines, making the results deterministic and pipeline friendly. Taken together, our Real-Time Live Demo will prove to the audience that they do not need to settle for the long execution times and history dependent nature of simulated crowds, and that with sufficiently a vectorized/parallelized framework, even stadium sized crowds can be generated/choreographed interactively.

In this Real-Time Live, we demonstrate RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations [Suzuki et al. 2020]. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational ways, so that the sketched elements dynamically move with the corresponding physical motion. The user can also quickly visualize and analyze real-world phenomena through responsive graph plots or interactive visualizations. This paper contributes to a set of interaction techniques that enable capturing, parameterizing, and visualizing real-world motion without pre-defined programs and configurations. Finally, we demonstrate our tool with several application scenarios, including physics education, sports training, and in-situ tangible interfaces.

In this demonstration, we exploit human perception characteristics and dynamic projection mapping techniques and realize overwriting of both textures and deformation of a real object. To keep the projection following a moving object and induce deformation illusion, we developed a 1000 fps projector-camera system and demonstrated augmentation of the real world. In the demonstration, the audience will see a plaster figure turning into a colorful and flabby object.

This is a demonstration of Sony PCL's virtual production, LED WALLIn-Camera VFX. Our virtual production is built with Sony's 8K Crystal LED and a cinema camera, VENICE. It achieves overwhelmingly low latency, and possible to shoot with focusing on the CG background in nearly real time. The short demo will be presented from our virtual production studio via live streaming.

In Japan, we have religious meditation training to stare at one place called "AJIKAN." This real-time live performance reveals the technical and cultural connection between the Brain Control Interface and AJIKAN. During the performance, the system lets the performer click a preferred location on a computer screen at which they need to stare during the performance. Then, the Brain Control Interface device from NextMind can steadily read the brain activity, and the system renders the information in real time. It connects the latest technologies and creativity in the fields, including live performance, installation, art, and XR.

We have implemented VRoid Studio, a 3D character maker that specializes in Japanese anime-like expressions. Our target is to make illustrators enable to create 3D models by themselves. Creating a 3D model is a very difficult task that involves many processes. We have defined several elements essential to express illustration-like characters, and that is textures, clothes, and hairs. We implemented several features to design them intuitively.