SIGGRAPH '19- ACM SIGGRAPH 2019 Real-Time Live!

New high performance physics and destruction system running in Epic Game's Unreal Engine. Features include new fracture and cutting tools in the engine, dynamic strain evaluation, interactive caching, fields and Niagara support.

We propose GauGAN, a GAN-based image synthesis model that can generate photo-realistic images given an input semantic layout. It is built on spatially-adaptive normalization, a simple but effective normalization layer. Previous methods directly feed the semantic layout as input to the deep network, which is then processed through stacks of convolution, normalization, and non-linearity layers. We show that this is sub-optimal as the normalization layers tend to "wash away" semantic information. To address the issue, we propose using the input layout for modulating the activations in normalization layers through a spatially-adaptive, learned transformation. Our proposed method outperforms the previous methods by a large margin. Furthermore, the new method enables natural extension to control the style of the synthesized images. Given a style guide image, our style encoder network captures the style into a latent code, which our image generator network combines with the semantic layout via spatially-adaptive normalization to generate a photo-realistic image that respects both the style of the guide image and content of the semantic layout. Our method will enable people without drawing skills to effectively express their imagination. GauGAN in the inference time is a simple convolutional neural network. It runs real-time on most modern GPU cards. GauGAN is one of the recent research efforts in advancing GANs for real-time image rendering. We believe this is of interest to the SIGGRAPH and real-time communities.

Presented are new methods to raytrace and simulate raytracing of different types of rays through a range of different participating media in real-time, including: *SDF ray-tracing to recreated refractive physics-based viscous fluids that interface with a surrounding soft-body environment and other physics objects in the scene. SDFs are blended seamlessly with each other and environment objects. Special considerations are made to maximize cache and memory-coherence on tile-based mobile GPU architectures. *X-Rays - Which follow logarithmic attenuation functions and fresnel-like behaviors as they are absorbed and scattered through different materials on their way from the emitter to the detector plate. X-ray tracing behaves more like a transparent shadowing technique than anything else. * Ultrasonic Sound waves - Used in real-time ultrasound imaging, these rays break all the rules - you can't even rely on their propagation speed to stay constant. Dozens of different artifact types (shadows, ringing, etc.) must be simulated through the tracing behavior. For example - highly reflective objects outside of the ultrasound beam 'slice' may bounce back into the frame, creating the appearance of 'ghost' objects that aren't actually there. Various mobile-friendly thread bundling approaches are taken to cast and bounce rays in the scene. Worth noting The SDF ray-marching technique in Pulm Ex was shown last year but has been drastically improved with refractions, performance optimizations, and new SDF shapes, blends, and techniques.

Here is some text in the style of last year's RTL lineup descriptions: We demonstrate a collaboration platform for viewing massive 3D assets in real-time using any web browser on any device. Technologies such as high framerate server-side rendering, low-latency video encoding, and artist friendly markup tools are combined to accelerate production pipelines. Additional text below: Nira is a powerful new collaborative design and art review platform for 3D production pipelines. Nira removes the burden of having to send proprietary files back and forth for reviews by providing a unified platform for viewing, reviewing, tracking, and version comparing of all assets. Upon upload of a production asset, the Nira asset ingestion pipeline converts the asset to a format optimized for very fast loading and display performance within its custom renderer. When the asset is subsequently viewed, Nira renders and then encodes the frame buffer to an efficient and high quality video stream, which is then decoded within a web browser on the viewer's device at 60 fps in real-time. The client device need only be capable of decoding a video stream, so this allows large production assets to be interactively viewed on lower powered mobile devices like never before. In addition to viewing capabilities, Nira also includes a host of collaborative drawing and markup tools allowing for effective visual communication between team members and stakeholders.

One of the aims of the project was to prove that the kinds of results we achieved with Rebirth are accessible, not only for studios with huge budgets and large teams but also for freelancers and independents. If an individual wanted to download and use the assets that we did in Rebirth it would cost the equivalent of around $75. While Rebirth may look complex, in reality, we used a small set of assets throughout the entire cinematic. As each Megascans asset is quite visually complex, this allowed us to repurpose the same assets across many different shots whilst creating a varied look.

"The Heretic", the latest real-time short film by Unity's Demo Team features two entirely vfx-based characters, Boston and Morgan. VFX-based characters are notoriously hard to conceptualize, especially in motion. Building and using real-time tools allowed us to explore the space of possibilities and quickly arrive at designs we were satisfied with. The character of Boston is made up of steel wires navigating the environment, conforming to the shape of a bird-like creature. Animating thousands of wires in a traditional way was unthinkable. The challenge then lied in building tools allowing to express the intention of the artists, allow for emergent behavior, but also make it directable at the same time. Boston is implemented as a set of tools, scripts, and shaders within the Unity engine. The character Morgan doesn't have a clearly- defined physical manifestation, morphs between its female and male forms, and constantly varies its size. Morgan is implemented using the Unity Visual Effect Graph, extended with additional tools and features. The presentation will be of value to all creative communities that base their process on real-time technology - both within game development and real-time filmmaking. From a film production perspective, such characters are typically entirely within the domain of expertise of post-production / VFX studios. With the introduction of realtime-based VFX characters, these characters can be included much earlier in the production process and are more open for experimentation and better connected to the rest of the production elements. From a game development perspective, real- time VFX-based characters allow for bolder and more unconventional creative ideas to be conceived and executed and thereby contribute to richer and more elaborate virtual worlds. They allow the creatives to express more complex, abstract or surreal ideas and develop interesting and original aesthetics.

We have built a real-time (60 fps) photo-realistic facial motion capture system which uses a single camera, proprietary deep learning software, and Unreal Engine 4 to create photo-real digital humans and creatures. Our system uses thousands of frames of realistic captured 3D facial performance of an actor (generated from automated offline systems) instead of a traditional FACS-based facial rig to produce an accurate model of how an actor's face moves. This 3D data is used to create a real-time machine learning model which uses a single image to accurately describe the exact facial pose in under 17 milliseconds. The motion of the face is highly realistic and includes region based blood flow, wrinkle activation, and pore structure changes, driven by geometry deformations in real-time. The facial performance of the actor can be transferred to a character with extremely high fidelity, and switching the machine learning models is instantaneous. We consider this a significant advancement over other real-time avatar projects in development. Building on top of our real-time facial animation technology, we seek to make interaction with our avatars more immersive and emotive. We built an AR system for the actor who is driving the human / character to see and interact with people in VR or others viewing in AR. With this technique, the character you are interacting with in VR can make correct eye contact, walk around you, and interact as if you were together all while still achieving the highest quality capture. This process allows for a much more tangible VR / AR experience than any other system. Another goal of ours is to achieve photo-real avatar telepresence with minimal latency. We have been able to successfully live-drive our digital humans from our office in Los Angeles to our office in Vancouver.

We will discuss how we achieved the goals for the production of Unity's "Reality vs illusion" demo, as well as give an overview of using real-time ray tracing technology in a games production pipeline. Our system is built on the insights learned by extending the engine architecture to support realtime ray tracing APIs and incorporating the power of realtime ray tracing to increase realism for resulting renders at interactive 30 fps rendering on high-end consumer hardware. Rendered in real-time, this demo spotlights one real car and another powered by Unity's rendering technology. A real-world car (2019 BMW 8 Series Coupe) was filmed, and then the scene was recreated using the exact camera / lighting conditions in CG. Then we transition the shots from a real-world car to the ray traced car. This shows off some of the effect that we will cover - global reflections, multi-layer transparency with refraction, area lights, shadows, ambient occlusion and more. We will share state-of-the art techniques developed for achieving high-visual quality in real-time rendering with our hybrid ray tracing / rasterization render pipeline, built on top of Unity's high definition rendering pipeline. The presentation will demonstrate the technology developed to extend the original rasterization-based pipeline to provide higher-fidelity rendering through the efficient usage of real-time ray tracing, for example, by rendering primary ray visibility for higher-fidelity materials including multi-layer smooth transparency, describe advanced approaches for shadowed textured realtime area lights, support of dynamic indirect diffuse and specular lighting as well as other global effects, such as ambient occlusion, reflections, and others, taking advantage of ray tracing algorithms and touch on runtime performance, including runtime BVH update.

We control robots through a simulated environment in game engine using VR and interact with it intuitively. A major breakthrough of this system is that, even if real-time robot control is not possible, the user can interact with the environment in real-time to complete tasks. Our system consists of a robot, vision sensor (RGB-D camera), game engine, and VR headset with controllers. The robot-side visual is provided as a scanned 3D geometry snapshot. We leverage point cloud as a visualization. Given the information to the user, two steps are required to control the robot. First, object annotation is needed. Given virtual 3d objects, the user is asked to place them roughly where they are in VR, therefore making the process intuitive. Next, computer vision based optimization refines the position to an accuracy level required for robot grasping. Optimization runs using non-blocking threads to maintain real-time experience. Second, the user needs to interact with objects. A robot simulation and UI will assist the process. A virtual robot gripper will provide a stable grasp estimation when it is brought close to a target. Once the object is picked up, placing it is also assisted. As in our example with block construction, each block's alignment with other blocks is assisted using its geometric characteristics, facilitating accurate placement. During the process, robot actions are simulated then visualized. The simulation and assistance is processed in real-time. Once interaction is given, simulated actions are sent and executed. Interaction and annotation processes can be queued without waiting for a robot to complete each step. Additionally, the user can easily abort planned actions then redo them. Our system demonstrates how powerful it is to combine game engine technologies, VR, and robots with computer vision/graphics algorithms to achieve semantic control over time and space.

While hair is an essential component of virtual humans, it is also one of the most challenging and time-consuming digital assets to create. Existing automatic techniques lack the generality and flexibility for users to create the exact intended hairstyles. Meanwhile, manual authoring interfaces often require considerable skills and experiences from character modelers, and are difficult to navigate for intricate 3D hair structures. We propose an interactive hair modeling system that can help create complex hairstyles that would otherwise take weeks or months with existing tools. Modelers, including novice users, can focus on the overall intended hairstyles and local hair deformations, as our system intelligently suggests the desired hair parts. Our method combines the flexibility of manual authoring and the convenience of data-driven automation. Since hair contains intricate 3D structures such as buns, knots, and strands, they are inherently challenging to create from scratch using traditional 2D interfaces. Our system provides a new 3D hair authoring interface for immersive interaction in virtual reality (VR). We use a strip-based representation, which is commonly adopted in real-time games due to rendering efficiency and modeling flexibility. The output strips can be converted to other hair formats such as strands. Users can draw high-level guide strips, from which our system predicts the most plausible hairstyles in the dataset via a trained deep neural network. Each hairstyle in our dataset is composed of multiple variations, serving as blendshapes to fit the user drawings via global blending and local deformation. The fitted hair models are visualized as interactive suggestions, that the user can select, modify, or ignore. We conducted a user study to confirm that our system can significantly reduce manual labor while improve the output quality for modeling a variety of hairstyles that are challenging to create using existing techniques.