Member Profile: Derek Gerstmann

1. What do you do, and how long have you been doing it?

I work for Adobe Research in the Programming Languages and Performance Lab, led by Shoaib Kamil. Specifically, I’m a High-Performance Computing Research Engineer that works on developing new graphics and imaging technologies, as well as integrating experimental research projects into shipping products. I also spend quite a bit of time advising other engineering teams on performance related design and/or helping them optimize their software. In between these responsibilities, I enjoy working with the open source community, and I’ve been helping our team maintain the Halide compiler. Our lab is full of brilliant research scientists and engineers, and I’m continually impressed by the depth and breadth of the work that they cover! I’m also fortunate enough to be able to work with some of the researchers I’ve spent years collaborating with, including Andrew Adams, Maaz Ahmad and Alex Reinking. It’s an amazing environment for solving technical challenges and building new technology, and a perfect fit for me and all my interests!

2. What was your first job?

My first real job was an assistant to the system administrator, Colin Stefani, at a small visual effects company in Seattle called Pinnacle Studios. It was started by Warren Franklin and some ILM alumni including John Nicholson, Bruno George, and Michael Kennedy. I was still an undergraduate at the University of Washington so I worked part-time to cover my rent and part of my tuition as I worked on my computing and software systems degree. It was such an exciting role for me! I learned so much so quickly, and ended up taking on a lot of the scripting and development responsibilities, including developing custom tools and workflows for the compositors, as well as looking after all the system upgrades and administrative tasks for all the SGI workstations, servers and desktop PCs. The studio worked on a variety of feature films and small commercials, and I was so lucky to be in the midst of the transitionary phase of the industry going from purely analog film, to digital compositing with analog film-out, to full digital workflows. It was so inspiring to see how things were done in a real production environment and this experience led me to pursue a career in the visual effects industry.

3. Where did you complete your formal education?

I did my undergraduate degree at the University of Washington, and majored in Computing & Software Systems, however, most of my computer graphics and software engineering skills I acquired at my part-time job working at Pinnacle Studios. After an internship at Industrial Light & Magic (ILM) and an entry level position at ESC Entertainment (the visual effects studio that did the Matrix films), I decided to get a masters degree so that I could take on more advanced roles.

So, at the recommendation of some of my colleagues, I went to the National Centre for Computer Animation (NCCA) at Bournemouth University in the UK and joined their MSc program. This was one of the best decisions I could have made … the skills and production oriented courses spearheaded my career, and I made several lifelong friends during my time there. This experience landed me in my dream job at Weta Digital, working at Peter Jackson’s visual effects studio in New Zealand on Avatar, The Hobbit films, and all of the Marvel Cinematic Universe films.

Many years later, after nearly getting burned out working in Silicon Valley, I decided to pursue a PhD, more as a challenge to myself than anything else, and to take time to really delve into topics that interested me. I applied to several programs in the USA, but, perhaps naively on my part, I dreaded doing several more years of coursework, so I ended up applying to several schools in Australia. After taking a trip to tour all the major universities down under, I decided to accept an offer from The University of Western Australia in Perth since there was a huge demand for high-performance computing with the recently announced Square Kilometer Array (SKA) telescope being built. So, after talking to many different departments, I ended up joining the International Centre for Radio Astronomy Research (ICRAR) as a PhD candidate developing tools and techniques for large scale scientific visualization with Andreas Wicenec and Paul Bourke as my advisors.

Sadly, I was pretty much on my own and struggled to find a niche area to tackle. So after four years, my scholarship ran out, and I didn’t feel like my publications were strong enough contributions to the field, especially given the caliber of work coming out of other schools with dedicated visualization labs. So I made the tough decision to leave the program and head back to the visual effects industry. But I have no regrets! I made so many friends, and I learned so much about astronomy, high-performance computing and computational simulation. Of course, I’m disappointed that I wasn’t able to finish, but, ultimately, it wasn’t a good fit for me. I’ve come to realize I’m a much stronger engineer than a pure researcher … I really enjoy building and improving things and making stuff work!

4. How did you first get involved with ACM SIGGRAPH?

My first introduction to ACM SIGGRAPH was the conference in Anaheim, CA back in 1993. We were in town on a family vacation visiting all the theme parks. My brother and I convinced my Dad to take us to the convention center to check out this computer graphics conference, and we were completely blown away! There were nearly 30,000 attendees, and almost 300 vendors showcasing future technology and innovative products, and we were absolutely hooked. Since then, I’ve tried to attend every year whenever my schedule would allow. Much later, I went on to organize the OpenCL course with my colleagues Justin Hensley, and Jason Yang which we ran for several years. I have also enjoyed participating in the SIGGRAPH paper committee, and more recently, I was fortunate enough to collaborate with my close friends Tomasz Bednarz and June Kim on several SIGGRAPH Asia events and helped bring the conference to Brisbane and Sydney while we were all still in Australia.

5. What is your favorite memory of a SIGGRAPH conference?

There are so many! But one that stands out in my mind was the year Jos Stam presented his “Stable Fluids” paper back in 1999. At the very end, after giving an excellent presentation, he casually mentioned how efficient it was to implement, and picked up his Palm Pilot and gave a live demo using his stylus to slosh fluid around on the tiny screen!

6. Describe a project that you would like to share with the ACM SIGGRAPH community.

Halide, the project I mentioned earlier that our team helps maintain, may be of interest to the wider SIGGRAPH community. It’s a domain specific language for fast and portable imaging operations being maintained by an incredible open source community under an MIT-based license. Not only is it a key piece of technology for Adobe Photoshop and other imaging products, but Google uses it for re-encoding videos for YouTube, and the camera of every Pixel phone uses it for processing sensor data, and the list just goes on!

It was originally published in SIGGRAPH back in 2012 by Jonathan Ragan-Kelley, Andrew Adams, Sylvain Paris, Marc Levoy, Saman Amarasinghe, and Frédo Durand in their seminal paper (recently voted one of the most influential graphics papers of all time!). However, there have been a bunch of follow up papers by other amazing researchers that have covered new ways to automatically schedule imaging pipelines (including a heuristics based approach, a genetic algorithm based approach, and a machine learning based model), as well as differentiable programming, applications for faster deep-learning algorithms and improved instruction selection for specific architectures.

What makes Halide so great is that it separates the way an algorithm is implemented, from the way it is scheduled or mapped onto the hardware. This allows programs to be portable, but you can also write schedules that apply device specific optimizations without having to rewrite the original imaging pipeline. It is a memory safe, user schedulable language with advanced vectorization and parallelization support, and it interleaves native host and device computation with appropriate API calls so that things like CPU to/from GPU transfers and synchronization are done automatically. From a technology stand-point, it has a very simple runtime that is extremely portable, so it runs on nearly every major processor architecture on every major operating system and it is relatively easy to add new hardware support. It can target Web Assembly, WebGPU, Metal, DirectX, Vulkan, CUDA, OpenCL, and has extremely good CPU code generation. And the community is full of brilliant engineers and researchers that are constantly improving the compiler and adding new features!

7. If you could have dinner with one living or non-living person, who would it be and why?

I think it would probably have to be Ansel Adams, a true pioneer of image making, a passionate artist, a dedicated environmentalist and gentleman of the outdoors! I think it would be thought provoking and insightful to hear his thoughts on art being affected by technology, and what opinion he would have on how it may impact his body of work. In particular, digital photography, digital cinematography, and generative AI have been so disruptive for traditional photographers, yet, they don’t achieve the same feeling or tangible aspects that made photography so successful.

A true artist will create regardless of their audience, and yet in this modern era with social media driving most of our online interactions, creators now focus entirely on their audience, rather than the art or craft itself. It would be absolutely incredible to sit down for a long chat with one of the most inspiring and articulate artists of our generation, and, especially, on a personal level, considering how much impact he has had with my own pursuits of being a landscape photographer.

8. What is something most people don’t know about you?

Given how many years I’ve spent working with technology, I actually prefer to spend most of my free time away from the computer, either outside being active, exploring the world or building stuff in my garage! One of my DIY projects I just finished up is a large format panoramic film scanner that I built so that I can shoot small sections of each frame with a 2x-3x macro lens on my mirrorless digital camera and automatically stitch together my 4×5, 6×17, and 6×12 negatives to preserve all the fine detail.

9. From which single individual have you learned the most in your life? What did they teach you?

I am forever indebted to my Father for all the wisdom that he’s shared with me! He got me started with computers at a very early age when I was barely in grade school, and I started composing music on our Commodore 64 before I even knew how to play piano. He even assigned me the task of working through the “C Primer” handbook before I was a teenager, which of course, definitely kept me out of trouble when I was a kid. I was just so excited by creating, innovating and solving problems! He has always supported me in all my endeavors and taught me everything I know about the fundamentals of science and technology. Now that he’s retired, we try to spend as much time together as we can, either shooting and developing film in his garage, or working on one of our endless DIY and/or car projects.

10. Is there someone in particular who has influenced your decision to work with ACM SIGGRAPH?

Without a doubt, my close friends, Tomasz Bednarz & June Kim. They have been such great collaborators over the years, and their dedication to ACM SIGGRAPH has been hugely inspiring.

11. What can you point to in your career as your proudest moment?

Perhaps the proudest moments in my career were during my tenure at Weta Digital. In particular, working on Manuka (their in-house path tracing global illumination renderer) under the leadership of Luca Fascione, along with Mark Leone, Tomas Davidovic, Marc Droske, Jorge Schwarzhaupt, Andrea Weidlich, Eugene d’Eon and Patrick Kelly (among many others!). It’s a rare opportunity for a computer graphics practitioner to get the opportunity to bring up a brand-new physically-based renderer in a production setting!

And then, once we had the renderer working, I was fortunate enough to be able to spend several years working alongside my close friends, Antoine Bouthors and Chris Armsden, to develop a custom level-of-detail system we called Cake (a play on “have your cake and eat it too”, and “bake”, a common pipeline technique of caching renderable data).

We were given the incredible challenge from Joe Letteri, the Senior VFX Supervisor, to develop a system that could preserve the incredible detail of all the assets used in complex environment layouts, with a focus of reducing render times and improving overall pipeline efficiency.

The system we developed, was, at its core, a level-of-detail system based on a pre-integrated visibility & material response cache that streamed data (on demand) directly into the bounding volume hierarchy of Manuka, allowing us to dynamically generate surface and material data with just the right amount of detail as directed by the camera’s parameters and other user controllable settings, resulting in significantly reduced memory usage and much faster render times all while maintaining a minimal loss in quality.

We had just gotten a few successful test shots done during the development of “The Hobbit: Desolation of Smaug”, when we got the greenlight from our head of department, Jed Wojtowicz, to use it in production for the first time, it was an exciting moment for us since our system finally made it possible to render all of LakeTown in a single pass during the fly-over sequence while Smaug attacked. At the time, there simply wasn’t any other way of doing this shot due to the amount of unique geometry (every building in the town had been modelled down to individual nails in every shingle for every roof!) and none of the machines on the renderwall had enough RAM to physically load the full environments into memory. Similarly, the other huge success story was the City Of London in “Mortal Engines” which actually won a BAFTA for Best Model (which could only be rendered with Cake). Other notable films it was used on included the remaining Hobbit films, Jungle Book, Alita Battle Angel, and Avengers: Infinity Wars.

The thrill of getting shots approved for a major Hollywood motion picture that are completely generated by the software you helped write is unlike anything I’ve ever felt, and I will always be grateful to Luca Fascione and Antoine Bouthors for letting me contribute to such ground breaking projects.