Labs to Features: Auto-Ground Plane in Project Felix
Welcome to Labs to Features, a new blog series offering a behind the scenes peek into how some of your favorite product features came to be. We’ll take you through the journey of projects and technologies that originate in Adobe Research labs and eventually transform into the tools and product features you rely on every day.
Serendipity in science has led to incredible innovations throughout history. Microwave ovens, super glue, and pacemakers are all examples of research making its way into a product that it was not originally intended for.
Such innovations are also common in the creative world. As creatives and designers, it’s not unusual for your final result to be drastically different from the idea you started with – and in many cases, that’s a good thing.
Our Adobe Research and product teams face similar experiences when it comes to projects and ideas. The Auto-Ground Plane functionality in Project Felix is a perfect example of this serendipity.
Project Felix needs a feature
In late 2015, the Project Felix product team began compiling its list of must haves for the new 3D compositing application. The ability to composite a 3D object with a 2D photo background in a photorealistic way topped the list.
“We knew we had a challenge,” Emiliano Gambaretto, Principal Scientist at Adobe, explains. “We needed to find a way to composite 3D assets with 2D images while ensuring that graphic designers with no 3D expertise could easily place a 3D object into a background image in a way that looks like it belongs.”
This particular workflow has existed for years as part of complex 3D design software platforms for 3D modelers and traditional 3D artists. The Project Felix team wanted to build technology that was made specifically for graphic designers.
The Project Felix team began searching for the best solution by reviewing technology and projects in development by Adobe Research. After speaking with Geoffrey Oxholm, Research Engineer at Adobe, they discovered the Upright technology, which was originally developed as a feature in Adobe Photoshop and Lightroom.
Already a critical tool used by photographers, Upright is a form of lens correction that enables you to easily straighten images, fix horizons, and reduce or eliminate the keystone effect in buildings.
While Upright seemed like it could solve the needs of Project Felix users, transforming the technology to address the unique aspects of working with 3D objects still posed a challenge to the team.
Adapting technology to meet unique needs
In applications like Photoshop CC and Lightroom CC, if Upright rotates an image and it happens to be wrong, you can simply rotate it one direction or another to achieve the correct result. However, working with 3D objects is typically much more complicated. A 3D object aligning with a camera operates on three different axes. If the image isn’t rotated correctly by the auto function, it can be extremely difficult to manually correct the alignment. With three rotation axes, it is easy to quickly lose track of the camera’s relationship with the rest of the 3D world.
So, the Project Felix team asked for an additional feature: It needed Upright to not only align the 3D camera with the image, but also generate a confidence metric indicating how confident it was about the accuracy of the result. In its current form, the Upright feature knew when it was finished aligning the camera, but it didn’t know if it had done a good job.
To gauge the success of the algorithm, the Project Felix team presented researchers with 450 sample images they felt replicated the types of images that graphic designers would want to use for compositing in Project Felix. The research team then used these as an evaluation benchmark for algorithm refinement.
Still, the worry was that sometimes the algorithm could present an image that was mathematically correct, but visually looked off and vice versa. Through extensive user experience testing, researchers trained the confidence metric to react in a way that accounted for human perception, approving results that would likely feel correct to the human eye.
Collaboration creates a wealth of positive results
The process of transferring the technology, adapting it to fit Project Felix’s needs, and producing a final product feature was a group effort that involved collaboration across multiple teams.
“Tech transfer can sometimes be seen as a collaboration between an engineer and a researcher,” says J Eisenmann, Computer Scientist at Adobe, “but in this case, I feel like 30 to 40 percent of the team was involved in the project.” The Project Felix team, Adobe engineers, a program manager, a tech transfer intern, and the Adobe Design team all chipped in to pull the project off.
The great thing about collaboration is how it helps all the teams involved. The Project Felix team developed the perfect technology to help its users, but by identifying problem points, the team was able to improve Upright’s core functionality in Photoshop and Lightroom as well. Now, photographers can better deal with tricky images too.
The Project Felix and Research teams worked so well together to transfer the technology that they have set new precedents for how the process should be done. “This tech transfer is setting the gold standard of how we want to deal with this process on all teams,” Geoff says. “It’s cool because you see this idea, of unexpected results, in the process itself as it is being refined too.”
About Project Felix: Project Felix makes it easy to composite 2D and 3D assets to build product shots, scene visualizations, and abstract art. Made for graphic designers, not 3D experts. In Beta — more exciting features coming soon.