"Godflesh" effect

During the early stages of American Gods, director David Slade asked us whether we knew of a way to avoid mocap and key-shape animation. It just so happened that we had noticed, over the course of previous tests, that we could use the optical flow from stereo pictures to retrieve some 3D volumes. We had to create the "Godflesh" effect, a psychedelic vision of various gods that can be seen as a chaotic and complex apparition from a high dimension intersecting our dimension. Made with procedural modeling, this effect needed an animated 3D capture of the actors. To avoid a painful 3d rotoscopy and obtain a more lively performance.
We tried to find a performance capture appropriate for this TV series context, i.e. easy to set up on the shooting, capturing both geometry and color in the shot light, and involving as few as possible rotoscopy in post production. In homage to David Slade, director of the American Gods series, we called this capture tool the Sladar. The solution we brought was to use 2 pairs of stereo cameras placed on the left and right side of the main camera, framing the character we want to capture. Each stereo cameras allows us to compute an optical flow sequence that we can convert in a depth map sequence. Each depth map is then used to generate an extruded volume, from the left and the right sides. These volumes are finally intersected to obtain a 3d mesh that is more or less faithful about 70 degrees around the main camera. This technique requires a high precision in several points and had some limitations: The optical flow building the depth maps is sensible to grain, implying to shoot in 4K. This optical flow needs as well detailed patterns, without reflection. That's why we had to shoot with polarized lighting and filters, reducing the shutter to minimize the motion blur. The 5 cameras rig (2 stereo pairs and main camera) needs to be perfectly synchronized. the virtual cameras calibrated from the Sladar rig have to be very precisely retrieved. That's why we built a special cube cover with checkers and thorny threaded rods to be shot as a camera calibration item. We had to tests different cameras, with various angles between the stereo pairs to find the best rig. At the end, the principal manual work to build the sladar model was to calibrate the 3d virtual cameras, the all following process being automatic.
The next step for this Godflesh effect was to "dress" this reconstructed 3d model. Many design tests were made,
depending on the God life span. For the Technical Boy, young God of technology, we created a procedural plastic cubes animation. Mr World had a similar dressing, colored tiny cubes, with a complex animation showing the great power of transformation of this god, taking on the appearance of the hero Shadow Moon, or turning into a photo-real Fleisher animated cartoon. For Easter, we use the Sladar to generate a torrent of rose petals and for Odin we mixed a 3d scan of the actor and the Sladar model to build a majestic and stormy vision.