×

Tutorial 1: Most of the videos contain a certain amount of distortion caused by camera lenses. Such a distortion can affect straight lines in the scene so that they are no longer straight in the camera view. If your videos contain lens distortion, we recommend to pre-process them using our command-line tools. They can estimate distortion coefficients and undistort the videos.

The first tool calib.exe reads a set of frames grabbed from a camera that was observing a calibration pattern. The pattern can be printed on a paper or displayed on a monitor. It should be captured several times from different viewpoints so that it covers as many viewing angles as possible. The tool can then automatically estimate lens distortion coefficients out of these images. This process has to be done once for each camera, and it does not need to be repeated if the camera's optical parameters remain fixed.

The second tool vundist.exe takes the input video, the file containing distortion parameters, and use them to create a new undistorted video. Moreover, it can also shift the output video by provided starting frame index, crop vertical borders of the video to reduce the final resolution, or split the video to multiple sequences.

×

Tutorial 2: A calibration sequence needs to be recorded to estimate the mutual position and orientation of individual cameras. Such recording can be done before or after the actual performance capture session. It is essential that during the capture, the relative position and orientation of cameras remain unchanged.

When recording the calibration sequence, a point light source should be placed into the active volume visible from all installed cameras. The light source should be slowly moved in the entire volume to cover as much visible space as possible. Make sure in each recorded frame the light source is visible from as many cameras as possible. This helps to increase the precision of the estimated parameters.

The used point light source should be as small as possible, and its luminance should be higher than all other light sources in the scene so that it can be easily detected and localized in the videos. We recommend to use small xenon light bulbs and increase camera shutter to make them less sensitive to incoming light.

×

Tutorial 3: The user has to provide a mesh of the character being captured. Such a mesh can be produced using a 3D laser scanner or photogrammetry. The person should stay in a T-pose or any other similar pose where hands and legs are sufficiently apart to simplify the process of mesh skinning.

There are two generic models containing skeleton and skinning weights. Skeleton and weights from the existing model can be retargeted to the provided mesh when the model is aligned as close as possible to the target mesh. The first step is to select a gender of the character. Then the application coarsely estimates scale and translation of the model. Such an initial estimate is usually not precise enough. Thus the user needs to fix the model deformation so that the position and orientation of individual body parts roughly correspond to its equivalents on the provided mesh. When the refinement is done, the resulting pose is used as an input to the solver. It aims to further minimize the overlap between the model and the target mesh. When the solver finishes the pose refinement, skinning weights can be remapped. Based on an appearance of resulting skinned mesh, the user can store the model to the specified file or fix visual artifacts by running the solver or manually modifying the model pose.

×

Tutorial 4: The motion tracking tool expects as an input a path to a directory containing files of camera parameters, the actual video files, background image files, and files with a skinned model. In addition, the user needs to specify name format for each file type used, the first and the last frame of the input sequence, and a file name for storing internal animation. Once the tracking tool is running, it is essential to check the accuracy of the estimated camera positions visually. If any of the projected camera positions is incorrect, it is necessary to rerun the camera calibration process. There can be the case when the cameras are in a correct posture; however, their coordinate system is not aligned approprietly, e.g., red x-axis or blue z-axis is not horizontal or green y-axis is not upright. In this case, push the button "Align Cameras" until the orientation is fixed.

The next step is model alignment. Place anchors on the model with the left mouse button and release the button on the corresponding position in the video frame. It is necessary to put at least three anchors from multiple views and on several body parts so the application can estimate the character's 3D position, orientation, and current pose. The right mouse button can remove incorrectly placed anchors.

When the model is aligned, anchors need to be removed ("Clear Anchors" button) and a ground-level defined by holding Control + left mouse button and moving the mouse. As soon as the setup is finished, the button "Save Cameras" needs to be pressed to update the original files with camera parameters.

The last setup step is to train model colors ("Train Colors" button). When colors are estimated, the tracking can start ("Track" button). Sometimes the automatic alignment fails, e.g., due to fast character movement, self-occlusion, abrupt illumination changes, etc. In those cases, the tracking needs to be paused and any possible misalignment fixed using the anchors. The position of the entire body can be changed by a selection and holding the F key + left mouse button. You can select multiple areas by pressing Shift. When the alignment is fixed, anchors need to be rereleased.

The captured animation can be saved anytime during the tracking phase ("Save Animation" button). When the tracking phase is finished, the final animation can be smoothed out ("Filter Animation" button) and exported to the FBX file ("Export Animation" button).