We owe a debt of gratitude to Thomas Haggerty for working with us to design the first generation of this mount, upon which all following mounts have been modeled.

To create a calibration we take pairs of images from each camera looking at a checkerboard. Shooting is done so that we capture pairs of images using Kinect and DSLR. It is important to remember that we need to shoot in video mode on DSLR and not in photo mode. If you forget this calibration will not work. Zoom on the DSLR should be at the widest setting, and it is not bad idea to put little piece of tape on zoom ring so that you make sure it will not move. If you accidentally move the ring you'll need to redo the calibration process.

Print and mount this checkerboard (https://github.com/downloads/obviousjim/RGBDToolkit/chessboard_a4.pdf) to hard wood or plexi. It is very important to use this particular checkerboard image as the algorithm is searching for exactly that number of corners. It also needs to be completely flat and rigid. During calibration place mount the board to a sturdy stand. It can not be hand held by hand.

Typically it takes between 13 and 19 pairs of images to get good calibration. We shoot 3 depths: the close one (30-40cm from the lens) takes only one exposure (checkerboard almost fills the frame), middle (80-90cm from the lens) and far (150-200cm from the lens). At each distance other than close, pan the tripod ones take 9 exposures each: Top left, top right, top left, middle right, centered ... all the way to bottom right.

You can move the checkerboard or the camera. It doesn't matter which one you move as long as you don't mess the relationship between Kinect and the camera (this is VERY important, so we will repeat this over and over).

Here is a step by step process to create the calibration

Pair Capturing

At each checkerboard position record one very short video and capture one calibration image from the Kinect in RGBDCapture using the spacebar. The calibration image from kinect appears on the bottom of the screen. Make sure to keep the SLR clips short as we will just be extracting single frames from them. Wait a second after pressing record to hit space bar on the computer, as inevitably shake the rig a little bit when you press record on the DSLR. You may also consider using the Canon software utility to trigger recording or a physical remote.

Repeat this process for every position of the checkerboard. As you capture images, pay attention to the reprojection error above the each calibration image inside of RGBDCapture. The value should be between 0.0 and 0.3 - the lower the value the better the calibration. If you get a high number in there (most likely due to bad lighting or the board being too far away), adjust the board slightly position and reshoot the same position. It's super important that every time you capture a calibration image on the software you also shoot a short clip on the SLR, this way the number of takes will be the same and it will be much easier to pair them in the next step. It's also a good idea to match the time on a DSLR to the time on a computer as accurate as possible. That way pairs will have the same creation time.

REMEMBER TO REFOCUS when you change the position of the checkerboard.

RGB Calibration Frame extraction

The next step is to extract single PNG images from every large video file for us in the calibration.

Congratulations, you're finished with the calibration process. It's time to shoot!

• in the RGBDCapture app switch to the middle tab: Record.
• Make sure you've remove the cover from the IR projector
• To begin capturing footage press space bar on the computer and start the recording on DSLR.
• Since the recordings will never begin at the same time we need to create some sort of distinct gesture to synchronize them later. We tend to have our subject or someone from the crew clap their hands firmly and quickly.
• Make magic moments happen in front of the rig. Space, color, and time are your playground.
• When you finish stop both recordings (press space bar on computer). It's good to get a finishing synchronization clap as well to check against.
• NOTE: if you see the RECORD bar underneath the application flashing or filling red it indicates there are frames in the buffer that are not saving. This can result in dropped frames or lost footage. To avoid this write to a fast hard drive and make sure all other applications are closed, that you have plenty of hard drive space. If you are writing to an external hard drive we recommend FireWire over USB
• When not filming the application is compressing the depth frames in the background to preserve disk space. This is indicated by the gray bar below each Take button. Keep the app open until all the clips have been converted.

MEDIA BIN PREPARATION

After filming we need to add the RGB recordings to the MediaBin.

The folder structure for the media bin looks like this:

MediaBin/
    _calibration/
        _matrices/
            6x .yml files generated from the calibration step
    TakeOne/
        depth/
            frame_000_millis_00000.png
            ....
        color/
            MyClip.mov //the movie clip that corresponds to this
            MySLRCLip_Small.mov //optional downresed version for offline
    TakeTwo/
        ...

• copy each recorded clip into the color/ folder for that take.
• You can have both on online and offline version of our video so visualization is performant. Download mpeg stream clip (or use something like Quicktime Pro or Compressor to batch conversion program). Convert the clip to to a 640x360 (assuming you shot in 16:9 aspect ratio) MJPEG @50% speed and remember to include sound. This is the clip that RGBDVisualize will load for preview, which it will then be swapped out automatically for rendering.

Each Scene can have any number of compositions. A composition let's you render a scene in a different way, with a different in and out point, different camera moves, and different rendering styles.

A few tips

• To create more contrast between positive and negative space, change the 'Simplify' level.
• To clean the background away from clips, change the Z-threshold
• The rendering styles are screen-blended onto one another. So you can get a lot of mileage out of combining points,wires, and meshes together and varying opacities.
• If your interface is getting crowded you can nudge the timeline widths by pulling on the green stripes at the bottom. Shrink them up and the interface will snap back to the bottom giving your render window some more room.

To create a camera track, move the camera to where you'd like to look at the scene and scrub to the point in time you'd like to view. Press "Set camera point" on the GUI or press SHIFT+T. You'll notice a blue marker appear on the Camera timeline. You can then move the timeline and the camera to a new location and press SHIFT+T again. By selecting "Lock to Track" or pressing SHIFT+L the camera will move along this path and is no longer free floating.

You can click and drag the camera points to change their location in time. You can delete them as well with the Delete or backspace key. By clicking on the beginning or end triangle and hitting the up and down arrows you can have the cameras ease in or out (represented by the curved arc), or cut (represented by the small)

The Depth of Field effect is most convincing for creating a virtual camera language. When Draw DOF is selected in the GUI. By setting the Depth Range you can select which distance is in focus. Tweak the range and blur to achieve the desired effect. Depth of field keyframes matched to camera moves can create really cool convincing rack focus effects. The white rectangle on the top left assists you in focusing. The geometry in focus shows up in black, while everything out of focus is white.

Save the composition when you are ready to export or want to experiment with something else. To switch compositions, click the name of the composition at the top and select a new scene.

EXPORTING

Exporting works by rendering the a given composition's camera track into a series of PNG frames that can be compiled into a movie using quicktime or After Effects.

To export, click on the composition name to view all the scenes. Selecting a scene will show all the compositions in that scene, each of which has a small 'R' button next to it. This stands for Render, and by toggling it on you will add this composition to the render queue.

Once you've selected all the compositions you wish to render, click the "Start Render Batch >>" button and sit back and relax as all the compositions you've selected render.

Render frames are 1920x1080 and saved into _RenderBin/ folder underneath the main MediaBin/ (right next to _calibration). Each folder is stamped with the scene, composition, and date so nothing will ever be overwritten.

NOTE: each rendered comp needs a camera track. Weird stuff may happen if you render without one.