Flipbook

ACMI asked for a new permanent interactive at the museum, one of the first things visitors see as they arrive. It had to be fun and engaging, so we created a flipbook machine: visitors stand in front of a large touchscreen and camera, capture themselves in motion with a striking visual effect, review the result, and print it as a physical flipbook they can flip through to play the animation back.

We started by prototyping 20 to 30 real-time effects, testing each against different lighting conditions and skin tones. We chose the Azure Kinect, with its RGB, depth and IR cameras, expecting to need depth to isolate visitors cleanly. Testing showed the RGB camera alone gave the best results, but keeping the Kinect means the work can use depth later if conditions on site ever demand it. To use it from Electron, we extended the Azure Kinect Node C++ extension.

The chosen effect splits the image into its red, green and blue channels, offsets each by a few frames and recombines them, leaving trails of colour behind anyone who moves. It looked right but ran far below the target frame rate, even after moving it into C++. Moving the work onto the GPU as a GLSL shader, where thousands of operations run in parallel, made it fast enough to generate and save every frame in real time.

Playback had its own problem: recording video with the effect applied dropped frames. So instead of saving a video, we store the raw frames in memory and regenerate each one with the effect as it plays back.