Input Data

For input data my setup supports MIDI clock, audio signal, and gamepad controls.
MIDI clock and audio are used for automated control, but are not strictly required.
These signals drive the sequencer, post-processing materials, and particle systems.

BPM Sync

The core of the system is video playback synchronized to music.
Video playback speed is measured in frames per second (FPS), while music speed is measured in beats per minute (BPM).

The playback speed is calculated as: FPS = BPM / 4

This results in 15 frames per beat and 60 fames per bar. The BPM source can be set manually or automated via MIDI clock from Pioneer mixers and other common DJ setups. The MIDI clock sends 24 pulses per beat with an accuracy of 1-2ms. These are buffered in a fine-tuned calculation for accurate sync.

Beat synchronization can also be triggered manually or automated via audio input with a beat drop detection system, particularly effective for electronic dance music.

Pre-rendered Videos

Video loops with rhythmic movements of 15, 30, 60… frames are stored as PNG, JPG or EXR sequence. This makes frame-accurate playback possible. PNG-Files also have an alpha channel, this means every pixel of every frame has an opacity value, making overlays possible.

Loop & Stutter FX

Different loop and stutter effects can be triggered via the gamepad.
They are driven by an internal MIDI clock matching the music’s BPM.
This clock triggers events at the start of each beat and bar, controlling playback position and direction.Loop length can be halved for visual build-up effects that match common DJ techniques.Beat sync vibrations of the gamepad, give feedback that a Loop is currently playing and if everything is in sync.

Post-Processing

Post-processing materials provide a variety of effects, either BPM-synchronized or directly controlled by audio amplitude. The gamepad allows for further controll via the sticks. For greater creative flexibility, i used Custom HLSL-Scripts.

Visual Mapping

Because Unreal Engine is a 3D engine, visual mapping is relatively straightforward. I use Rhinoceros 3D (CAD) to prepare the mappings. If I have the planned stage design in advance, I model it in 3D and later align everything precisely at the event. Cutouts—such as the area where the DJ stands or overlapping elements—are adjusted on-site once the projector is set up. I then create a black-and-white 2D mask, which is applied as a texture in every post-process material and multiplied onto the final output.

Projector

For projection, I use a DLP Optoma UHD38. Its gaming mode significantly reduces input lag, which makes it well-suited for my visuals that feature fast-moving objects and must remain tightly synced with the music.

4K resolution is essential for large stages and screens. Although my pre-rendered animations are limited to 1080p for performance reasons, the post-processing effects, particle systems, and 3D objects are rendered in 4K, enhancing the overall resolution and sharpness of the final visuals.

Gamepad as a Bridge to the Audience

Using a Bluetooth gamepad for controls opens up new possibilities. You can mingle with the crowd, feel the mood on the dancefloor firsthand, and see the stage from their perspective. Many people ask what exactly I’m doing, when they see me on the dancefloor with a gamepad and whenever this happens, I try to give them the controller and encourage them to try it for themselves. Unfortunately, many feel overwhelmed by the opportunity. However, there have been occasions where I didn’t get the gamepad back for several hours. I have always enjoyed interactive elements at music events myself and are looking for ways to make it more inviting for visitors to engage with my visuals.