
Gesture Beat Game
A Rhythm Game Controlled Entirely by Your Body
“Rhythm games have always asked you to press buttons in time. This one asks you to move.”
Concept
Gesture Beat is a rhythm game where the controller is your body. Using a live webcam, the game tracks hand and arm position in real time and prompts the player to make specific gestures in sync with a beat. Accuracy is scored based on how closely the player's movement matches the target zone at the moment of the beat.
Technical Design
MediaPipe's holistic model tracks both hand landmarks and full-body pose simultaneously, giving the system visibility into wrist position, elbow angle, and hand orientation. Beat triggers are pre-timed to the audio track. At each trigger point, the system evaluates whether the player's hands are inside the target zone, rendered as coloured on-screen targets, and scores the hit as Perfect, Good, or Miss based on proximity and timing.
The Challenge of Latency
Real-time rhythm games are notoriously sensitive to latency. Even a 50ms delay between gesture and detection can make a game feel unresponsive. To keep latency minimal, the entire pipeline (frame capture, landmark extraction, zone evaluation, and screen rendering) runs in a single synchronous loop optimised to process each frame in under 33ms. Audio playback is handled separately to avoid blocking the vision pipeline.
Project Gallery
12 items




Explore More Projects
Swipe or click arrows to see other work
















