CALIBRATING LAB
Loading interface
Gesture Beat Game
May 2025

Gesture Beat Game

A Rhythm Game Controlled Entirely by Your Body

OpenCVMediaPipePythonGesture RecognitionReal-Time

“Rhythm games have always asked you to press buttons in time. This one asks you to move.”

01

Concept

Gesture Beat is a rhythm game where the controller is your body. Using a live webcam, the game tracks hand and arm position in real time and prompts the player to make specific gestures in sync with a beat. Accuracy is scored based on how closely the player's movement matches the target zone at the moment of the beat.

02

Technical Design

MediaPipe's holistic model tracks both hand landmarks and full-body pose simultaneously, giving the system visibility into wrist position, elbow angle, and hand orientation. Beat triggers are pre-timed to the audio track. At each trigger point, the system evaluates whether the player's hands are inside the target zone, rendered as coloured on-screen targets, and scores the hit as Perfect, Good, or Miss based on proximity and timing.

03

The Challenge of Latency

Real-time rhythm games are notoriously sensitive to latency. Even a 50ms delay between gesture and detection can make a game feel unresponsive. To keep latency minimal, the entire pipeline (frame capture, landmark extraction, zone evaluation, and screen rendering) runs in a single synchronous loop optimised to process each frame in under 33ms. Audio playback is handled separately to avoid blocking the vision pipeline.

Project Gallery

12 items
Gesture Beat Game media 3
Gesture Beat Game media 5
Gesture Beat Game media 2
Gesture Beat Game media 4
Gesture Beat Game media 6

Explore More Projects

Swipe or click arrows to see other work

Built and designed by Tharsshen Govikrishnan · © 2026