Project detail
VR Gesture Detection Training System
Real-time recognition of 50+ VR gestures from headset and controller motion.
A custom ML model that recognizes 50+ unique VR gestures in real time from headset and controller motion. Trained in TensorFlow, exported to ONNX, run inside Unity. Built in under two months.
Media and demos
See the work in motion
Includes 2 demo videos.
More demos
YouTube: Sample applications
Case study
How the project was shaped
Problem
VR controllers can replicate buttons, but the headset and controllers as a whole carry rich physical motion data. Gesture-based interaction feels more natural than menus and buttons. The challenge is converting continuous 6DOF motion across multiple inputs into a fixed-size representation a model can classify.
Architecture
Encode physical gestures into image representations, treating motion sequences as 2D pixel arrays. Train a CNN in TensorFlow Python on hundreds of recorded instances per gesture. Export to ONNX. Run ONNX inference inside Unity C# at runtime against the same encoding pipeline used during training.
Outcomes
- 50+ unique gestures classified in real time
- Full system completed in under two months
- Used as core technology in Mystic Sign, a VR spell-casting game
- Cross-platform VR support across Quest and SteamVR
Hardest bits
- Encoding continuous 6DOF motion across multiple inputs into a stable 2D image representation
- Keeping the encoding pipeline identical between Python training and Unity inference paths
- Real-time classification at VR frame rates without dropping frames