Robo-Saber: A Full-body Physics-based Virtual Reality Playtesting Agent
Results
Summary
We trained and evaluated Robo-Saber, a physics-based AI player for the VR game Beat Saber using gameplay sequences from the BOXRR-23 dataset aligned with map files from BeatSaver. Robo-Saber is the first physics-based full-body player model for VR and capable of expert-level gameplay. Robo-Saber can simulate diverse and human-like gameplay behaviors which can be used for evaluating and playtesting brand-new maps with zero real players.
We combine a kinematic 3p motion generator with a physics-based tracking controller to build a robot player. For kinematic generation, we build on Categorical Codebook Matching [Starke et al., 2024]. For 3p tracking, we fine-tune PHC [Luo et al., 2023] with custom motion capture data featuring Beat Saber gameplay movements.
Our kinematic 3p generator uses a transformer architecture for both input signal encoder and Gumbel-Softmax VAE. Beat Saber game objects, i.e., colored notes, bomb notes, and obstacles, as well as the history of generated output are used as input signal. The Gumbel-Softmax VAE auto-encodes reference 3p motion segments. The latent categorical distributions resulting from the input signal and the Gumbel-Softmax VAE are matched using a Jensen-Shannon divergence-based loss.
Cite Us
@inproceedings{kim2025robo,
title={Robo-Saber: Full-body Physics-based User Modeling for Virtual Reality},
author={Kim, Nam Hee and Liu, May and H{\"a}m{\"a}l{\"a}inen, Perttu and O'Brien, James and Peng, Xue Bin},
booktitle={arXiv preprint},
pages={1--10},
year={2025}
}
Acknowledgements
We would like to thank Vivek Nair and CyberRamen for technical discussions during the primary stage of this project. We forked AllPoland’s ArcViewer for visualization. Credits for all songs and maps appearing in our study are to be attributed to the original authors.