Finger-worn IMUs have been shown to be very accurate at detecting touch events on physical surfaces, enabling input to physically-bound XR interfaces. In this work, we describe a new XR headset method for detecting finger touch events on ad hoc surfaces, without needing the user to wear anything on their fingers or hands. Instead, our method uses a pair of high-speed headset cameras to track and triangulate the position of the user’s fingertips in 3D space. We take the first and second derivatives of this signal to estimate a finger’s velocity and acceleration. At almost 800 Hz, our system’s output is similar to a worn IMU, but with bare hands. Using this signal, we can then detect rapid and characteristic decelerations of the finger, indicative of touch.

Research Team: Nathan DeVrio, Vimal Mollyn, Chris Harrison