Hand tracking in virtual reality allows users to interact with virtual objects and environments using their hands. This technology, while simple in real life, is incredibly sophisticated for virtual reality. When achieved, the virtual reality experience gets more immersive and the VR more compelling. What in actuality is hand tracking and how does it work?
In this article you will discover:
- What is Hand tracking
- What is camera optical tracking?
- What is LiDAR laser tracking?
- The difference between optical camera tracking and LiDAR laser tracking
- What is inside-out tracking?
- What is outside-in tracking?
- Conclusion
What is hand tracking
Hand tracking is when the headset cameras, LiDAR array, or external sensor stations track your hand’s position, depth, pace, and orientation. This tracking data is then analyzed and translated into a virtual, real-time representation of your hands and their movements inside the virtual world. This representation is then sent to whatever application or video game you are playing, enabling you to naturally interact with the world using your hands.
What hand tracking technologies exist?
There are two ways of tracking your hands in virtual reality: from the headset itself or external sensors.
What is camera optical tracking?
When the tracking originates from a camera array, it is called optical tracking.
Optical hand tracking uses the difference in pixels to determine your hands’ movements. As a result, it requires more processing for software to translate into your virtual hands.
Likewise, optical tracking is bound by the field of view of the cameras (i.e., the active tracking area), their sensitivity to light, and their capabilities for capturing motion (frame rate). Usually, optical tracking is cheaper to install, while a camera array also can enable video passthrough for the wearer.
What is LiDAR laser tracking?
Alternatively, LiDAR tracking (light detection and arranging) pulses lasers to determine the distance, position, and movement of the environment relative to your hands. This tracking method is more precise, as lasers are faster than a camera’s assessment of pixels, while the bounce-back of the laser into the LiDAR sensor immediately generates a 3D representation of what is tracked.
Alternatively, the tracking can also be done by infrared LEDs, which work the same way as LiDAR. Infrared LEDs, called IR, combined with optical, camera tracking produces high-fidelity tracking. A solution Ultraleap is doing.
For example, Varjo’s XR-3 headset uses LiDAR depth sensing in combination with optical tracking to achieve precision hand tracking. While an IR accessory is included in the hand tracking setup for the Pico Neo 3 Pro and Pro Eye accessory kit.
The difference between optical camera tracking and LiDAR laser tracking
Because LiDAR uses bouncing lasers – similar to how echolocation works – the quality of tracking is not lost based on distance. Oppositely, because cameras use the changes of pixels to determine movement and position, the quality of the pixels diminishes based on the distance between the camera and your hands.
Types of hand tracking systems used in Virtual Reality
Now that the types of tracking are known it is important to understand the different types of hand tracking systems used in virtual reality
What is inside-out tracking?
Tracking originating from the headset is inside-out tracking and is found in standalone VR headsets. For this to function, either a set of cameras or LiDAR sensors on the headset keep track of your hands in relative position to everything, crunching this into raw positional data.
Inside-out tracking works similar to a lighthouse in that a 3D representation of the world must be generated from only one direction. This lessens tracking quality and precision in comparison to outside-in tracking but frees up the wearer to not be constrained within a designated ‘tracking zone’.
What is outside-in tracking?
When tracking is conducted from multiple external sensors (usually LiDAR or IR technology) forming a ‘tracking zone’, it is called outside-in tracking.
Outside-in tracking can generate a much more precise representation of a person’s position and movements (hands included) from external sensors and the headset itself. A 3D map of your hands within the tracking zone is much easier to compute as your hands are tracked from multiple angles.
Conclusion
Hand tracking lowers the barriers to virtual reality adoption because natural interaction is possible. It improves communication within the virtual world because we talk with hand gestures and it creates an incredible sense of immersion.No matter the hand tracking solution used, it is a critical part of virtual reality. Soon gloves and controllers will step aside for natural hand interactions, and we are here for it.
