Hand Tracking
This guide will introduce what NRSDK can & cannot do while tracking hands. The tutorial will demonstrate how to add the hand tracking feature to your app and use your hands as an interaction model.
Notice
Hand Tracking with NRSDK 1.7.0 and above has been fully tested on the following Android phones:
Nreal Dev-Kit
OnePlus:9R / 7T / 8T / 8 Pro 5G
LG:V60 / V50S ThinQ 5G / V50 ThinQ 5G / G9 (Velvet 5G) / Wing
SONY:Xperia 5 II / Xperia 1
SAMSUNG:Galaxy Note20 5G / Galaxy S10+ / Galaxy S20+ 5G / Galaxy Z Fold 2 5G / Galaxy S21 5G / Galaxy Note20 Ultra / Galaxy Note10+ 5G / Galaxy A90 5G
OPPO: Find X2 / Find x2 Pro / Find x3 Pro
ZTE Axon 10 pro
Black Shark 2 Pro
Besides the above, the actual operation on specific models is not guaranteed, although it is supposed to work with most Android phones.
Introduction
NRSDK’s Hand Tracking capability tracks the position of key points of your hands and recognizes hand poses in real-time. Hand poses are shown in the first-person view and used to interact with virtual objects immersively in-world.
Capabilities
The NRSDK can track hands through the world coordinate frame and annotated the position and orientation of twenty-three key points;
The NRSDK currently supports six hand poses from either hand;
While tracking hand poses, the NRSDK also returns to a state of whether there are hands being tracked;
Left/Right hand detection is available in NRSDK;
When using hands as input, the hand’s pose drives a laser cursor-pointer that behaves like the standard controller cursor.
Hand Poses
General Gesture
Select Gesture
As long as the index finger and thumb pinch together (regardless of the pose of other fingers), it is considered a pinch pose.
Gestures above will all be recognized as pinch/select.
System Gesture
Keep this gesture for 1.2s to evoke the home menu.
Both left and right hands will be recognized.
Hand Pointer
Similar to a controller, NRSDK provides a pointer for each hand to
interact with targets. The pointer pose and whether the pointer pose is
being tracked can be obtained from HandState
of each hand.
The pointer pose must meet the following conditions to be correctly tracked:
The hand is recognized
The palm direction is pointing forward
A basic hand pointer style is included in NRHand_R
and NRHand_L
prefabs located in Assets>NRSDK>Prefabs>Hands
. You can also customize the style of Hand
Pointer based on the pointer pose and related data, combined with some
data in NRPointerRaycaster.
Joint
The NRSDK hand tracking system identifies the position of 23 key points on the hand for every recognized hand pose(position and orientation).
Joint Label
Index
Index |
API Name |
---|---|
0 |
Wrist |
1 |
Palm |
2 |
ThumbMetacarpal |
3 |
ThumbProximal |
4 |
ThumbDistal |
5 |
ThumbTip |
6 |
IndexProximal |
7 |
IndexMiddle |
8 |
IndexDistal |
9 |
IndexTip |
10 |
MiddleProximal |
11 |
MiddleMiddle |
12 |
MiddleDistal |
13 |
MiddleTip |
14 |
RingProximal |
15 |
RingMiddle |
16 |
RingDistal |
17 |
RingTip |
18 |
PinkyMetacarpal |
19 |
PinkyProximal |
20 |
PinkyMiddle |
21 |
PinkyDistal |
22 |
PinkyTip |
Joint Orientation
Common Usage Of Hand Tracking
Sample Use Case:
1 //returns true if input source switch to hand tracking success
2 bool switchToHandTracking = NRInput.SetInputSource(InputSourceEnum.Hands);
3
4 //returns true if input source switch to controller success bool
5 switchToController = NRInput.SetInputSource(InputSourceEnum.Controller);
6
7 //returns true if hand tracking is running bool isRunning =
8 NRInput.Hands.IsRunning;
9
10 //returns the NRHand of right-handness NRHand hand =
11 NRInput.Hands.GetHand(HandEnum.RightHand);
12
13 //returns true if user is now performing system gesture bool
14 isPerformingSystemGesture = NRInput.Hands.IsPerformingSystemGesture();
Details Of HandState:
1 //returns the HandState of right-handness
2 HandState handState = NRInput.Hands.GetHandState(HandEnum.RightHand);
3
4 //returns the handness of this hand
5 HandEnum handEnum = handState.handEnum;
6
7 //returns true if this hand is tracked
8 bool isTracked = handState.isTracked;
9
10 //returns the start pose of hand ray pointer
11 Pose pointerPose = handState.pointerPose;
12
13 //returns ture if hand ray pointer pose is valid
14 bool pointerValid = handState.pointerPoseValid;
15
16 //returns true if this hand is performing pinching
17 bool isPinching = handState.isPinching;
18
19 //returns the current pinch strength value of hand. The range is from 0 to 1
20 float pinchStrength = handState.pinchStrength;
21
22 //returns the current gesture of hand
23 HandGesture handGesture = handState.currentGesture;
24
25 //returns the pose which contains position and orientation of thumb tip joint
26 Pose thumbTipPose = handState.GetJointPose(HandJointID.ThumbTip);
Tutorial
Enabling Hand Tracking
Create a new project in Unity with NRSDK. Refer to Quickstart for Android Unity for more setting up instructions.
Delete the
Main Camera
from the scene hierarchy.Find
NRCameraRig
andNRInput
prefab fromAssets>NRSDK>Prefabs>NRCameraRig
. Drag them to the scene hierarchy.Select the
NRInput
GameObject in theHierarchy
window to open theInspector
window, and chooseHands
asInput Source Type
.Find
NRHand_R
andNRHand_L
fromAssets>NRSDK>Prefabs>Hands
. Add them as child GameObjects ofLeft
andRight
anchor inNRInput
correspondingly.Now you are ready for hand tracking, refer to Building a Project with User Input for adding more interactions with objects.
Samples are included in the plugin. Please refer to
Assets>NRSDK>Demos>HandTracking
for details.
Requirements & Limits
Hand tracking SDK uses on-board camera(s) to detect hands, so make sure hands are visible from the camera.
We are improving the accuracy of the model, please pay attention to the following situations if your hand is not detected:
Backgrounds
Avoid complicated backgrounds, solid backgrounds are preferred;
Avoid backlight or low light, or unbalanced lighting conditions in camera frame;
Gesture
Avoid stacked or interlaced fingers for either hand;
Avoid the hands of different people;
Avoid fast-moving;
There are chances that the orientations of the joints are recognized converted. If so, please move your hands out of the visible fields and move them back again.