NRSDK Coordinate System for Plugins

This document describes the definition of coordinate systems of the Nreal glass components. Please note that this document is not for application developers (upward rendering purposes), but for third-party software solution providers (downward plugins).

Glass Components and Coordinate Systems

  • The Nreal glass consists of the following key components

    • 2 x Grayscale Cameras

    • 2 x Display Cameras

    • Head / IMU

    • RGB Camera

  • The placement of the above components and their corresponding coordinate systems, as defined in NRSDK, are as follows image0

Interface for Head(IMU) Pose

  • Interface function: This function returns the 6dof pose, of the target_device in the source_device.

1   // head pose(IMU,"NRCameraRig" transform in Unity)
2   Pose headpose = NRFrame.HeadPose;

  • Enum for component (device) names

 1   public enum NativeDevice
 2   {
 3       /// <summary>
 4       /// Left display.
 5       /// </summary>
 6       LEFT_DISPLAY = 0,
 7
 8       /// <summary>
 9       /// Right display.
10       /// </summary>
11       RIGHT_DISPLAY = 1,
12
13       /// <summary>
14       /// RGB camera.
15       /// </summary>
16       RGB_CAMERA = 2,
17
18       /// <summary>
19       ///  The left grayscale camera.
20       ///  Only supported in the version of preview.
21       /// </summary>
22       LEFT_GRAYSCALE_CAMERA = 3,
23
24       /// <summary>
25       ///  The right grayscale camera.
26       ///  Only supported in the version of preview.
27       /// </summary>
28       RIGHT_GRAYSCALE_CAMERA = 4,
29   }

Example 1: Getting the Pose Between the Head/IMU and the Left Display, Right Display, RGB Camera

 1   // camera external parameters
 2   var eyePoseFromHead = NRFrame.EyePoseFromHead;
 3
 4   // left display local pose from "NRCameraRig"
 5   Pose left_display = eyePoseFromHead.LEyePose;
 6
 7   // right display local pose from "NRCameraRig"
 8   Pose right_display = eyePoseFromHead.REyePose;
 9
10   // rgb camera local pose from "NRCameraRig"
11   Pose rgb_camera = eyePoseFromHead.RGBEyePose;

Example 2: Getting the Extrinsics Between the Head/IMU and the Left Grayscale Camera

1   // This interface only supported in the version of Preview, not release version
2   Pose leyeGrayEyePose = NRFrameExtension.GrayEyePoseFromHead.LEyePose;

image2

Head/IMU Pose in Global Coordinate System

  • The global coordinate frame of the tracking system is as follows

image3

Camera Coordinate System (OpenGL)

  • The definition of the camera’s coordinate frame in the NRSDK follows the OpenGL convention, i.e. X-right, Y-upward, Z-backward. For developing and using computer vision algorithms, you may need to convert the camera coordinates to the OpenCV convention, which simply requires negating the Y and Z coordinates.

image4

Image Pixel Coordinate System and Camera Intrinsics (OpenCV)

  • The definition of the image pixel coordinates and the camera intrinsics in the NRSDK follows the OpenCV convention. The image data is stored row wise in memory.

image5

  • Interface for getting the camera’s intrinsic parameters

1   NRFrame.GetDeviceIntrinsicMatrix(NativeDevice device);

Wireframes

image6 image7 image8