Esky - Project North Star

Search…
Esky
Project Esky aims to be the software companion to allow development with the Deck X/North Star Display system out of the box! (Utilizing the t265/1 or a ZED system for 6DoF tracking)
Project Esky is a complete unity framework that handles
Rendering (with 2D and 3D (V1 and V2) Optical Calibrations)
MRTK Integration with the Leap Motion controller (aligned with the user's view)
6DoF Head Tracking + Re-Localization events (Save map, load map, add persistence, callbacks, etc.)
Peer to Peer based co-localization and networking at the click of a button (Provided devices are on the same local network)
Spatial mapping (Via the ZED SDK, with future plans to generate point clouds via the t261/t265 sensor)
Temporal Reprojection (2D calibration only)
​Project Esky on Github
Project Esky on Research Gate​
 Written Guide (Config File Setup)
In order to use Esky you'll need Unity, A Northstar headset, and a 2D Calibration or 3D Calibration file. 
Esky runs within the Unreal engine, but the Unity integration is more complete
Esky's Unity Integration uses Unity 2020.3.X, if it complains that you're using a newer version, as long as you're using Unity 2020.3 it should be ok to upgrade. We currently use 2020.3.11f1
Note that the EskySettings.json file is located in the root of your unity project folder. (Not visible within Unity)
For the V2 calibration, Copy the contents of your NorthStarCalibration.json file into Esky's EskySettings.json file, you will be replacing the following values within the v2CalibrationValues section:
left_uv_to_rect_x 
left_uv_to_rect_y 
right_uv_to_rect_x 
right_uv_to_rect_y
​
For the V1 calibration, Copy the contents of your NorthStarCalibration.json file into Esky's 
EskySettings.json file, you will be replacing all of the values within the v1CalibrationValues section

Make sure Northstar Display is not flipped via windows display manager, additionally it can be helpful to have your Northstar Display setup to the far right of your monitor setup, though if necessary it can be placed anywhere, you'll just need to take note of the position for later. 

Then, within the EskySettings.Json file, edit the displayWindowSettings so that the DisplayXLoc and DisplayYLoc  reflect the position of your northstar display relative to your main monitors. 
In the below example, Monitor #2 is my NorthStar, and Monitor #3 is marked as my 'Main Display'. Since my 'Main Display' is 1920x1080 pixels in size, my DisplayXLoc and DisplayYLoc values will be 0, and 1080, respectively
​
​


Save the EskySettings.json file. You're now free to proceed to open the Unity Project to complete the calibration
Hand Alignment and MRTK Setup
Before you can begin doing cool northstar stuff, you'll need to align your hands so that the virtual image matches the real world, and configure the MRTK to your specific North Star Setup.
For V1:
Observe the following section of your V1 (3D) calibration json:

1
"leapTracker": {
2
    "name": "Leap Odometry Origin",
3
    "serial": "",
4
    "localPose": {
5
      "position": {
6
        "x": 0.0,
7
        "y": 0.039777886122465137,
8
        "z": 0.0804821103811264
9
      },
10
      "rotation": {
11
        "x": 0.30401188135147097,
12
        "y": -0.016626091673970224,
13
        "z": 0.00530654564499855,
14
        "w": 0.9525083899497986
15
      }
16
    }
17
  }
Copied!
You will need to copy the position, and rotation values into the following area within the EskySettings.json file. the position maps to TranslationEyeToLeapMotion, and the rotation maps to RotationEyeToLeapMotion.
1
"myOffsets": {
2
        "TranslationEyeToLeapMotion": {
3
            "x": -0.002040371298789978,
4
            "y": 0.07525337487459183,
5
            "z": 0.09259902685880661
6
        },
7
        "RotationEyeToLeapMotion": {
8
            "x": 0.04397745057940483,
9
            "y": 0.02938256785273552,
10
            "z": -0.08629350364208222,
11
            "w": 0.9948649406433106
12
        },
13
}
Copied!
Save when complete, you can see an example of a completed EskySettings.json file in the Extras section
For V2:
Open the Unity project, then navigate to Assets/Scenes/Calibration/LMHandCalibration.unity
Hit play, then, click on the unity Game window (so that unity can receive the keyboard input)
Alignment is pretty straightforward, following the instructions displayed in your headset:
1) Hold your right hand up, so that the virtual hand appears in the center of the screen
2) Hold the space bar
3) While holding space, translate your right hand so that it aligns with the frozen virtual hand, 
NOTE: Try not to move your fingers as you do this
4) Once aligned, release the space bar
5) Repeat steps 1-4 3 or 4 times, the alignment system will let you know when it has collected enough samples. 
If you are happy with the alignment, hit 's' to save, if not, stop unity (hit the play button again) then hit play to start the process again, repeating steps 1-5
Further adjustment can be done with the arrow keys
Setting up the MRTK Example
Now we get to the fun stuff! As of RC1, the MRTK handles all of the configuration for Project Esky!
Open Project Esky in Unity, then navigate to Assets/Scenes/Examples/HandInteractionExamples.unity
If you look at the Scene Heirarchy, you will notice the MixedRealityToolkit gameobject. 
​
The inspector will show the Mixed Reality Toolkit configuration. Click Input -> Input Data Providers

You will see the following window for configuring all of esky's settings.

The settings are explained as follows:
Rig To Use: This controls which optical setup is used for your northstar (V1, V2), Project Ariel, or a custom rig can be selected.

Filter System to Use: We work hard to develop Project Esky, and while we have a newer (and in our opinion better designed) pose filtering system, you can change this value to revert back to the old filtering pipeline here

Reprojection Settings (V2/Ariel Only): This enables/disables the late-stage temporal reprojection built into Project Esky's native rendering pipeline.
Native Shader To Use (V2/Ariel Only, Currently not implemented): This changes the undistortion method used by the native rendering pipeline, we recommend not editing this.
Target Frame Rate (V2/Ariel Only): This changes the target frame rate for Unity. You can select 120, 90, 60, and 30 frames per second!
NOTE: This frame rate is independent of the native rendering pipeline that handles composition, which always runs at 120 frames per second!
Leap Controller Orientation: This changes the way the MRTK handles the leapmotion controller, we recommend leaving this as 'Esky'
Enter/Exit pinch distance: This changes the distance between the index and thumb before the MRTK considers a pinch start/finish action (in meters).
Save after stopping editor: The default behaviour of Project Esky is to dump your current MRTK settings back into the EskySettings.json file, which is also copied to any build directory when you built your unity project. You can disable this behaviour but we don't recommend it!
Use Tracker Offsets (Not Implemented): This places the virtual camera relative to the 6DoF tracker, good for external between-sensor calibrations.
Uses Camera Preview: This changes whether you intend to use image passthrough preview with the t261, keep in mind you must have USB 3.0 connected to your NorthStar in order to use it!
Uses External RGB Camera: For those with an RGB sensor, you can enable this to use the RGB image passthrough, unless you know what you're doing Keep this disabled!!
All Done?
Then simply hit 'play' and you're good to go :D
Extras
Below is an Example of my complete EskySettings.Json File, with the updated V1 and V2 calibration for a 25cm focal length optical setup.
Note: Some of the values you see are either controlled by the MRTK in editor, or not yet in use.
1
{
2
    "rigToUse": 0,
3
    "targetFrameRate": 120,
4
    "reprojectionSettings": 0,
5
    "nativeShaderToUse": 0,
6
    "myOffsets": {
7
        "TranslationEyeToLeapMotion": {
8
            "x": -0.002040371298789978,
9
            "y": 0.07525337487459183,
10
            "z": 0.09259902685880661
11
        },
12
        "RotationEyeToLeapMotion": {
13
            "x": 0.04397745057940483,
14
            "y": 0.02938256785273552,
15
            "z": -0.08629350364208222,
16
            "w": 0.9948649406433106
17
        },
18
        "TranslationFromTracker": {
19
            "x": 0.0,
20
            "y": 0.0,
21
            "z": 0.0
22
        },
23
        "RotationFromTracker": {
24
            "x": 0.0,
25
            "y": 0.0,
26
            "z": 0.0,
27
            "w": 1.0
28
        },
29
        "RGBSensorTranslationFromTracker": {
30
            "x": 0.0,
31
            "y": 0.0,
32
            "z": 0.0
33
        },
34
        "RGBSensorRotationFromTracker": {
35
            "x": 0.0,
36
            "y": 0.0,
37
            "z": 0.0,
38
            "w": 1.0
39
        }
40
    },
41
    "v2CalibrationValues": {
42
        "left_uv_to_rect_x": [
43
            -0.6252081394195557,
44
            0.3324834108352661,
45
            1.299890398979187,
46
            -0.8253355622291565,
47
            -1.0325636863708497,
48
            7.521162033081055,
49
            -13.799334526062012,
50
            7.8558759689331059,
51
            2.1261489391326906,
52
            -14.113080978393555,
53
            26.34162139892578,
54
            -14.930045127868653,
55
            -1.3233585357666016,
56
            8.506293296813965,
57
            -15.606451034545899,
58
            8.893977165222168
59
        ],
60
        "left_uv_to_rect_y": [
61
            -0.37523153424263,
62
            -0.34262487292289736,
63
            0.7635243535041809,
64
            -0.6682243347167969,
65
            0.38151007890701296,
66
            4.049591541290283,
67
            -7.806105136871338,
68
            4.473989486694336,
69
            1.0507723093032838,
70
            -8.660529136657715,
71
            15.536046981811524,
72
            -8.565287590026856,
73
            -0.492713987827301,
74
            4.902211666107178,
75
            -8.764571189880371,
76
            4.757476329803467
77
        ],
78
        "right_uv_to_rect_x": [
79
            -0.7759054899215698,
80
            1.748594880104065,
81
            -0.8957473039627075,
82
            0.5134794116020203,
83
            -0.06296022981405258,
84
            0.5289550423622131,
85
            -1.9657267332077027,
86
            0.9775575399398804,
87
            0.34445589780807497,
88
            -2.996290445327759,
89
            6.617568016052246,
90
            -3.72568416595459,
91
            -0.33747774362564089,
92
            2.3275089263916017,
93
            -4.799829483032227,
94
            2.7676336765289308
95
        ],
96
        "right_uv_to_rect_y": [
97
            -0.40155869722366335,
98
            0.0788295716047287,
99
            0.18848803639411927,
100
            0.03440302610397339,
101
            1.162545919418335,
102
            -0.0871603861451149,
103
            -0.47248953580856326,
104
            0.2951107323169708,
105
            -0.28124600648880007,
106
            -0.8912697434425354,
107
            1.6107137203216553,
108
            -0.6361001133918762,
109
            0.19606836140155793,
110
            0.5201933979988098,
111
            -0.8043093681335449,
112
            0.28155842423439028
113
        ],
114
        "left_eye_offset": [
115
            0.0,
116
            0.0
117
        ],
118
        "right_eye_offset": [
119
            0.0,
120
            0.0
121
        ]
122
    },
123
    "v1CalibrationValues": {
124
        "leftEye": {
125
            "ellipseMinorAxis": 0.24494899809360505,
126
            "ellipseMajorAxis": 0.3099985122680664,
127
            "screenForward": {
128
                "x": 0.5140379071235657,
129
                "y": -0.2891080379486084,
130
                "z": 0.8075775504112244
131
            },
132
            "screenPosition": {
133
                "x": -0.07188151031732559,
134
                "y": -0.008429261855781079,
135
                "z": 0.026248455047607423
136
            },
137
            "eyePosition": {
138
                "x": -0.029943088069558145,
139
                "y": -0.003366167191416025,
140
                "z": -0.00427968380972743
141
            },
142
            "eyeRotation": {
143
                "x": 7.395533370627759e-32,
144
                "y": 1.7763568394002506e-15,
145
                "z": 1.1241009818395605e-14,
146
                "w": 1.0
147
            },
148
            "cameraProjection": {
149
                "x": -0.699999988079071,
150
                "y": 0.699999988079071,
151
                "z": 0.30000001192092898,
152
                "w": -1.399999976158142
153
            },
154
            "sphereToWorldSpace": {
155
                "e00": 0.05101080238819122,
156
                "e01": 0.06071346998214722,
157
                "e02": 0.29330453276634219,
158
                "e03": -0.11233785003423691,
159
                "e10": 0.0,
160
                "e11": 0.23695310950279237,
161
                "e12": -0.07855913043022156,
162
                "e13": 0.026850363239645959,
163
                "e20": -0.23957861959934236,
164
                "e21": 0.012927045114338398,
165
                "e22": 0.062450066208839419,
166
                "e23": -0.046845193952322009,
167
                "e30": 0.0,
168
                "e31": 0.0,
169
                "e32": 0.0,
170
                "e33": 1.0
171
            },
172
            "worldToScreenSpace": {
173
                "e00": 1.8454549312591553,
174
                "e01": 14.493172645568848,
175
                "e02": 4.013805866241455,
176
                "e03": 0.14946463704109193,
177
                "e10": -14.294207572937012,
178
                "e11": -0.6365562677383423,
179
                "e12": 8.87063980102539,
180
                "e13": -1.265695571899414,
181
                "e20": 0.5140379071235657,
182
                "e21": -0.2891080975532532,
183
                "e22": 0.8075775504112244,
184
                "e23": 0.013315193355083466,
185
                "e30": 0.0,
186
                "e31": 0.0,
187
                "e32": 0.0,
188
                "e33": 1.0
189
            }
190
        },
191
        "rightEye": {
192
            "ellipseMinorAxis": 0.24494899809360505,
193
            "ellipseMajorAxis": 0.3099985122680664,
194
            "screenForward": {
195
                "x": -0.42177313566207888,
196
                "y": -0.18166834115982057,
197
                "z": 0.8883152604103088
198
            },
199
            "screenPosition": {
200
                "x": 0.07054391503334046,
201
                "y": -0.009491059929132462,
202
                "z": 0.028982438147068025
203
            },
204
            "eyePosition": {
205
                "x": 0.029943088069558145,
206
                "y": -0.003366167191416025,
207
                "z": -0.00427968380972743
208
            },
209
            "eyeRotation": {
210
                "x": 7.395533370627759e-32,
211
                "y": 1.7763568394002506e-15,
212
                "z": 1.1241009818395605e-14,
213
                "w": 1.0
214
            },
215
            "cameraProjection": {
216
                "x": -0.699999988079071,
217
                "y": 0.699999988079071,
218
                "z": 0.30000001192092898,
219
                "w": -1.399999976158142
220
            },
221
            "sphereToWorldSpace": {
222
                "e00": 0.05101081356406212,
223
                "e01": -0.060713451355695727,
224
                "e02": -0.29330453276634219,
225
                "e03": 0.11538401991128922,
226
                "e10": 0.0,
227
                "e11": 0.23695312440395356,
228
                "e12": -0.07855910062789917,
229
                "e13": 0.025588704273104669,
230
                "e20": 0.23957861959934236,
231
                "e21": 0.0129270413890481,
232
                "e22": 0.062450066208839419,
233
                "e23": -0.04654233157634735,
234
                "e30": 0.0,
235
                "e31": 0.0,
236
                "e32": 0.0,
237
                "e33": 1.0
238
            },
239
            "worldToScreenSpace": {
240
                "e00": -1.1463184356689454,
241
                "e01": 14.899345397949219,
242
                "e02": 2.502774715423584,
243
                "e03": 0.1497398465871811,
244
                "e10": -15.211103439331055,
245
                "e11": 0.041456401348114017,
246
                "e12": -7.213770389556885,
247
                "e13": 1.2825169563293458,
248
                "e20": -0.42177313566207888,
249
                "e21": -0.18166831135749818,
250
                "e22": 0.8883152604103088,
251
                "e23": 0.002283765934407711,
252
                "e30": 0.0,
253
                "e31": 0.0,
254
                "e32": 0.0,
255
                "e33": 1.0
256
            }
257
        }
258
    },
259
    "displayWindowSettings": {
260
        "DisplayXLoc": 0,
261
        "DisplayYLoc": 1080,
262
        "DisplayWidth": 2880,
263
        "DisplayHeight": 1600,
264
        "EyeTextureWidth": 1600,
265
        "EyeTextureHeight": 1440
266
    },
267
    "sensorModuleCalibrations": {
268
        "camID": 0,
269
        "fx": 0.0,
270
        "fy": 0.0,
271
        "cx": 0.0,
272
        "cy": 0.0,
273
        "d1": 0.0,
274
        "d2": 0.0,
275
        "d3": 0.0,
276
        "d4": 0.0,
277
        "SensorWidth": 0,
278
        "SensorHeight": 0,
279
        "SensorChannels": 0,
280
        "SensorFoV": 0.0
281
    },
282
    "usesExternalRGBCamera": false,
283
    "UsesCameraPreview": true,
284
    "UseTrackerOffsets": true
285
}
Copied!
​
Software - Previous
Getting Started with Software
Next - Software
Monado
Last modified 10mo ago
Export as PDF
Copy link
Contents
Written Guide (Config File Setup)
Hand Alignment and MRTK Setup
Setting up the MRTK Example
All Done?
Extras