Given that Northstar is a project that combines many different fields of research and expertise, it's unlikely that any one person will ever know or be able to recall all the terms, acronyms and technical jargon that is used to discuss these various industries. This glossary intends to define some of the more particular terms, starting with the field of electrical engineering, but will eventually grow to cover other topics in more depth, like firmware, software etc. This page is open to pull requests, please feel free to submit PRs with more terms!

General Terms

XR

XR is an umbrella term for Virtual Reality, Augmented Reality and Mixed Reality. Some say it stands for Extended Reality, others say the X is a variable, check twitter if you want to see people argue about it for literal years. In general, XR is the term the northstar community uses when discussing VR/AR or MR. For more general discussions we tend to enjoy using "Spatial Computing".

Combiner

A Combiner is an Optical component that takes rays of light and focuses them to a single end point.

6DOF

So, DOF is a fun one because there are two meanings for the acronym. In traditional camera work it means Depth of Field. For Northstar and other XR devices it means "Degrees of Freedom". In a 6-DOF system you can rotate on 3 Axis and also move on 3-axis.

IPD

IPD stands for Interpupilary distance. This is the term for the distance between your pupils, which is useful information when correcting for optical distortion.

Eye Box

The Eye Box is the area in which the image from a headset is clearly visible. Northstar has a fairly large Eye Box meaning you can adjust the headset to be more comfortable without worrying too much about whether or not you've got it positioned just right.

FOV

FOV stands for field of view, this is usually referring to how much you can see through a headset, and can also be used to describe the amount a camera can see. FOV can be measured horizontally, vertically or diagonally.

Latency

Latency is an overall term for how long a signal takes to get from one location to another, this is applicable to many areas of the northstar project. For example there is latency from when an image is rendered by the computer to when it is displayed on the northstar screen, or latency from when the image from a tracking camera makes its way to the onboard processor. In short, latency is time, and the longer something takes the more latency it has.

VAC

VAC stands for the Vergence Accomodation Conflict. This term describes the difficulty our eyes have in focusing on objects that optically appear a certain distance away and via stereo disparity appear a different distance away.

DP

DP stands for Display Port. This is the display signal type that northstar uses. In general, converting from DP to HDMI is easy, converting from HDMI to DP is incredibly difficult and rarely works, in our experience it is a waste of time trying to look into HDMI to DP adapters.

DP Alt Mode

DP alt mode is a specific subsection of the USB Type C spec that allows it to carry a display port signal.

PD

PD stands for power delivery. This is another subsection of the USB specification that specifies a USB port has the neccessary components to supply more power than a standard port.

Electrical Engineering Terms 🔋

PCB

PCB stands for Printer Circuit Board. These are typically sheets of multi-layered copper that contain traces which allow electrical signals to be transmitted on the board. These signals can include power and data.

FPC

FPC stands for Flexible Printed Circuit, these are similar to PCBs, but the main difference is that these are flexible and can bend. These cables are used to connect the displays and other components and are often used in board to board connections when PCBs may be located in different locations that a rigid connection isn't suited for.

MCU

MCU stands for Micro-controller. Microcontrollers are little self-contained computers in a chip that execute programs called firmware. These chips are on PCBs, or printed circuit boards. The programs control various peripherals that are either built into the chip or connected externally. Popular examples of Micro-controllers include Arduino and Rasberry Pi.

MIPI

MIPI is a general industry group and standard for display signal interfaces and camera signal interfaces, among others. For Northstar's case we care mostly about display and camera signals. You can read more about MIPI here.

HID

HID stands for Human Interface Device, it a set of standards used to connecting peripherals like keyboards, mice etc.. intended for 'driverless' operation. You can read about the specification here, in addition Microsoft have a great resource on HID here.

I2C

I2C stands for Inter-Integrated Circuit, It is widely used for attaching lower-speed peripheral ICs to processors and microcontrollers in short-distance, intra-board communication.

SPI

The Serial Peripheral Interface (SPI) is a synchronous serial communication interface specification used for short-distance communication, primarily in embedded systems. The interface was developed by Motorola in the mid-1980s and has become a de facto standard. Typical applications include Secure Digital cards and liquid crystal displays.You can learn more about SPI here.

DP to MIPI Bridge

A DP to MIPI bridge is a specific chip designed to convert signal from a DP input to a MIPI signal to be read by displays.

Software Terms 🧑🏽‍💻

Git

Git is a common type of Source Control, Allowing developers to maintain a history of their code which is incredibly helpful in diagnosing and figuring out undesired results of changes to that code. You can learn more about Git here: https://docs.gitlab.com/ee/topics/git/

Pull Request

A Pull Request is a term used to describe the act of submitting a piece of code for review and merging into the main code base. You can learn more about pull requests here

Runtime

A runtime is an intermediary process between end user applications and hardware. In Northstar's case runtimes can include SteamVR, Monado and Esky.

OpenXR Runtime

An OpenXR runtime is software which implements the OpenXR API. There may be more than one OpenXR runtime installed on a system, but only one runtime can be active at any given time.

OpenXR

OpenXR is an API (Application Programming Interface) for XR applications. XR refers to a continuum of real-and-virtual combined environments generated by computers through human-machine interaction and is inclusive of the technologies associated with virtual reality (VR), augmented reality (AR) and mixed reality (MR). OpenXR is the interface between an application and a runtime. The runtime may handle such functionality as frame composition, peripheral management, and raw tracking information.

SLAM

SLAM is a term that originated in robotics meaning Simulatenous Localization and Mapping. It's become the generic term for systems that can see their environment, localize themselves to a previously known location, and generate a map of that environment for other devices to localize against.

VIO

VIO stands for Visual Intertial Odometry, VIO is an important part of the processing needed for SLAM, but does not provide localization or mapping. VIO is soley performing the act of combing visual data with inertial measurements from an IMU.

Hardware/Manufacturing Terms 🛠

IMU

IMU stands for Inertial Measurement Unit, it is a device that can read out vectors like acceleration, and gravity.In Northstar's case these devices are used for 3DOF movement and can help with VIO and SLAM https://en.wikipedia.org/wiki/Inertial_measurement_unit

What is Project North Star?

Project North Star is an open source augmented reality headset. It was initially released by LeapMotion (now UltraLeap) in 2018 and is currently supported by its open source community.

There have been several iterations of the headset since the project began. The latest build is Northstar Next.

Augmented Reality vs Virtual Reality

As opposed to Virtual Reality, where the user's view is generated entirely by a computer, Augmented Reality projects a rendered image over the real-world perspective of the user. This gives an effect of the rendered objects existing in the real world.

To achieve this effect, an AR headset can use either a "pass-through" or a transparent optical system. A pass-through display streams the outside world to an enclosed LCD/LED screen. A transparent optical system projects an image which is reflected off a lens in front of the user's field of view. Project North Star uses a transparent optical system.

How the headset works

Though an AR headset is a very complex device, we can simplify it by separating its hardware and software into different components, each of which serve a specific function.

Project North Star develops and/or sources the parts for each of these components, and provides instructions to combine them into a complete headset. Each headset version is comprised of different iterations of some or all of these parts.

Each of these components can separated into the following categories:

• Display: The hardware that produces the images.

• Optics: Lenses which bend light from the image so it is in-focus for the user.

• Sensors: Hardware and software that track the user's position and orientation through space.

• Hand-tracking: Hardware and software that allows the user to interact with the artificial objects and environment.

• Integrator: Hardware and software that allows each component to communicate with each other.

• Runtime: Software that acts as an intermediary between user applications and hardware.

• Assembly: Hardware that houses all the components to create an complete headset.

This separation of responsibility allows for different open or closed source solutions to be used in a headset, resulting in a modular design.

Sourcing and Manufacturing

The challenges in this space are not limited to technological problems. Few people have the means to fabricate Printed Circuit Boards (PCBs), lenses, and LCD screens. So maintaining Project Northstar also means project contributors find and work with manufacturers and distributors for bespoke electronics. This influences both the design and availability of the headsets.

Advantage of Open Source Design

With an active community, open documentation, and modular design, Project Northstar's headsets are customizable and repairable in a way that no other closed-source headset is. Anyone can contribute to the project. So the more its users customize and upgrade their devices, the more the project benefits as a whole.

A primary goal of Project NorthStar is to think and work in collaboration, so we can transcend the potential of each of us, and others. We aim to be inclusive to the largest number of contributors, with the most varied and diverse backgrounds possible. We are committed to providing a friendly, safe, and welcoming environment for all, regardless of gender, sexual orientation, skills and education, ability, ethnicity, socioeconomic status, and religion (or lack thereof). This Code of Conduct outlines our expectations for all those who participate in our community, as well as the consequences for unacceptable behavior. We invite all those to help us create safe and positive experiences for everyone, and we also include ways for our attendees to report any violations. Together we can make a fun, harassment-free, magical experience for everyone, regardless of gender, gender identity, sexual orientation, disability, physical appearance, body size, race, or religion.

A supplemental goal of this Code of Conduct is to increase open [source/culture/tech] citizenship by encouraging participants to recognize and strengthen the relationships between our actions and their effects on our community. Communities mirror the societies in which they exist, and positive action is essential to counteract the many forms of inequality and abuses of power that exist in society. If you see someone who is making an extra effort to ensure our community is welcoming, friendly, and encourages all participants to contribute to the fullest extent, we want to know..

‍The following behaviors are expected and requested of all community members:

• Communicate! Participate in an authentic and active way. In doing so, you contribute to the health and longevity of this community.

• Exercise consideration and respect in your speech and actions. Come from a place of understanding.

• Attempt collaboration before a conflict.

• Refrain from demeaning, discriminatory, or harassing behavior and speech.

• Be mindful of your surroundings and your fellow participants. Alert community leaders if you notice a dangerous situation, someone in distress, or violations of this Code of Conduct, even if they seem inconsequential.

The following actions are considered harassment and are unacceptable within our community:

• Being a jerk. Being mean. Unkind intentions.

• Violence, threats of violence or violent language directed against another person.

• Sexist, racist, homophobic, transphobic, ableist or otherwise discriminatory jokes and language.

• Posting or displaying sexually explicit or violent material, posting or threatening to post other people’s personally identifying information ("doxing").

• Personal insults, particularly those related to gender, sexual orientation, race, religion, or disability.

• Inappropriate photography or recording. Inappropriate physical contact. You should have someone’s consent before touching them.

• Unwelcome sexual attention. This includes sexualized comments or jokes; inappropriate touching, groping, and unwelcome sexual advances.

• Deliberate intimidation, stalking or following (online or in person).

• Advocating for, or encouraging, any of the above behavior.

• Sustained disruption of community events, including talks and, presentations.

Don't be that person. We help people help people.

Unacceptable behavior from any community member, including sponsors and those with decision-making authority, will not be tolerated. Anyone asked to stop unacceptable behavior is expected to comply immediately. If a community member engages in unacceptable behavior, moderators may take any action they deem appropriate, up to and including a warning, temporary ban, or permanent expulsion from the community/event without warning. You can make a report either personally or anonymously.

Anonymous Report

Helpful Content

This page has a handful of links that will help you learn more about #ProjectNorthstar and connect with the community, we highly recommend checking out the discord and hanging out with some of the awesome people there! It's the quickest way to get your questions answered.

Join The Discord!

Joining the #ProjectNorthstar discord server is the best way to get help with any troubles you run into! It's also a fun and friendly community, come hang out!

Ahead.IO Kits!

You can order ready to build kits, or pre-built kits here!

Ali Heston's "The Design of Virtual and Augmented Reality"

Alexandria Heston put together an incredible collection of knowledge about design for VR and AR. You can check it out here! https://aliheston.gitbook.io/the-design-of-virtual-and-augmented-reality/

Official Leap Motion Links and Guides

GitHub Build Guide

GitHub Repository

Forums

Community Links

This medium article by @Tasuku is really good! Check it out for relevant links for Exii, 1-10.inc and other tweaks.

@mdrjjn put together this guide on how to build Exii Version 1, He made a video too!

Here's a link to Psychic Vr Lab's guide on going through the calibration process

@atlee19 put together this cool website with a bunch of Open Source Demos

@eswar made this awesome calibration walkthrough

@eswar also made this tutorial for 6dof tracking using a vive tracker!

Just joining? @callil made this awesome presentation for the New York meetup, it’s a great summary of what has happened so far!

The Northstar Next platform is designed to be modular, this means you can swap in or use any tracking camera/platform you'd like! Given how quickly the landscape around tracking changes, and how often new components/parts come out it's important to be able to upgrade electronics instead of having to build a whole new HMD, it's the reuse part of reduce/reuse/recycle!

Below are a few recommendations for various sensors, they all provide different features and have pros/cons so be sure to ask on discord if you have any specific questions!

Dedicated Hand Tracking Sensors

Generally when it comes to out of the box hand tracking, Ultraleap (formerly leap motion) are the best in the field. They have a variety of sensors you can choose. For Northstar headsets we generally recommend using the SIR170 as it's the lightest available sensor, and is designed for integrating into headsets. However you can also use the 1st or second gen leap motion cameras. The LeapMotion platform currently works on Windows and Linux, with MacOS support coming soon!

Position Tracking Sensors

Throught the duration of Project Northstar there have been many 6DOF sensors on the market. In this category, we'll show sensors that can be used "out of the box" with minimal configuration.

Xvisio XR50

The Xvisio XR50 sensor is one of the smallest 6DOF tracking sensors available. It uses a mix of onboard and on host compute to calculate pose, and has some other available features like Plane Detection.

Intel Realsense T261/T265

While these cameras are now in the end of life stage, they do still function and are generally one of the simplest tracking platforms to get started with. The t261 is just the embedded version of the T265, which means it's better suited for use in a weight-sensitive project like this one, but either sensor will function the same. The T26x sensors provide access to a stream of camera poses directly from the device, which means your computer doesn't have to process the poses.

SteamVR Lighthouse solutions

If you want more reliable 6DOF tracking at the cost of portability, you can choose to use a SteamVR Lighthouse based solution. There are a variety of trackers available for this system, the most common being from HTC Vive and Tundra Labs.

Lighthouse based solutions use two external laser sources to track the position of the tracker. The Lighthouse sensors themselves can be acquired via Valve or HTC.

General Sensors

The Luxonis cameras also sport an onboard movidus AI chip which means you can run other computer vision tasks as well, directly on the sensor itself.

There are many variants of Luxonis cameras to choose from, we generally recommend the Oak-D-W or Oak-D-Pro-W. It is important to get the -W variants, as these are the wide FOV versions.

Northstar Development Platforms

There are currently three methods of developing software on Northstar headsets.

1. This is the recommended developer experience. Esky is currently built on Unity and has built in support for the MRTK. It has video passthrough with the t261/t265, temporal reprojection, and supports both the 2D and 3D calibration methods.

2. This is the default unity experience, it's barebones and built for the 3D calibration rig. If you're experienced with unity and want to tinker with the original source code for Northstar this is the place for you.

3. The SteamVR integration allows any SteamVR game to run on a Northstar headset. Hand tracking isn't a replacement for controllers yet so you won't have a fun time in beat saber, but for demos like cat explorer or the infamous cubes demo you'll have full support for hand tracking.

4. Prebuilt Examples There are a handful of prebuilt demos for Northstar including LeapPaint, Galaxies and others. These will be linked on a separate page/database at a future date, for now, and check the #showcase channel.

Verify your headset is working properly.

You'll want to make sure that your headset works properly, plug in the power to the integrator board or driver board first, then the display port connection. On some headsets there can be an issue where plugging the display in first causes the driver board to get enough power through the display port connection to boot, but not enough power to run properly. If you run into this issue unplug your headset and then plug-in power first, then the display connection.

Set Windows Display Settings

Your Northstar display will show up as a normal monitor and will look "upside-down" if viewed through the headset. This is normal, we compensate for this in software written for the headset. Make sure your headset is set up so it's to the right of your main display(s), and that the resolution is 2880x1600. You'll also want to make sure that the scale and layout section is set to 100%. By default, the headset will run at 90hz. You'll also want to ensure that your headset is set to extended mode and not mirrored.

Install Drivers

Leap Motion

RealSense SDK

1. Push down on both circular buttons on the headset for four seconds to power cycle the integrator board. This should cause the RealSense device to enumerate and cause windows to detect it.

2. If step 1 did not work you can try unplugging the headset and plugging it back in, make sure your USB connection is plugged into a USB 3.1 port.

3. If both of the above steps did not work you can try resetting the USB hub in device manager. This solution has solved most edge cases we've seen so far.

North Star Development

There are currently three methods of getting software running on North Star headsets.

Project Esky is a complete unity framework that handles

• Rendering (with 2D and 3D (V1 and V2) s)

• MRTK Integration with the Leap Motion controller (aligned with the user's view)

• 6DoF Head Tracking + Re-Localization events (Save map, load map, add persistence, callbacks, etc.)

• Peer to Peer based co-localization and networking at the click of a button (Provided devices are on the same local network)

• Spatial mapping (Via the ZED SDK, with future plans to generate point clouds via the t261/t265 sensor)

• Temporal Reprojection (2D calibration only)

Written Guide (Config File Setup)

Note that the EskySettings.json file is located in the root of your unity project folder. (Not visible within Unity)

• For the V2 calibration, Copy the contents of your NorthStarCalibration.json file into Esky's EskySettings.json file, you will be replacing the following values within the v2CalibrationValues section:

  • left_uv_to_rect_x

  • left_uv_to_rect_y

  • right_uv_to_rect_x

  • right_uv_to_rect_y

• For the V1 calibration, Copy the contents of your NorthStarCalibration.json file into Esky's

  EskySettings.json file, you will be replacing all of the values within the v1CalibrationValues section

• Make sure Northstar Display is not flipped via windows display manager, additionally it can be helpful to have your Northstar Display setup to the far right of your monitor setup, though if necessary it can be placed anywhere, you'll just need to take note of the position for later.

• Then, within the EskySettings.Json file, edit the displayWindowSettings so that the DisplayXLoc and DisplayYLoc reflect the position of your northstar display relative to your main monitors.

  • In the below example, Monitor #2 is my NorthStar, and Monitor #3 is marked as my 'Main Display'. Since my 'Main Display' is 1920x1080 pixels in size, my DisplayXLoc and DisplayYLoc values will be 0, and 1080, respectively

• Save the EskySettings.json file. You're now free to proceed to open the Unity Project to complete the calibration

Hand Alignment and MRTK Setup

Before you can begin doing cool northstar stuff, you'll need to align your hands so that the virtual image matches the real world, and configure the MRTK to your specific North Star Setup.

For V1:

• Observe the following section of your V1 (3D) calibration json:
• You will need to copy the position, and rotation values into the following area within the EskySettings.json file. the position maps to TranslationEyeToLeapMotion, and the rotation maps to RotationEyeToLeapMotion.
• Save when complete, you can see an example of a completed EskySettings.json file in the Extras section

For V2:

• Hit play, then, click on the unity Game window (so that unity can receive the keyboard input)

• Alignment is pretty straightforward, following the instructions displayed in your headset:

  • 1) Hold your right hand up, so that the virtual hand appears in the center of the screen

  • 2) Hold the space bar

  • 3) While holding space, translate your right hand so that it aligns with the frozen virtual hand, NOTE: Try not to move your fingers as you do this

  • 4) Once aligned, release the space bar

  • 5) Repeat steps 1-4 3 or 4 times, the alignment system will let you know when it has collected enough samples.

  • If you are happy with the alignment, hit 's' to save, if not, stop unity (hit the play button again) then hit play to start the process again, repeating steps 1-5

  • Further adjustment can be done with the arrow keys

Setting up the MRTK Example

If you look at the Scene Heirarchy, you will notice the MixedRealityToolkit gameobject.

The inspector will show the Mixed Reality Toolkit configuration. Click Input -> Input Data Providers

You will see the following window for configuring all of esky's settings.

The settings are explained as follows:

Rig To Use: This controls which optical setup is used for your northstar (V1, V2), Project Ariel, or a custom rig can be selected. Filter System to Use: We work hard to develop Project Esky, and while we have a newer (and in our opinion better designed) pose filtering system, you can change this value to revert back to the old filtering pipeline here Reprojection Settings (V2/Ariel Only): This enables/disables the late-stage temporal reprojection built into Project Esky's native rendering pipeline.

Native Shader To Use (V2/Ariel Only, Currently not implemented): This changes the undistortion method used by the native rendering pipeline, we recommend not editing this.

Target Frame Rate (V2/Ariel Only): This changes the target frame rate for Unity. You can select 120, 90, 60, and 30 frames per second! NOTE: This frame rate is independent of the native rendering pipeline that handles composition, which always runs at 120 frames per second!

Leap Controller Orientation: This changes the way the MRTK handles the leapmotion controller, we recommend leaving this as 'Esky'

Enter/Exit pinch distance: This changes the distance between the index and thumb before the MRTK considers a pinch start/finish action (in meters).

Save after stopping editor: The default behaviour of Project Esky is to dump your current MRTK settings back into the EskySettings.json file, which is also copied to any build directory when you built your unity project. You can disable this behaviour but we don't recommend it!

Use Tracker Offsets (Not Implemented): This places the virtual camera relative to the 6DoF tracker, good for external between-sensor calibrations.

Uses Camera Preview: This changes whether you intend to use image passthrough preview with the t261, keep in mind you must have USB 3.0 connected to your NorthStar in order to use it!

Uses External RGB Camera: For those with an RGB sensor, you can enable this to use the RGB image passthrough, unless you know what you're doing Keep this disabled!!

All Done?

Then simply hit 'play' and you're good to go :D

Extras

Below is an Example of my complete EskySettings.Json File, with the updated V1 and V2 calibration for a 25cm focal length optical setup.

Note: Some of the values you see are either controlled by the MRTK in editor, or not yet in use.

This steamVR driver is still a work in progress, if you run into any issues, please reach out on Discord.

Working with some polish needed

- Head Tracking - Hand Tracking - View Projection - Skeletal tracking - Basic input - T265 Sensor integration

Notable unfinished parts

- Gesture recognizer

Prerequisites

Versions of vendor libraries not included, here is where to get them:

You will need to install the leap motion multi-device drivers in order for this driver to work. - LeapDeveloperKit 4.0.0+52238

If using the structure core you will need the CrossPlatform SDK 0.7.1 and the Perception Engine 0.7.1 https://structure.io/

If using the intel realsense t265, you should install the Intel® RealSense™ SDK 2.0

Using prebuilt binaries

We have a prebuilt version of the driver available here. You can place it in the following directory, or use vrpathreg as shown below.

C:\Program Files (x86)\Steam\steamapps\common\SteamVR\drivers

vrpathreg adddriver <full_path_to>/resources/northstar

Generating Project Files

In order to build from source you will need to install visual studio 2019 with c++, .net and C++ modules v142. In addition you'll also need to install git and cmake.

All commands below are run in windows command prompt

In the folder in which you want the repo to exist, run the following commands:

git clone https://github.com/fuag15/project_northstar_openvr_driver

cd project_northstar_openvr_driver

mkdir build

cd build

cmake -G "Visual Studio 16 2019" -A x64 ..

Building the driver in Visual Studio 2019

- Open the generated solution and set northstar to the startup project (right click the project and choose the set as startup where the gear icon is) and build.

Make sure to target x64 and a Release build to remove any object creation slowness.

- The release will be in `build/Release/`and will be comprised of dll files.

- Copy all the dll's to wherever you want to install from, they should be combined into the `resources/northstar/bin/win64` directory, make this if it does not exist and put all generated dll's inside.

Initializing the Driver in SteamVR

- Next register the driver with steamVR (using vrpathreg tool found in SteamVR bin folder). This is located at

C:\Program Files (x86)\Steam\steamapps\common\SteamVR\bin\win64\vrpathreg.exe

vrpathreg is a command line tool, you have to run it via the command prompt. To do this, follow these steps.

1) open command prompt

2) run cd C:\Program Files (x86)\Steam\steamapps\common\SteamVR\bin\win64\

3) run vrpathreg adddriver <full_path_to>/resources/northstar

4) you can verify the driver has been added by typing vrpathreg in command prompt, it will show you a list of drivers installed.

- at this point vrpathreg should show it registered under "external drivers", to disable it you can either disable in StamVR developer options under startup and manage addons, or by using vrpathreg removedriver <full_path_to>/resources/northstar

- Running steamvr with the northstar connected (and no other hmd's connected) should work at this point but probably not have the right configuration for your hmd. Take the .json file from calibrating your nothstar and convert it to the format found inresources/northstar/resources/settings/default.vrsettings

- restart steamvr. If driver is taking a long time to start, chances are it got built in debug mode, Release mode optimizes a way a bunch of intermediate object creation during the lens distortion solve that happens at startup. If things are still going south please issue a bug report here:

- if you wish to remove controller emulation, disable the leap driver in SteamVR developer settings.

What is Calibration?

Believe it or not, most hardware doesn't work perfectly the exact moment it's assembled. Assembling headsets by hand leads to small variances in each headset that need to be compensated for in software. In order to get the best experience with your ProjectNorthstar headset, you'll need to follow a few steps in order to compensate for these variances.

Optical Calibration

The first type of calibration you'll need to do is classified as Optical Calibration. This type of calibration uses stereo cameras to calculate the image distortion caused by the parabolic reflectors in the ProjectNorthstar headset. There are two versions of Optical Calibrations you can do based on what hardware you have access to. Both versions of optical calibration currently require a Calibration Stand to be 3D printed. The calibration methods distort the normal image displayed on the screens to make it appear correctly on the headset. Unity, unreal and other engines/compositors need to be able to take the information generated by the calibration method, and plug that into a rendering method in order to properly compute the image. Please see the table below for which methods are currently supported on specific platforms.

3D Optical Calibration

3D Optical Calibration calculates the exact position of your headset's screens and Optical Combiners, and uses raytracing (not the RTX kind, don't worry) to calculate the distortion for the headset. This allows it to be adjusted for multiple Interpupillary distances and eye reliefs. This method requires using two Stereo cameras and an external monitor (in addition to your main screen) to work. This makes it more expensive/complicated to do compared to 2D Optical Calibration. We currently recommend this method for maker spaces or groups that have to calibrate multiple headsets. This is the method used for preassembled and calibrated headsets from smart-prototyping and CombineReality.

2D Optical Calibration

Unlike the 3D Optical Calibration, 2D Optical Calibration only calculates the 2D distortion generated by the headset's Optical configuration. This method is much simpler to setup. It only requires one stereo camera, which you likely already have for your headset! You can use the intel t261, t265 currently, but if you have another stereo camera reach out on the discord and we will see what we can do to support it.

Calibrating Hand Position

Hand position is dependent on the position of the leap motion sensor, make sure your Leap motion sensor has the bottom metal bezel hidden behind the 3d printed housing. You can use this application to set up your hand position: https://github.com/fmaurer/NorthStarToolbox

3D and 2D Calibration Feature Comparison

Calibration Stand Assembly

Both the 2D and 3D versions of the calibration setup share the same 3D printable stand. You can find the assembly instructions below. (Note that they show the instructions for the original, 3D dual camera, stand.)

3D Optical Calibration

2D Optical Calibration

Displays and Display Driver Board

The display assembly for Northstar Next kits include:

The new display driver board supports two BOE 1440x1600 displays at ~90hz. The board has a USB-C input, and a 12pin serial output to connect to the USB hub board included in the next gen kits.

Connecting the ribbon cables to the driver board.

The Driver board and break out boards use special FPC connectors. These connectors have the pins wrap around the connector, meaning that either side can make contact with the ribbon cable pins.

The ribbon connections on the Driver board and Adapter boards use an FPC connection called a Backflip Latch. The gates have pins on both the top and bottom of the connector. This means you must connect the display ribbon cable as shown in the instructions, otherwise you will short your displays.

In order to connect the ribbon cables to the adapter boards correctly you must do so as pictured below:

Welcome to the Community Built Documentation for Project North Star!

We envision a future where the physical and virtual worlds blends together into a single magical experience. At the heart of this experience is hand tracking, which unlocks interactions uniquely suited to virtual and augmented reality. To explore the boundaries of interactive design in AR, we created and open sourced Project North Star, which drove us to push beyond the limitations of existing systems." -Leap Motion

Project North Star is an open-source Augmented Reality headset originally designed by LeapMotion (now UltraLeap) in June 2018. The project has had many variations since its inception, by both UltraLeap and the open-source community. Some of the variations are documented and linked to here but visit the discord server for more to-the-moment information. The headset is almost entirely 3D printable, with a handful of components like reflectors, circuit boards, cables, sensors, and screws that need to be sourced separately.

Project North Star at MIT Reality Hack 2020, photo credit: Matthew Daiter

There's also a large community of Northstar developers and builders on Discord, you can join the server and share your build, ask questions, or get help with your projects by joining the server

For a more detailed look into the project, checkout our General FAQ, or Mechanical pages.

Variations

Project North Star has seen its fair share of revisions and updates since the original open-source files were released. To clear up any ambiguity from the outset, Release 1 was an internal release. Release 2, the first public open-source release (sometimes referred to as the initial release), was in 2018. Release 3 came in 2019 and improved on the mechanical design in many ways. The Deck X innovated on this and provided an integrated circuit board to reduce cables from the headset to just two, (USB 3 / mini-DP), by combining USB devices into a custom-built USB hub + Arduino module. As shown from the table below, the newly released Northstar Next is probably what new users will want to start with, it has an emphasis on modularity and affordability.

The above video shows the general process of performing a 3D calibration on your project northstar headset.

1. Build the Calibration Rig

This requires:

• 3D Printing the mechanical assembly, search for it here: https://leapmotion.github.io/ProjectNorthStar/

• Affixing TWO of these Stereo Cameras to it: https://www.amazon.com/ELP- Industrial- Application- Synchronized- ELP- 960P2CAM- V90- VC/dp/B078TDLHCP/

• Acquiring a large secondary monitor to use as the calibration target

• Find the exact model and active area of the screen for this monitor; we'll need it in 3)

2. Calibrate the lenses on your stereo cameras:

• Print out an OpenCV calibration chessboard, and affixing it to a flat backing board.

  • Flatness is absolutely crucial for the calibration.

• Editing the config variables at the top of dualStereoChessboardCalibration.py with the correct:

  • Number of interior corners on each axis

  • Dimensions of each square on the checkerboard (in meters!)

• Install Python 3 on your machine and run pip install numpy and pip install opencv-contrib-python

  • Run it from the python scripts folder, usually something like C:\Users\*USERNAME*\AppData\Local\Programs\Python\Python36\Scripts

• Running dualStereoChessboardCalibration.py

  • First, ensure that your upper stereo camera appears on top in the camera visualizer

  • If not, exit the program, and unplug/replug your cameras' USB ports in various orders/ports until they do.

  • Hold your checkerboard in front of your camera array, ensuring to move it around to gain good coverage.

  • Every time the calibrator takes a snapshot, it will print a notice in the terminal.

  • After 30 snapshots in both camera views, it will run the calibration routines and display rectified views.

  • If the calibration went well, you will see your live camera stream rectified such that all straight lines in the real world will appear straight in the camera image, and the images will look straightened and vertically aligned between the screens.

    • If this happened, congratulations! Exit out of the program and run it one more time.

    • Running it again verifies the calibration can be loaded AND GENERATES THE CALIBRATION JSON

  • If the calibration did not go well (you see horrible warping and badness), you can attempt the calibration again by:

    • Deleting the created dualCameraCalibration.npz AND cameraCalibration.json files from the main folder

    • Trying again: ensuring the checkerboard is flat, the config parameters are correct, and that you have good coverage (including along depth)

3. Calibrate the Optical Assembly on your North Star

• You should have a good cameraCalibration.json file in the main folder (from the last step)

• Ensure that your main monitor is 1920x1080 is and the Calibration Monitor appears to the left of the main monitor, and the north star display appears to the right of it.

  • This ensures that the automatic layouting algorithm detects the various monitors appropriately.

• Edit config.json to have the active area for your calibration monitor you found earlier.

• Download this version of the Leap Service: https://github.com/leapmotion/UnityModules/tree/feat-multi-device/Multidevice%20Service

  • The calibrator was built with this version; it will complain if you don't have it :/

• Now Run the NorthStarCalibrator.exe

  • You should see the top camera's images in the top right, and the bottom camera's images on the bottom.

  • If this is not so, please reconnect your cameras until it is (same process as for the checkerboard script)

  • You should also see a set of sliders and buttons running along the top.

  • These control the calibration process.

• First, point the bare calibration rig toward the calibration monitor

  • Ensure it is roughly centered on the monitor, so it can see all of the vertical area.

• Then Press "1) Align Monitor Transform"

  • This will attempt to localize the monitor in space relative to the calibration rig. This is important for the later steps.

• Next, place the headset onto the calibration rig and press "2) Create Reflector Mask"

  • This should mask out all of the camera's FoV except the region where the screen and reflectors overlap the calibration monitor.

  • If it does not appear to do this, double check that all of the prior steps have been followed correctly...

• Now, before we press "3) Toggle Optimization", we'll want to adjust the bottom two sliders until the both represent roughly equal brightnesses.

  • This is important since the optimizer is trying to create a configuration that yields a perfectly gray viewing area.

• Now press "3) Toggle Optimization" and observe it.

  • It's switching between being valid for the upper and lower camera views, so only one image is going to appear to improve at a time.

  • You should it see it gradually discovering where the aligned camera location is.

  • This is the finickiest step in the process, it's possible that the headset is outside the standard build tolerances.

    • If you suspect this is the case, increase the simplexSize in the config.json to increase the area it will search.

• If it does converge on an aligned image, then congratulations! Toggle the optimization off again.

• Press button 4) to hide the pattern, put the headset on, and use the arrow keys to adjust the view dependent/ergonomic distortion and the numpad 2,4,6,8 keys to adjust the rotation of the leap peripheral.

• When satisfied, press 5) to Save the Calibration

  • This will save your calibration as a "Temp" calibration in the Calibrations Folder (a shortcut is available in the main folder).

  • You can differentiate between calibrations by the time in which they were created.

Bringing new worlds to life doesn’t end with bleeding-edge software – it’s also a battle with the laws of physics. Project North Star is a compelling glimpse into the future of AR interaction and an exciting engineering challenge, with wide-FOV displays and optics that demanded a whole new calibration and distortion system.

Just as a quick primer: the North Star headset has two screens on either side. These screens face towards the reflectors in front of the wearer. As their name suggests, the reflectors reflect the light coming from the screens, and into the wearer’s eyes.

As you can imagine, this requires a high degree of calibration and alignment, especially in AR. In VR, our brains often gloss over mismatches in time and space, because we have nothing to visually compare them to. In AR, we can see the virtual and real worlds simultaneously – an unforgiving standard that requires a high degree of accuracy.

North Star sets an even higher bar for accuracy and performance, since it must be maintained across a much wider field of view than any previous AR headset. To top it all off, North Star’s optics create a stereo-divergent off-axis distortion that can’t be modelled accurately with conventional radial polynomials.

North Star sets a high bar for accuracy and performance, since it must be maintained across a much wider field of view than any previous augmented reality headset. How can we achieve this high standard? Only with a distortion model that faithfully represents the physical geometry of the optical system. The best way to model any optical system is by raytracing – the process of tracing the path rays of light travel from the light source, through the optical system, to the eye. Raytracing makes it possible to simulate where a given ray of light entering the eye came from on the display, so we can precisely map the distortion between the eye and the screen.

But wait! This only works properly if we know the geometry of the optical system. This is hard with modern small-scale prototyping techniques, which achieve price effectiveness at the cost of poor mechanical tolerancing (relative to the requirements of near-eye optical systems). In developing North Star, we needed a way to measure these mechanical deviations to create a valid distortion mapping.

One of the best ways to understand an optical system is… looking through it!. By comparing what we see against some real-world reference, we can measure the aggregate deviation of the components in the system. A special class of algorithms called “numerical optimizers” lets us solve for the configuration of optical components that minimizes the distortion mismatch between the real-world reference and the virtual image.

Leap Motion North Star calibration combines a foundational principle of Newtonian optics with virtual jiggling. For convenience, we found it was possible to construct our calibration system entirely in the same base 3D environment that handles optical raytracing and 3D rendering. We begin by setting up one of our newer 64mm camera modules inside the headset and pointing it towards a large flat-screen LCD monitor. A pattern on the monitor lets us to triangulate its position and orientation relative to the headset rig.

With this, we can render an inverted virtual monitor on the headset in the same position as the real monitor in the world. If the two versions of the monitor matched up perfectly, they would additively cancel out to uniform white. (Thanks Newton!) The module can now measure this “deviation from perfect white” as the distortion error caused by the mechanical discrepancy between the physical optical system and the CAD model the raytracer is based on.

This “one-shot” photometric cost metric allows for a speedy enough evaluation to run a gradient-less simplex Nelder-Mead optimizer in-the-loop. (Basically, it jiggles the optical elements around until the deviation is below an acceptable level.) While this might sound inefficient, in practice it lets us converge on the correct configuration with a very high degree of precision.

This might be where the story ends – but there are two subtle ways that the optimizer can reach a wrong conclusion. The first kind of local minima rarely arises in practice. The more devious kind comes from the fact that there are multiple optical configurations that can yield the same geometric distortion when viewed from a single perspective. The equally devious solution is to film each eye’s optics from two cameras simultaneously. This lets us solve for a truly accurate optical system for each headset that can be raytraced from any perspective.

In static optical systems, it usually isn’t worth going through the trouble of determining per-headset optical models for distortion correction. However, near-eye displays are anything but static. Eye positions change for lots of reasons – different people’s interpupillary distances (IPDs), headset ergonomics, even the gradual shift of the headset on the head over a session. Any one of these factors alone can hamper the illusion of augmented reality.

Fortunately, by combining the raytracing model with eye tracking, we can compensate for these inconsistencies in real-time for free![6] We’ll cover the North Star eye tracking experiments in a future blog post.

How can I build it?

Good Question! There are multiple variations of the reference design from leap motion. The reference design can be found on this page

What are the Minimum System Requirements?

The minimum system requirements for Northstar can vary based on the type of application you want to run. Your system will need a display port connection that can output 4k@60hz.

How does project Northstar compare to other AR Platforms?

Project Northstar uses the same displays commonly used in VR headsets to provide a high resolution and high field of view experience.

Hardware Comparison

Software Comparison

Where can I get the lenses and electronic components?

While most of the design is 3d printable, there are components, like the screens, driver board, combiners, and leap motion controller that you will have to order.

For Northstar Next you can get them from the CombineReality shop.

What 3d printer should I get?

Most components fit within a print volume of 130mm*130mm*130mm, however the two largest prints will need a print volume of 220mm*200mm*120mm. It is possible for the parts to be split, using MeshMaker to allow them to fit on smaller print volumes. The ender series (220 x 220 x 250mm) by Creality seems to be a fan favorite among the discord if you're just getting started with 3d printing. If you want something that has a larger print area, check out the creality pro (300 x 300 x 400mm). If you want other recommendations, feel free to ask on the discord.

How can I track my position?

The Intel RealSense T265 is the most commonly used device currently. It supports 6dof (degrees of freedom) but does not support world meshing.

The occipital structure core is great since it's cross platform and non GPU dependent and has more features, but it's more expensive than the Realsense. (Note that if you order this you need the black and white camera version and NOT the color version). There are members of occipital here in the discord to answer more questions, check out the #occipital-structure-core channel. Occipital has discontinued support of the perception engine and is no longer recommended.

If you have a windows PC with a 1070 or above you can use the Zed Mini, but it only works with Nvidia CUDA which limits its use.

If you have a Vive already, you can use a vive tracker for 6dof tracking, however the vive tracker requires external "lighthouse" base stations in order to function, making it more difficult to transport the headset or use it in different environments without extra setup.

What is Calibration?

Due to the nature of 3D printing and assembly each headset is going to be slightly unique and will require going through a calibration process to display the image correctly. There are currently two ways to calibrate a northstar headset. The first method uses two stereo cameras to calculate the 3D position of the displays and reflectors. The second method uses a single stereo camera, and is currently setup to be able to use the intel t265 camera, which we currently recommend for 6DOF. This allows northstar developers to reuse the t265 rather than purchase two seperate stereo cameras.

What is the difference between the 25cm, 50cm, and 75cm version of the headset?

These numbers refer to the focal distance that the images appear from the user. The only difference between the two is the location of the screens relative to the combiner. The focal distances can be switched by replacing the display tray. Typically, we recommend starting with a 25cm build since they are easier to get started with. 25cm is sharper for items attached to your hand or right in front of your face.

- 25cm provides a much better experience when using all the virtual wearable interfaces from Leap Motion

- 25cm allows for a slightly wider FOV

- 25cm is also much brighter because of the angle of incidence and collimation layer in the display panel

- for wandering about and batting stuff around, throwing things, sticking stuff to your wall, or making Characters run around the room 75cm is way more convincing. With 25cm, the Vergence accommodation effect is noticeable, even if you get your IPD just right

- 75 cm is harder to calibrate than 25cm

In general, 75 cm is better if a lot of things you’re dealing with are further away, while 25 cm is best if you’re prototyping hand interactions. You can still tell how far away things are with either of the focal distances via stereo overall.

So to conclude, whichever decision you opt for will work depending on what you plan on doing. However, something to keep in mind is that if you wish to switch between the two display holders, you have to put the screens into the new trays and recalibrate the headset using the stand.

What are the cables connecting to the display driver board?

There are a total of four cables connecting to the board (see picture below): - two are the two ribbon cables that send data and power to the two displays. The connections are on either side of the board one for power one for the mini display port (for transferring data)

Troubleshooting Power Requirements

The expected behavior for the board having adequate power and working properly is having yellow led turned on. The LED's location has also been labeled in the image attached above. If there is also a red led lighting up upon plugging it in, this is reflective of insufficient power being fed into the board. If you have a voltmeter handy you might be interested in checking if after plugging the USB into a computer or a wall socket to usb converter that the voltmeter shows 5V of potential. This is the expected output for a USB connection. If the output voltage is correct or if you see no led light lighting whatsoever then there must be an issue with the device driver board. If this is the case reach out in the #noa-labs-display-adapter channel on the discord server for help in debugging. Secondly, if you do see the board led lighting up but don’t see anything on the screen you can do a couple of things to get things going:

- plugging the power USB in the same computer where you are plugging your display adapter makes things sync up (as this common source shares the same ground)

- disconnect everything and connect things in the following order: displays first (make sure you align the pins correctly and don’t force push the cables into the board as that might damage the pins), connect the power cable into your laptop and finally the display adapter. DO NOT under any circumstance unplug the display cables directly from the board while power is being connected. Also, always opt for operating with the cables that connect to the computer

Compatibility

These assets require the Multi-Device Beta Service to display hands.

These assets are dependent on Release 4.4.0 of the Leap Motion Unity Modules (included in the package).

Getting Started:

• Make sure your North Star AR Headset is plugged in

• In windows display settings make sure the headset is showing at the correct resolution (2880x1600) and is to the right of the main monitor.

• Create a new Project in Unity 2018.4 LTS

• Import "LeapAR.unitypackage" from the Github Repo

• Navigate to LeapMotion/North Star/Scenes/NorthStar.unity

• Click on the ARCameraRig game object and look for the WindowOffsetManager component

• Here, you can adjust the X and Y Shift that should be applied to the Unity Game View for it to appear on the North Star's display

• When you're satisfied with the placement; press "Move Game View To Headset"

• With the Game View on the Headset, you should be able to preview your experience in play mode!

  Key Code Shortcuts in NorthStar.unity (in the Editor with the Game View in focus and playing)

• C to Toggle Visibility of Calibration Bars

Calibrating your Headset

We have included a pre-built version of the internal calibration tool. We can make no guarantees about the accuracy of the process in DIY environments; this pipeline is built from multiple stages, each with multiple points of failure. Included in the .zip file are a python script for calibrating the calibration cameras, a checkerboard .pdf to be used with that, and Windows-based Calibrator exe, and a readme describing how to execute the entire process.

While this page is under construction, check out the working drawings here:

These are also available via PDF

Step 1- Set up the Optics Bracket

You will need the following

1. 3D Printed Optics Bracket

2. Heat Threaded Inserts (x8)

3. Soldering Iron

You may also find it helpful to acquire these special specifically for inserts.

Step 2- Power Filter Installation

You will need the following

1. Optics Bracket Assembly

2. Power Filter Board

3. M2.5x0.45 5mm Button Head Phillips Screw (1)

Step 3- Install Leap Motion Controller

You will need the following components for this step:

1. Leap Motion Controller

2. Micro-USB to 6-conductor ribbon adapter board

3. 6-conductor ribbon cable

Step 4- Display Driver Board Installation

1. Optics Bracket Assembly

2. Leap Motion Controller mounting spacer (3D printed Part)

3. Display Driver Board

4. Self Threading Screws (x3)

Step 5- Install Power Jumper

You will need:

1. Optics Bracket Assembly

2. JST Power Jumper

Step 6- Prepare Display Trays

1. Display Trays (25, or 75cm depending on which focal length you want. If you're unsure choose the 25cm to start with.)

2. Heat-set inserts (8)

Step 7- Peeling back the display ribbon cables

It will be very helpful to read through this step in full before proceeding. Pay Careful Attention and exercise patience. This is the hardest step in assembly of the headset. (Note that step 7-4 has multiple photos, you have to scroll down to see them)

1. Patience

2. BOE 3.5" 1440px1600p displays

Step 8- Install Displays

Step 9a- Installing the Display Assembly (Left Side)

Step 9a- Installing the Display Assembly (Right Side)

Step9b - Installing the Displays with the Ribbon Cable Extensions

We recently introduced a new product to improve the installation and safety of the display process. The ribbon cable extensions remove the need to peel the display cable and provide a cleaner overall installation experience. Please see the following video for instructions:

Step 10- Preparing the T261 mounting bracket

Newer headset kits come with a piece that does not use the heat set inserts, you can skip this step if your part t261 mount looks different from the one shown below.

Step 11- Installing heatsinks on the t261

Step 12- Installing the Fan

Step 13- Installing the t261 sensor

Step 14- Installing the Integrator Board

Step 15- Integrator to t261 Connection

Step 16- Attaching the Integrator to the Optics Bracket

It's the home stretch now! Let's attach the Integrator to the Optics Bracket!

Step 17- Connecting Power

Step 18- Building the Lid

Step 19- Attaching the Lid to the Optics Bracket

Step 20- Adding Padding

Step 21- Installing the Combiners

Step 22-Final Adjustments

Mechanical Parts (Excluding 3D printed parts

3D Printed Parts- Main Assembly

3D Printed Parts- Headgear

3D Printed Parts- Deck X Addition

Electronics and Optics

Deck X Electronics

Modules/Sensors

Intel t261 Accessories

The Combine Reality Deck X is a variant of Release 3 designed by Noah Zerkin's team at smart-prototyping. It includes a new hub called "The Integrator" which includes microSD card storage, an Arduino and USB hub, an embedded Intel Realsense t261 sensor, and a control board for adjusting ergonomics like IPD and eye relief.

3D Printing Variants

Integrator Board

The Integrator is our custom-built USB hub system originally created for the CombineReality Project North Star Deck X. The Integrator cuts down the use of cables and adds customizable buttons to the headset with the following components & features:

• USB-C hub, two USB 3.1 ribbon connectors, and one USB 2.0 ribbon connector 3GB on-board flash drive (only works when connected to a USB 3.0 host) Arduino-compatible microcontroller, featuring a Qwiic connector that can be used to connect additional sensors like an IMU, as well as HID buttons that can emulate keyboard keys.

• A button breakout board is included, and the microcontroller is preflashed with firmware that maps the buttons to the default ergonomics adjustment keys. (Eye relief, eye position, and IPD) Also allows for manual power reset of sensor USB ports via a GPIO pin.

• A fan, the speed of which is controlled by the Arduino-compatible microcontroller. A thermistor for a more intelligent fan speed control.

• A ribbon connector that lets the Arduino on the hub relay commands and debug output to and from the serial UART of the display driver board Ribbon adapter board for Intel® RealSense™ T261 embedded 6-DOF module (ribbon cable included) Ribbon adapter board for Leap Motion Controller (ribbon cable included)

Installing Third Party Dependencies

Intel RealSense

Download librealsense v2.53.1 from Intel's github repo.

Debian / Ubuntu Install

Installing Monado Dependencies

Testing the build

To test that Monado was built properly run the following tests with the example configuration files: