What is AR? An intro guide to Spatial computing

 
 Image Credit MICROSOFT 2018

Image Credit MICROSOFT 2018

 
 

AR stands for Augmented Reality.  Augmented Reality is an interactive experience of a real-world environment "augmented" by computer generated images or holograms. The primary value of AR, is its ability to enrich the real world with context specific information such as directions or instructions.

While Augmented Reality can be experienced on a screen, such as a phone or tablet with AR capabilities, only AR headsets can provide stereo 3D imagery with correct environmental lighting and accurate mapping. There are a number of key components that make this happen, which I will discuss below. I should also note that heads-up displays like Google Glass, are not considered to be Augmented Reality headsets even though they do provide a degree of augmentation, as these devices do not take into account the environment the user is in, and cannot display 3D information in the real world.

 
 
 AR Headsets : Left:  Microsoft Hololens ; Right:  Magic Leap

AR Headsets : Left: Microsoft Hololens; Right: Magic Leap

 
 

So how does it work?

1. SLAM

SLAM (Simultaneous Localization and Mapping) is a technique used to map a surrounding environment using a variety of sensors and cameras while simultaneously keeping track of the agent's position (the sensors doing are mapping). AR makes full use of this to place holograms in your environment, as well as know where exactly you are placed in the world and where you are looking at.

 
 

On mobile phones or tablets, SLAM is done using only the built in camera and computer vision algorithms like ARCore and ARKit. Headsets like the HoloLones have built in depth sensors as well as cameras, allowing for a much higher degree of accuracy.

 

2. Optics

Unlike VR headsets, which produce visuals via a standard display situated right in front of your eyes, AR headsets like HoloLens and MagicLeap are pass through devices. That means that, you see the real world through the device’s clear lenses, and the images (holograms) are projected onto a separate set of layered lenses, producing the illusion that the images exist within the real world.

 
 Hololens close-up optics. Image credits  TIME

Hololens close-up optics. Image credits TIME

 

On mobile, the holograms are overlaid onto the device camera feed, with the most well known example Pokemon Go.

 

3. Interaction

Interaction can vary from device to device. HoloLens uses hand gesture and gaze input as the primary mode of interaction. This mode can become strenuous, as hand tracking is far from perfect and having to turn your head around to click on something isn't something you'll want to do for long periods.

MagicLeap opted for a to a hand controller with 3DOF (degrees of freedom) which works similarly in a sense to a computer mouse. This is also the current go-to standard for Mobile VR devices. While input methods still miss on some key features, like haptics and eye tracking (so you can just look at something instead of having to turn your head), we can soon expect major improvements as adoption increases.

 
 Hand Controller for AR interation. Image credit MagicLeap 2018

Hand Controller for AR interation. Image credit MagicLeap 2018

 
Emanuel Tomozei