Add Mixed Reality to Your VR Headset with ZED Mini

Image credit: Stereolabs

If Microsoft’s mixed reality push is making you feel bad about your plain-Jane HTC Vive or Oculus Rift, you will soon be able to use a device called the ZED Mini to give your headset the power of mixed reality. A company called Stereolabs is making this device which can easily be attached to the backside of the Oculus Rift or the HTC Vive.

Mixed Reality?

If you don’t know what mixed reality is, you are not alone. This new technology is just starting to hit the market. Microsoft’s super expensive HoloLens headset is the first and only currently available headset that offers a complete mixed reality experience. Explaining how the HoloLens works is a great way to describe how mixed reality functions and can be used.

https://www.youtube.com/watch?v=NwY-6sQDYnk

Virtual Reality Compared to Mixed Reality

Virtual reality headsets and their corresponding games and simulations are designed to bring you to other places or other realities. Mixed reality headsets, like the HoloLens, are designed to keep you in this reality while bringing in virtual elements to your surrounding environment. Described differently, mixed reality is like a souped up version of augmented reality.

The HoloLens mixes virtual elements with real-world elements by projecting virtual objects on a clear visor element that allows users to still see their surrounding environments. Using depth sensing sensors that map the geometry of the room, virtual objects are projected onto the clear visor in ways that make said virtual objects appear as though they are interacting with the user’s surroundings.

Image credit: Microsoft

For example, in a few tech demos, small virtual characters can be made to appear to run around the user’s environment (see above). This works because virtual characters are actually interacting with spatial data gathered from the HoloLens depth sensors. Depth data is used to create a simple 3d mesh of the environment which the headset keeps aligned with the real world.

This mesh is kept invisible to the user, however, virtual objects and characters interact with the invisible aligned mesh. Because the mesh is invisible, it appears that virtual objects are interacting with real-world objects. This creates the illusion that a virtual object is mixed into our reality.

How is Mixed Reality Possible with a Vive or Rift?

Since the HTC Vive and Oculus Rift do not have transparent visors, reproducing this process seems impossible. How could mixed reality functionality be possible with the Oculus Rift or the HTC Vive? To solve this problem, Stereolabs is attaching video sensors to the back of conventional virtual reality headsets. The video sensor effectively makes the headset work as a transparent video feed visor. This add-on will retail for $349. This is a fairly significant chunk of change but it could totally be worth the price.

https://youtu.be/7mxoOR2nQi0

Together, the sensors provide a 110-degree mixed reality field of view. This is much more than the limited field of mixed reality view that the HoloLens provides. That said, instead of looking through a transparent visor, users will have to be looking through a somewhat limited 720p video feed that runs at 60fps.

Image credit: Stereolabs

The Zed Mini also senses 3d space. The way in which it achieves this is quite different than Microsoft’s HoloLens which uses specialized depth sensors. Using the two cameras together, the Zed Mini calculates 3d space in a way that sounds similar to Google’s new WorldSense tracking system. Describing how this works, Stereolabs CEO Cecile Schmollgruber says, “It takes the point of view of the left and right cameras and understands the distances of objects.”

Beyond the hardware, Stereolabs is also developing software. They will be releasing a ZED SDK because developers will need to use specialized software to utilize the hardware. Using the SDK, developers can place virtual objects, people, and places in a user’s environment with relative ease. This opens the door for the Oculus Rift and the HTC Vive to provide users with a huge variety of mixed reality games and simulations. The SDK will drop alongside the release of the ZED Mini in September.