Merging the Real and the Virtual Worlds

Martin Labrecque, Dec 5th, 2015

Among the numerous people who have tried our demos, a few people asked us if we could actually merge some elements of the real world in a virtual reality simulation. “Of course!”, we replied, “because our tracking system works in the real world coordinate system”. Obviously, because VR is something that we experience, it's easier shown than explained, so here we go.


We made a simple Unity scene with 3 cubes that can be moved with the click of each of the mouse buttons. We adjusted the game's camera position to approximately reflect the distance between my eyes and my hands when my forearm is horizontal and folded close to my chest. We made a script such that we can apply some adjustments with the keyboard, but the default worked for most of us in the office. Finally, we made sure that the scene was scaled such that my hand displacement in the game matched approximately my hand movement in the real world. No measurements were taken, this was purely qualitative. For the real world content, we positioned 3 boxes on a table. I'm wearing an Oculus DK1 headset so I cannot see anything outside of the headset.


First, I find the box corners by feel and use the mouse (in my left hand) to snap the boxes to my index finger tip position. I do this once only, and don't touch the mouse afterwards. Then I reach for the box corners in the virtual world, guided only by the VR display, and hope that I'm touching the box corners in the real world.


As you can see, the tracking is reliable and allows us to overlay a virtual world on top of a real world setup. It could have been a bit easier if I had a clear line of sight between my eyes and my fingertips on the second and third boxes in the virtual world (the finger tips are partially occluded by my hand), but there was really no difficulty in reaching the targets. Thinking of industrial simulations with physical props, or even a flight simulation inside a physical cockpit, it is important that the virtual and real worlds align.

Real world targets and virtual worlds objects align!

Hopefully you now understand the value of having an accurate tracking system that operates in real worlds coordinates and doesn't drift. Mission accomplished!

Powered by CouchCMS