What is Mixed Reality?
The term “mixed reality” typically describes a system that blends the physical world with virtual environments and digital information. As the name implies, a mixed reality system allows you to experience a combination of the real world and digital effects. Broadly speaking, the goal of a mixed reality system is to connect physical and virtual objects.
With mixed reality technology, these connections are intended to help users share and enhance their experience within the system. The experience ranges from simple concepts, like wearable biometrics and games, to profoundly immersive test systems, like the original prototypes designed in the 1990s.
There’s also a growing middle range full of home and business applications. As the technology evolves, the term will continue to encompass this variety of applications, but they’ll all be created with the goal of overlapping physical reality and designed environments.
By using a variety of tracking and sensing technologies, the physical and virtual worlds are coordinated and overlaid with one another to create what we see as mixed reality. As a result, your location in a physical space can correspond with the digital.
This creates a basic framework for new interactions based on the overlap, in which your location or task can potentially be connected to a whole range of digital interactions.
This is the basic premise of augmented reality - expanding the real world by linking it with more complex, adaptive, and intelligent digital overlays. As much as the technologies involved have changed over time, mixed reality has only risen to prominence quite recently.
Origins of the term and the first mixed reality systems
While the term “mixed reality” has become more common these days, the first actual mixed reality setup was designed by scientist Louis Rosenberg in 1992 [1]. He crafted an immersive remote operator system at the now-defunct United States Air Force’s Armstrong Labs in Texas.
Rosenberg integrated the three most critical human senses (sight, sound, and touch) with a basic exoskeleton, two remote-controlled robots, and a powerful headset system that used magnification to match the proportions of the robots to the human user.
The equipment was a successful proof of concept that introduced engineers and designers to an incredible new idea.
With this system, Rosenberg was interested in how safety could be improved for people working hazardous or difficult jobs that still require human supervision. Heavier workloads or specialized tasks could be accomplished by a robotic body with the human mind as the pilot.
It’s a fitting advancement when you consider that many early applications centered on training practices for complex tasks.
Virtual fixtures and the virtuality continuum
While Rosenberg used the term “virtual fixtures” to describe his system, researchers Paul Milgram and Fumio Kishino coined the term "mixed reality" in 1994 [2]. They were writing about a “virtuality continuum” to help contextualize different sorts of augmented experiences [3].
Rosenberg’s designs anticipated a variety of uses for the system, while Milgram and Kishino intended to clarify the field’s working terminology for other scientists and engineers.
Putting it all together
After developing his system, Rosenberg continued his work with a string of other developments and companies related to virtual reality, rendering, and artificial intelligence. He was mostly interested in advancing the more balanced synthesis needed for truly immersive systems.
Rosenberg’s companies experimented with haptic technology for wearable devices and a whole range of 3D digitization techniques used in films. His Immersion Corporation famously sued Apple in 2016 for claimed infringements against his widespread haptic technology [4]. And in 2017, Immersion announced its expansion to gaming by implementing its tech with the widely used Unity gaming engine.
Rosenberg’s involvement in the tech landscape is longstanding, and his work has found multiple applications over the years.
How does mixed reality connect to VR or augmented reality?
Augmented reality is essentially a form of mixed reality
Despite technological advances and new products, there is still frequent confusion surrounding this terminology because of the scope of each expression and the relationships they describe.
Mixed reality
Milgram and Kishino used the term “mixed reality” to describe the full range of augmented experiences that a person may have.
While some of their other ideas aren’t as commonly discussed, their main vision helps distinguish between different applications. Primarily, they understood that how you experience mixed reality depends on the technology you have available.
Mediated reality
There is also some overlap with the concept of a mediated reality, an older expression that describes an entire spectrum of interactions and applications done via computer.
The distinction here is control, because mediated reality describes the entire range of interactions. A mediated reality is anything that relies on an external system to augment reality or combine both the real world and virtual environments. Most of the confusion has to do with how the terms have been used during periods of major development.
Virtual or augmented reality
As the technology diversifies in how it’s used, something that began as a specialist technology begins looking more like the next stage in human-computer interactions. As a result, it needs to be called something other than “virtual reality.”
For now, “mixed reality” is generally used consistently with Milgram and Kishino’s original definition to explain an entire range of experiences, and often includes virtual reality, augmented reality, and the less frequently used “augmented virtuality.”
So how does the technology work?
As we’ve shown, the definition of “mixed reality” covers a variety of physical-virtual experiences. Similarly, the associated technology is just as wide-ranging and includes the following:
- The haptic technology associated with movement tracking
- The visual tech that helps render virtual and augmented realities
But how does it work exactly?
Computer mediation
In addition to the technology itself, mixed reality depends on a device to mediate or connect the virtual and the real.
Typically, a computer is used to receive and interpret input from a human user’s physical reality and their virtual interactions. The same device also typically generates the virtual environment, so these experiences depend on processing power.
Some of the most noticeable developments focus on improving information from users. Typically, feedback comes from simple tools, such as a wired glove or location information. Augmented reality systems can be as straightforward as a smartphone app like Pokémon Go, but more immersion requires more elaborate equipment.
Unfortunately, the most complex systems aren’t exactly accessible to the general public right now, but new mixed reality products are certainly beginning to compete for attention.
Motion-tracking technology
To facilitate more complex interactions, like those performed by Rosenberg, developers often incorporate other elements into mixed reality setups. These often include motion-tracking tech found in haptic gloves, more streamlined exoskeletons, and controllers.
Headsets
While it’s true that early mixed reality headsets were less dynamic and clunkier than the latest technology, they all have the same goal: accurately syncing key factors, like the direction of your eyes, with a virtual environment.
The headset is crucial because it’s the primary source of input for the user. It also marks a boundary between less complex, projector-based systems, versus the true immersion that’s possible with a semi-enclosed headset.
Optics and imaging
Mixed reality also depends heavily on graphics technology. And as this technology improves, we’ll see bigger, better, and more nuanced experiences for users.
There are many visual technologies and techniques that help push the field for mixed reality forward and sometimes they’re inspired by real-world uses. Consider the story of Steve Mann, one of the inventors of high-dynamic-range (HDR) imaging. He explained his inspiration for creating HDR in an interview [5]:
My grandfather taught me how to weld as a child. When welding, you see only a little speck of light, and everything else is almost completely black. I thought there must be a better way of doing this, so I invented HDR to help people see.
In Mann’s example, advanced imaging facilitates augmented reality functions that could help exaggerate the “little speck of light” his grandfather had to follow into something easier to work with.
Gaming potential and office applications
The potential for mixed reality gaming and office devices are massive and becoming limited only by the imagination or desired use.
In the past few years, there has been an explosion of new apps and games designed with mixed reality in mind. In PC gaming, the idea is simple: take the physically passive activity and rework it for mixed reality. This has led to adaptations of exploration games, ambient experiences, and a variety of fun environments designed for freedom play.
Some focus on solving puzzles like The Talos Principle VR, while others provide humorous takes on traditional activities like Job Simulator. Many others center on different ways of getting immersed in physics, such as the simplistic wall-climbing game Climbey.
Beyond gaming, developers are also looking into using augmented reality to facilitate office work and collaboration. For example, Microsoft® has designed integrated systems that support new routines and equipment. With their powerful new headset-based Microsoft HoloLens, the focus is on information, design, and facilitating collaboration.
For designers and engineers especially, it’s easy to see how a mixed reality system could be helpful to bring projects to life or provide extra layers of information as you engage with virtually any complex task.
Other general applications for mixed reality technology
There are other uses for mixed reality technology that go beyond gaming, design, and science. While we’ve already discussed the idea of remotely operated equipment or exploration, the full range of possibilities stretches from the mundane to the incredible.
As a tool for training and education, mixed reality has far-reaching applications, such as providing opportunities to enrich basic learning. Imagine students being able to experience interactive models of ancient cities, all from their classroom. And when you couple this with data-driven applications, the potential for education is almost unparalleled.
The integrated Windows Mixed Reality system
Microsoft’s Windows Mixed Reality platform is among the systems available with benefits for both entertainment and business. It starts with a compatible Windows 10 laptop or desktop PC as the base, and then users wear a headset and controllers to play and interact with the growing number of VR games and apps.
The latter can include everything from simply talking with your friends to building your own virtual home. As with Microsoft’s HoloLens system, the emphasis is on improving access to digital content and improving your real-world experience.
VR-enabled products and the HP Mixed Reality Headset
With ever more games and apps being designed for VR, existing software is constantly being ported to the different VR systems.
At HP®, we have a number of different ways for you to experience this brand-new tech, including a wearable VR backpack, VR workstation PCs, and the HP Windows Mixed Reality Headset.
If you’re looking to embrace the future, you have multiple options to turn that thought into a reality today.
[2] IEICE Transactions on Information Systems, Taxonomy of Mixed Reality Visual Displays
[4] Recode.net, Immersion Sues Apple for Patent Infringement Over Touch Feedback
[5] IEEE, Q&A with Steve Mann
