Augmented reality will give us superpowers

Over the next decade, the handheld mobile phone will be replaced by augmented-reality glasses that you will wear during most of your waking hours. 

The first wave of immersive eyewear products will reach consumer markets in the next few years. The biggest companies in the world will introduce the technology — companies from Apple, Google, and Meta to Microsoft, Samsung, LG, and Snap. And while many consumers, myself included, are skeptical that we’ll ever want to wear digital hardware on our faces for hours each day, we will.  

The reason is simple: Augmented reality will give us superpowers.  

We cannot afford to overlook the impact this technology will have on our lives in the near future. One need only consider the billions being spent on the so-called metaverse. Still, I have been involved in this field from the very beginning, and I appreciate the skepticism that surrounds virtual and augmented reality. We lived through a false start in the 1990s, and another in the mid-2010s. 

But this time is different. 

VR and AR technologies are finally able to offer real-world value. While I can point to many examples, no field of endeavor makes the case more clearly than medicine. Last week I participated as a panelist at the Digital Orthopedics Conference in San Francisco, where a major theme was to imagine the medical field in the 2030s. As part of this effort, a small group of us carefully reviewed the latest research and assessed the potential impact of immersive technologies. 

The first superhumans

I have to admit, I was deeply impressed by how far augmented reality has progressed over the last 18 months for use in medicine. I expect that by 2030, augmented reality headsets will be a common tool for surgeons, radiologists, and many other medical professionals. 

This early adoption of augmented reality will make doctors the first humans with superpowers. I’m talking about superhuman abilities to visualize medical images, patient data, and other vital clinical content. The costs associated with these new capabilities are already quite reasonable, and they will fall rapidly as AR hardware is produced in much higher volumes over the coming years. 

The first superpower is X-ray vision — the ability to peer into a patient and see evidence of trauma or disease at the exact location in their body where it resides. Of course, doctors can already look under the skin using tools such as computerized tomography and magnetic resonance imaging. But these images display on flat screens, leaving medical professionals to imagine how the image relates to the patient on the table. This type of extrapolation is an impressive skill, but it takes time and mental effort, which inspires many doctors to wish they could simply gaze directly into the human body.  

With AR headsets and new techniques for medical imaging, the superpower of x-ray vision is now a reality. An impressive study from Teikyo University’s School of Medicine tested an experimental emergency room with the ability to capture whole-body CT scans of trauma patients. The medical team wore AR headsets, allowing personnel to peer into the patient on the operating table and see the trauma. This allowed the team to discuss the injuries and plan the treatment without referring to a flat screen. It saved time, reduced distractions, and cut down on the need to extrapolate from limited information.  

X-ray vision is just the start 

Augmented reality will provide doctors with assistive content overlaid onto and into the patient’s body and visible at the location where the information is most needed. For example, surgeons performing a delicate procedure will see navigational cues projected onto the patient in real time. This guidance will help them intervene with added precision. The goal is to increase accuracy, reduce mental effort, and speed the procedure. The potential value for surgery is extreme, from minimally invasive procedures such as laparoscopy and endoscopy, to freehand surgical efforts such as placing orthopedic implants.   

In fact, I predict augmented reality will become required equipment for many surgical procedures within the next ten years.  I say that as someone who has been involved with the concept of augmenting surgery for decades, as the idea goes back to the first AR system — the Virtual Fixtures platform — developed at the Air Force Research Laboratory in the early 1990s. The goal of that early effort was to show that AR could boost human dexterity in precision tasks such as surgery. 

The project was a success. But to appreciate the remarkable progress the field has made in the decades since, consider this: When testing whether virtual overlays could enhance manual precision, the early system required users to move metal pegs between holes spaced 2 feet apart. Thirty years later, surgeons at Johns Hopkins, Thomas Jefferson University Hospitals, and Washington University performed a delicate spinal procedure on 28 patients. They used augmented reality to help place metal screws to a precision of less than 2 mm. As published in a recent study, the system registered the real patient with such accuracy that surgeons scored 98% on standard performance metrics. 

Looking forward, we can expect augmented reality to impact all aspects of medicine. That is because its precision has reached clinically viable levels. In addition, major breakthroughs are in the works that will make it faster and easier to use augmented reality in medical settings. As described above, the biggest challenge for any precision AR application is to accurately align virtual content with the real patient (a process called registration). In medicine, this currently means attaching physical markers to the patient, which takes time and effort. In a recent study from Imperial College London and the University of Pisa, researchers tested a marker-less AR system for surgeons that uses cameras and artificial intelligence to accurately align the virtual content. Their method was faster and cheaper, but not quite as accurate. This is still early days — in the coming years, this technology will make AR-supported surgery viable without the need for costly markers.

Camera-based registration techniques will soon take augmented reality out of highly controlled environments like operating rooms and bring them to a wider range of medical applications. I predict that by 2030, general practitioners will commonly see patients with the benefit of AR headsets, and this brings me to another medical superpower — the ability to peer back in time. Doctors will use AR headsets to capture spatially registered images of their patients, and they will view prior images perfectly aligned with the patients’ bodies. For example, a physician could quickly assess the healing progress of a skin lesion by examining the patient through AR glasses, while scrolling back in time to compare the current lesion with what it looked like during prior visits. 

Researchers on the medical uses of virtual and augmented reality have made impressive progress, with significant implications for medical education and medical practice. To quote Stefano Bini of the UCSF Department of Orthopaedic Surgery, “The beneficial role of AR and VR in the upskilling of the healthcare workforce cannot be underestimated.” 

Augmented reality will make you omniscient

I agree with Bini and would go even further. I believe augmented reality in the 2030s will impact all aspects of society. After all, the superpowers of X-ray vision, navigational cues, dexterity support, and the ability to peer back in time will be useful for everything from construction and auto repair to engineering, manufacturing, and agriculture. But the biggest impact will be on mainstream consumers, who will have a wide range of additional superpowers for daily use.

One such superpower is omniscience. With AR eyewear you will turn your gaze on almost anything in your surroundings, squint thoughtfully, and immediately view informational content about the object or activity of focus. A quick head nod to confirm you have seen it, and the information will fade away. This will feel like a genuine superpower. 

Augmented reality will also give you superhuman recall. It will be impossible to forget someone’s name, as this will appear the instant you see them, along with the names of their spouse and kids. You’ll even be reminded of the last time you met and what you discussed. All this information will be seamlessly provided.

Augmented Reality will also integrate with smart homes, allowing users to glance at a light switch across the room and give a quick, Harry Potter-like gesture to turn it on. This will feel like a true magical power, and it will extend far beyond lights to all electronic devices. With glances and gestures we will learn to control all aspects of our augmented world. And when it comes to music, the eyewear will allow you to do more than just listen — it will have the ability to alter the look of your surroundings in sync with the song, making it feel like you are suddenly transported into your own real-time music video as you walk down the street. 

There are some concerns

This brings me back to my thesis: Over the next ten years, augmented reality will replace the mobile phone as our primary interface for digital content. Early adopters will embrace the lure of new, magical capabilities. Everyone else, skeptics included, will quickly find themselves at a disadvantage without omniscience, x-ray vision, superhuman recall, and dozens of other capabilities that are not even on the drawing board yet. This will drive adoption as quickly as the transition from flip phones to smartphones. After all, not upgrading your hardware will mean missing out on layers of useful information that everyone else can see.

An augmented world is coming — one with the potential to be magical, embellished with artistic content and infused with superhuman abilities. At the same time, there are risks we must avoid, as augmented reality will give tech platforms unprecedented ability to track our activities and mediate our experiences. For these reasons, we need to push for a safe and regulated metaverse, especially the augmented metaverse. It will impact all of our lives in the very near future.