How augmented reality-assisted surgery is revolutionizing the medical industry

Recently, the augmented reality (AR) knee arthroplasty navigation system Knee+ from French company Pixee Medical received CE certification, which is the first orthopaedic navigation system approved to use augmented reality (AR) in total knee arthroplasty. Similar applications of augmented reality technology in surgical navigation are booming around the world. Augmented reality-assisted surgical technology is of great significance for preoperative planning, intraoperative guidance and postoperative rehabilitation. This article will explain the basic concepts, technical principles, market conditions, major companies and representative products, and describe the future development is discussed.

1. Basic Concepts

1. Augmented Reality (AR)

Augmented reality (AR) is a technology that superimposes computer-processed images of virtual models into real scenes to enhance real scenes. It enhances people’s perception of the real world around them, and integrates simulated scenes, objects, and related prompt information (such as sound, video, graphics or GPS data) into the real scene to enhance the effect. The fundamental discipline of augmented reality (AR) technology is computer vision. It uses tools such as displays, cameras and sensors to overlay digital information onto the real world.

  How augmented reality-assisted surgery is revolutionizing the medical industry

2. Surgical navigation system

Surgical navigation is a technology that accurately corresponds the image data of the patient before or during the operation with the patient’s anatomical structure, tracks the instruments during the operation, and updates and displays their position ON the patient’s image in real time in the form of a virtual probe. Surgical navigation enables doctors to see the location of instruments at a glance, making surgery faster, more precise, and safer. After years of development, surgical navigation has become the standard for neurosurgical treatment and is gradually gaining popularity in other fields. Currently in the US surgical navigation market, 578,375 procedures are performed each year, which is expected to grow to 718,224 by 2025. In the United States, surgical navigation is most commonly used in neurosurgery, accounting for approximately 43.3% of the total number of cases; in Europe, the most common is total knee arthroplasty (TKA).

2. Technical principle

Augmented reality surgical navigation system includes three cores: virtual image or environment modeling, virtual environment and real space registration, and Display technology that combines virtual and real environments.

1. Virtual image or environment modeling

AR systems use color or texture distinctions between anatomical structures in CT or MRI tomography and angiography to accomplish 3D reconstruction of subsurface objects in a computer. Non-realistic rendering or inverted reality techniques can improve visualization and depth perception.

2. Registration of virtual environment and real space

Registration can be accomplished by a variety of means, and a three-dimensional Cartesian system based on frame technology can determine the position and attitude of the imaging device.

3. display technology that combines virtual and real environments

display technology can be broadly classified into head-mounted displays (HMDs), enhanced external displays, enhanced optical systems, enhanced window displays, and image projection. Using HMD, the virtual environment can be covered not only in the real world under the user’s field of vision (optical perspective), but also in the video source of the real environment (video perspective). Augmented displays are simple stand-alone screens that display virtual content over real-world video. Optically enhanced displays refer to the direct enhancement of the eyepieces of a surgical microscope or binocular. Window-enhanced display is the placement of a translucent Screen directly over the surgical site, allowing virtual objects to be displayed directly on the screen over real objects. The virtual environment can be projected directly onto the patient with a projector.

3. Market situation

Augmented reality (AR) applications in the medical market are growing strongly with an expected CAGR of 33.36%, and the market value is expected to grow from $627 million in 2018 to $3.497 billion in 2024. AR technology is gaining a lot of attention from physicians due to its wide range of applications, from assessing surgical preparation to minimally invasive surgery and rehabilitation. Furthermore, according to ResearchandMarkets, the global surgical imaging market size is expected to reach USD 1.7 billion by 2025, growing at a CAGR of 5.4% during the forecast period. The latest role and integration of augmented reality (AR) in healthcare is enhancing the surgical experience. The rapid development of real-time visualization platforms has also led to better surgical treatments. Furthermore, increasing government funding, growing prevalence of sports injuries, and expanding geriatric population are factors contributing to the growth of the global surgical imaging market. Based on application, the market is segmented into neurosurgery, orthopedic and trauma surgery, cardiac and vascular surgery, general surgery, and other surgeries.

4. Main companies and representative products

1. Knee+ system of Pixee Medical

Founded in France in October 2017, PixeeMedical aims to leverage existing advanced computer vision and artificial intelligence technologies to create high-performance computer-assisted surgical solutions while keeping prices affordable in challenging health system environments. Knee+ just recently received CE certification for augmented reality knee arthroplasty navigation. This is the first orthopaedic navigation system to use augmented reality (AR) in total knee arthroplasty. FDA 510(k) approval for the system is currently being sought.

Knee+ patented technology combines proprietary computer vision and deep learning algorithms to work with off-the-shelf AR glasses to track instruments and implants during surgery. Navigation software installed in the smart glasses is combined with a reduced-scale MIS instrument with markers that can be sterilized in an autoclave. Size and ligament balance features can be quickly integrated into the product. As an alternative to bulky and expensive robotic systems, the Knee+ is simple to use, cost-effective, and does not require preoperative DICOM or disposable equipment. Future applications may be introduced into shoulder and hip surgery.

2. XVisionSpine system from Augmedics

Augmedics is a company dedicated to the development of surgical treatment technology, established in 2014. The first product, the xvision-spine (XVS) system, is an augmented reality surgical navigation system. With the XVS system, surgeons can see the real-time position and trajectory of surgical tools under the skin and tissue and navigate inside the patient, making surgery easier, faster and safer. The XVS uses patented see-through optics to project 3D images, as well as axial and sagittal planes, onto the surgeon’s retina in real time with surgical precision and excellent depth perception.

The XVS system includes a transparent near-eye display headset and has all the elements of a traditional navigation system. It accurately determines the location of surgical tools in real-time and superimposes it on the patient’s computed tomography data. The navigation data is then projected onto the surgeon’s retina using a transparent near-eye display headset, allowing the surgeon to simultaneously gaze at the patient and view the navigation data without having to move the eye to a remote screen. In a study conducted at Rush University Medical Center in Chicago, XVision was found to increase surgeon accuracy to an impressive 98.9 percent. The accuracy of the entire system is about 1.4 mm, which meets the US FDA’s requirement of less than 2 mm.

3. VOSTARS Video Optical Perspective Augmented Reality Surgical System

VOSTARS is an innovative action project funded by the Horizon 2020 program. Augmented reality (AR) surgical goggles developed by European scientists are able to see X-ray images and all key data perfectly superimposed in 3D with anatomical structures at once, and move freely at the same time. The VOSTARS Video Optical Fluoroscopy Augmented Reality Surgical System will conveniently display the patient’s anesthesia data, heart rate, body temperature, blood pressure and respiratory rate, etc. within the surgeon’s field of vision. Surgical accuracy will be significantly improved while reducing surgical time (20 minutes for every three-hour procedure), time under anesthesia and the costs involved in any surgery. The project plans to achieve mass production in 2022.

The system combines two existing AR technologies: video see-through (VST) and optical see-through (OST). Neither VST nor OST alone are suitable for surgery on live patients. OST systems such as Microsoft Hololens use translucent mirror surfaces to provide users with a direct view of the natural environment by superimposing computer-generated images on the user’s field of vision. VST systems such as the OcculusRift submerge the user in a virtual world with an enclosed head-mounted display (HMD) and stereoscopic cameras and screens.

4. TrueVision? 3DSurgical system

TrueVision™ 3DSurgical is the global leader in digital 3D visualization and the benchmark for microsurgery. Founded in 2003 and headquartered in Santa Barbara, California, TrueVision® helps doctors perform microsurgery through a self-developed digital 3D visualization platform. The intelligent, real-time, 3D visualization surgery and computer-aided guidance platform developed by the company has been patented.

The system enables surgeons to record surgical procedures in 3D and stream them live, making it an entirely new teaching tool. The company has developed a 3D guidance application for microsurgery that can improve surgical efficiency and patient outcomes. The system can be used in microsurgery, ophthalmology, and neurology, and can be integrated with a variety of application platforms and, in some cases, robotic surgery. The system is used by hundreds of mainstream hospitals and institutions around the world.

5. Discussion on the future development of augmented reality (AR) surgical navigation system

The application of augmented reality in clinical surgery spans many disciplines such as computer science, computer vision, sensors, communications, clinical medicine, and ergonomics, and the technology covers a wide range of areas. It can be seen from the following roadmap of augmented reality (AR)-assisted surgery: future tracking technology and image processing will be more inclined to intelligent technologies represented by deep learning, displays and sensors will be more inclined to technologies that are highly related to human organs, Such as retina display and human-computer symbiosis technology.


At the same time, there are challenges and obstacles in the development of augmented reality-assisted surgery. For example, for some display technologies, there are challenges in displaying 3D virtual objects as real-world images; time synchronization between virtual and real environments is another challenge for all AR systems. , especially rapid fluoroscopy changes; in the process of surgery, in some cases, image composite needs to be performed by a professional computer team; the compatibility and interoperability between the augmented reality surgical navigation system and related equipment and solutions need to be optimized improvement; data privacy issues, etc.


Medical surgery navigation is one of the important applications of augmented reality technology. In the era of image-guided surgery, augmented reality (AR) technology represents the next frontier in incorporating guidance systems into surgical workflows. With the rapid development of display technology and interactive technology, the role of augmented reality (AR) in the modern surgical operating room will increase.