Future Surgery: How Doctors in Kharkiv Are Using Augmented Reality for Operations
Industry giants like Meta, Apple, and Microsoft are investing billions of dollars into the development of this technology. The Apple AR glasses, created by the iPhone manufacturer, have become the most expensive mass-market gadget in the company's history.
However, augmented reality technology can be utilized beyond the entertainment sector. It has the potential to transform several scientific fields. For instance, in medicine, AR provides doctors with additional visual data during surgeries, including three-dimensional images of patients' organs.
During the procedure, the surgeon can see not only the patient but also a detailed 3D model of their brain overlaid on the actual body. This is not science fiction, but a reality created by a team from the Ukrainian healthtech startup eXtra Vision.
The startup was founded by three doctors: Vladislav Kalyuzhka (neurosurgeon, CEO), Mykyta Markevych (cardiologist, CTO), and Mark Glukhensky (neurosurgeon, COO). The technology is being used by dozens of doctors, who have performed over 150 surgeries with its help. In 2024, the startup won the IT Arena and plans to enter the EU market.
How did doctors from Kharkiv manage to turn the idea of virtual surgery into a successful startup?
Idea, dozens of surgeries, and a ready startup
"Every surgery is a challenge that requires maximum precision and speed, with a focus solely on the pathology to avoid damaging healthy tissues. In neurosurgery, the risks are extremely high for both the doctor and the patient. To minimize them, the idea arose to create a three-dimensional model of the patient's brain considering the pathology," recalls Kalyuzhka about the startup's inception.
However, it took several years from the idea of creating three-dimensional models to the emergence of the first MVP (minimum viable product). Initially, there was a lengthy search for developers: not everyone understood how to technically implement the doctors' vision. Additionally, challenges arose with integrating the obtained 3D images into AR glasses.
Colleagues came to the rescue. "The first prototype of our technology was created by my colleague, a cardiologist who had a technical education before attending medical university," Kalyuzhka explains.
Seeing the future: how digital twins are changing the world around us
The first 50 successful surgeries using AR glasses were performed at the Kharkiv Regional Clinical Hospital. These surgeries were conducted on the brain and spinal cord. Doctors tested their product on patients with various pathologies, ranging from tumors and vascular issues to injuries and congenital defects.
The effectiveness of the solution convinced them to develop the project as a startup with the prospect of attracting investment to create their own business. Eventually, the eXtra Vision team expanded by adding IT specialists, including Unity developers, artificial intelligence (machine learning) engineers, and designers.
How eXtra Vision works
To create a 3D model of the patient's brain, data is required. Computer tomography (CT) or magnetic resonance imaging (MRI) of the patient is ideal for this purpose. The obtained digital two-dimensional images are uploaded into an application, where a volumetric digital copy of the human head is created using AI.
In the first stage of creating the 3D model, anatomical structures are highlighted: skin, bones, vessels, and nerves. In the second stage, pathological structures, such as tumors or hematomas, are identified. Currently, the process of creating 3D models is fully automated.
The completed model is uploaded to a cloud environment, which synchronizes with the augmented reality glasses. Typically, the time from creating a three-dimensional model to its integration with AR glasses ranges from several minutes to half an hour, depending on the data volume.
Using AR glasses, the doctor sees the completed model in real space. The glasses themselves are equipped with several cameras and sensors that scan the room where the doctor is located, creating a coordinate system. Additionally, the sensors are used to track the surgeon's tools and hands.
A breakthrough in medicine or a new bubble: what's wrong with Elon Musk's Neuralink startup
"The doctor sees both virtual and real objects. The technology allows the 3D model to be overlaid on the patient, enabling the surgeon to view the skull as if in cross-section. This helps accurately determine the incision site, the volume of bone tissue that needs to be removed for access to the pathology, and to avoid important vessels," explains the CEO of eXtra Vision.
The startup primarily uses HoloLens glasses from Microsoft. Moreover, the developers are working on importing models to Apple Vision Pro, MetaQuest 3, and MetaQuest Pro from Oculus and Meta.
The startup focuses on the work of neurosurgeons: performing surgeries on the brain and spine. In the future, the team plans to develop this technology in orthopedics and traumatology, hepatobiliary surgery (diagnosis and treatment of liver, gallbladder, and bile duct diseases), and maxillofacial surgery.
How eXtra Vision eases the workload of doctors
3D models assist not only during surgeries but also in the planning phase. The doctor can examine the brain model, rotate it, and thoughtfully devise an intervention plan. Previously, two-dimensional CT or MRI images were used for this purpose, which required extensive practice and skills to analyze.
Three-dimensional models are equally important for patients and their relatives. With the help of the 3D model, the surgeon can explain how the intervention will proceed. "Brain surgeries are a significant stress for patients, but when they understand what they're going into and see that the doctor clearly knows how everything will go, it reassures them," notes Kalyuzhka.
The technology can also be utilized during surgeries. Before making an incision, the model is projected onto the area of the body where the intervention will occur. This allows the doctor to visualize the tumor and other critical elements. The structure of the three-dimensional model depends on the specific surgery and the tasks that need to be accomplished.
According to the CEO of eXtra Vision, the use of their technology is most critical at the beginning of the surgery—before the incision, which needs to be carefully planned. The actual procedure is better conducted without AR glasses; however, they can be used at any moment to assess the situation.
Additionally, the team is working on creating an AI-powered assistant for the doctor. Currently, the neural network is more involved in model creation, but in the future, the team plans to engage AI in data analysis as well. The AI assistant will be able to identify the type of tumor and, based on the 3D model, recommend the best surgical path to avoid damaging critical functional areas of the brain.
Not just Pokémon. How augmented reality is changing art
Developers plan to add another solution: model matching and tool tracking.
"When a surgeon picks up a tool equipped with special markers, the system automatically creates its virtual copy. This allows the doctor to see the exact location of the tool in the surgical field. If the surgeon inserts the tool 3 centimeters into the tissues, they will see this on the screen with millimeter accuracy. This enables surgeries to be performed with maximum precision, which is crucial in neurosurgery," Kalyuzhka explains.
Business model and clients
The startup has 37 users in Ukraine. Among them are doctors from Kharkiv, Lviv, and Kyiv, including those from the Romodanov Institute of Neurosurgery. Using eXtra Vision technologies, they have already performed over 150 surgeries.