Uses of a dedicated 3D reconstruction software with augmented and mixed reality in planning and performing advanced liver surgery and living donor liver transplantation(with videos)

The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery simulation and using the created scenarios in real-time surgery using mixed reality.In this article,we described our experience on developing a dedicated 3 dimensional visualization and reconstruction software for surgeons to be used in advanced liver surgery and living donor liver transplantation.Furthermore,we shared the recent developments in the field by explaining the outreach of the software from virtual reality to augmented reality and mixed reality.

Chinese expert consensus on laparoscopic hepatic segmentectomy and subsegmentectomy navigated by augmented-and mixed-reality technology combined with indocyanine green fluorescence imaging

Augmented-and mixed-reality technologies have pioneered the realization of real-time fusion and interactive projection for laparoscopic surgeries.Indocyanine green fluorescence imaging technology has enabled anatomical,functional,and radical hepatectomy through tumor identification and localization of target hepatic segments,driving a transformative shift in themanagement of hepatic surgical diseases,moving away from traditional,empirical diagnostic and treatment approaches toward digital,intelligent ones.The Hepatic Surgery Group of the Surgery Branch of the Chinese Medical Association,Digital Medicine Branch of the Chinese Medical Association,Digital Intelligent Surgery Committee of the Chinese Society of ResearchHospitals,and Liver Cancer Committee of the Chinese Medical Doctor Association organized the relevant experts in China to formulate this consensus.This consensus provides a comprehensive outline of the principles,advantages,processes,and key considerations associated with the application of augmented reality and mixed-reality technology combined with indocyanine green fluorescence imaging technology for hepatic segmental and subsegmental resection.The purpose is to streamline and standardize the application of these technologies.

Clinical use of augmented reality,mixed reality,three-dimensionalnavigation and artificial intelligence in liver surgery

A precise knowledge of intra-parenchymal vascular and biliary architecture and the location of lesions in relation to the complex anatomy is indispensable to perform liver surgery.Therefore,virtual three-dimensional(3D)-reconstruction models from computed tomography/magnetic resonance imaging scans of the liver might be helpful for visualization.Augmented reality,mixed reality and 3Dnavigation could transfer such 3D-image data directly into the operation theater to support the surgeon.This review examines the literature about the clinical and intraoperative use of these image guidance techniques in liver surgery and provides the reader with the opportunity to learn about these techniques.Augmented reality and mixed reality have been shown to be feasible for the use in open and minimally invasive liver surgery.3D-navigation facilitated targeting of intraparenchymal lesions.The existing data is limited to small cohorts and description about technical details e.g.,accordance between the virtual 3D-model and the real liver anatomy.Randomized controlled trials regarding clinical data or oncological outcome are not available.Up to now there is no intraoperative application of artificial intelligence in liver surgery.The usability of all these sophisticated image guidance tools has still not reached the grade of immersion which would be necessary for a widespread use in the daily surgical routine.Although there are many challenges,augmented reality,mixed reality,3Dnavigation and artificial intelligence are emerging fields in hepato-biliary surgery.

Leveraging Augmented Reality,Semantic-Segmentation,and VANETs for Enhanced Driver’s Safety Assistance

Overtaking is a crucial maneuver in road transportation that requires a clear view of the road ahead.However,limited visibility of ahead vehicles can often make it challenging for drivers to assess the safety of overtaking maneuvers,leading to accidents and fatalities.In this paper,we consider atrous convolution,a powerful tool for explicitly adjusting the field-of-view of a filter as well as controlling the resolution of feature responses generated by Deep Convolutional Neural Networks in the context of semantic image segmentation.This article explores the potential of seeing-through vehicles as a solution to enhance overtaking safety.See-through vehicles leverage advanced technologies such as cameras,sensors,and displays to provide drivers with a real-time view of the vehicle ahead,including the areas hidden from their direct line of sight.To address the problems of safe passing and occlusion by huge vehicles,we designed a see-through vehicle system in this study,we employed a windshield display in the back car together with cameras in both cars.The server within the back car was used to segment the car,and the segmented portion of the car displayed the video from the front car.Our see-through system improves the driver’s field of vision and helps him change lanes,cross a large car that is blocking their view,and safely overtake other vehicles.Our network was trained and tested on the Cityscape dataset using semantic segmentation.This transparent technique will instruct the driver on the concealed traffic situation that the front vehicle has obscured.For our findings,we have achieved 97.1% F1-score.The article also discusses the challenges and opportunities of implementing see-through vehicles in real-world scenarios,including technical,regulatory,and user acceptance factors.

Investigation of Inside-Out Tracking Methods for Six Degrees of Freedom Pose Estimation of a Smartphone in Augmented Reality

Six degrees of freedom(6DoF)input interfaces are essential formanipulating virtual objects through translation or rotation in three-dimensional(3D)space.A traditional outside-in tracking controller requires the installation of expensive hardware in advance.While inside-out tracking controllers have been proposed,they often suffer from limitations such as interaction limited to the tracking range of the sensor(e.g.,a sensor on the head-mounted display(HMD))or the need for pose value modification to function as an input interface(e.g.,a sensor on the controller).This study investigates 6DoF pose estimation methods without restricting the tracking range,using a smartphone as a controller in augmented reality(AR)environments.Our approach involves proposing methods for estimating the initial pose of the controller and correcting the pose using an inside-out tracking approach.In addition,seven pose estimation algorithms were presented as candidates depending on the tracking range of the device sensor,the tracking method(e.g.,marker recognition,visual-inertial odometry(VIO)),and whether modification of the initial pose is necessary.Through two experiments(discrete and continuous data),the performance of the algorithms was evaluated.The results demonstrate enhanced final pose accuracy achieved by correcting the initial pose.Furthermore,the importance of selecting the tracking algorithm based on the tracking range of the devices and the actual input value of the 3D interaction was emphasized.

Single-center experience with Knee+^(TM) augmented reality navigation system in primary total knee arthroplasty

BACKGROUND Computer-assisted systems obtained an increased interest in orthopaedic surgery over the last years,as they enhance precision compared to conventional hardware.The expansion of computer assistance is evolving with the employment of augmented reality.Yet,the accuracy of augmented reality navigation systems has not been determined.AIM To examine the accuracy of component alignment and restoration of the affected limb’s mechanical axis in primary total knee arthroplasty(TKA),utilizing an augmented reality navigation system and to assess whether such systems are conspicuously fruitful for an accomplished knee surgeon.METHODS From May 2021 to December 2021,30 patients,25 women and five men,under-went a primary unilateral TKA.Revision cases were excluded.A preoperative radiographic procedure was performed to evaluate the limb’s axial alignment.All patients were operated on by the same team,without a tourniquet,utilizing three distinct prostheses with the assistance of the Knee+™augmented reality navigation system in every operation.Postoperatively,the same radiographic exam protocol was executed to evaluate the implants’position,orientation and coronal plane alignment.We recorded measurements in 3 stages regarding femoral varus and flexion,tibial varus and posterior slope.Firstly,the expected values from the Augmented Reality system were documented.Then we calculated the same values after each cut and finally,the same measurements were recorded radiolo-gically after the operations.Concerning statistical analysis,Lin’s concordance correlation coefficient was estimated,while Wilcoxon Signed Rank Test was performed when needed.RESULTS A statistically significant difference was observed regarding mean expected values and radiographic mea-surements for femoral flexion measurements only(Z score=2.67,P value=0.01).Nonetheless,this difference was statistically significantly lower than 1 degree(Z score=-4.21,P value<0.01).In terms of discrepancies in the calculations of expected values and controlled measurements,a statistically significant difference between tibial varus values was detected(Z score=-2.33,P value=0.02),which was also statistically significantly lower than 1 degree(Z score=-4.99,P value<0.01).CONCLUSION The results indicate satisfactory postoperative coronal alignment without outliers across all three different implants utilized.Augmented reality navigation systems can bolster orthopaedic surgeons’accuracy in achieving precise axial alignment.However,further research is required to further evaluate their efficacy and potential.

Large-scale spatial data visualization method based on augmented reality

Background A task assigned to space exploration satellites involves detecting the physical environment within a certain space.However,space detection data are complex and abstract.These data are not conducive for researchers'visual perceptions of the evolution and interaction of events in the space environment.Methods A time-series dynamic data sampling method for large-scale space was proposed for sample detection data in space and time,and the corresponding relationships between data location features and other attribute features were established.A tone-mapping method based on statistical histogram equalization was proposed and applied to the final attribute feature data.The visualization process is optimized for rendering by merging materials,reducing the number of patches,and performing other operations.Results The results of sampling,feature extraction,and uniform visualization of the detection data of complex types,long duration spans,and uneven spatial distributions were obtained.The real-time visualization of large-scale spatial structures using augmented reality devices,particularly low-performance devices,was also investigated.Conclusions The proposed visualization system can reconstruct the three-dimensional structure of a large-scale space,express the structure and changes in the spatial environment using augmented reality,and assist in intuitively discovering spatial environmental events and evolutionary rules.

Exploration of the Role of Virtual Reality and Augmented Reality in Revolutionizing Art Education

Virtual reality(VR)and augmented reality(AR)technologies have become increasingly important instruments in the field of art education as information technology develops quickly,transforming the conventional art education approach.The present situation,benefits,difficulties,and potential development tendencies of VR and AR technologies in art education will be investigated in this study.By means of literature analysis and case studies,this paper presents the fundamental ideas of VR and AR technologies together with their several uses in art education,namely virtual museums,interactive art production,art history instruction,and distant art cooperation.The research examines how these technologies might improve students’immersion,raise their learning motivation,and encourage innovative ideas and multidisciplinary cooperation.Practical application concerns including technology costs,content production obstacles,user acceptance,privacy,and ethical questions also come under discussion.At last,the article offers ideas and suggestions to help VR and AR technologies be effectively integrated into art education through teacher training,curriculum design,technology infrastructure development,and multidisciplinary cooperation.This study offers useful advice for teachers of art as well as important references for legislators and technology developers working together to further the creative growth of art education.

Augmented Reality Marketing:Breakthroughs and Opportunities

The impact of augmented reality(AR)technology on consumer behavior has increasingly attracted academic attention.While early research has provided valuable insights,many challenges remain.This article reviews recent studies,analyzing AR’s technical features,marketing concepts,and action mechanisms from a consumer perspective.By refining existing frameworks and introducing a new model based on situation awareness theory,the paper offers a deeper exploration of AR marketing.Finally,it proposes directions for future research in this emerging field.

Application of Mixed Reality Technology in Visualization of Medical Operations

With the continuous progress of virtual simulation technology,medical surgery visualization system has been developed from two-dimensional to three-dimensional,from digital to network and intelligentization.The visualization system with mixed reality technology will also be used in all stage of medical surgery,such as case discussion,surgical planning,intraoperative guidance,post-operative evaluation,rehabilitation,so as to further promote high intelligence,high precision of medical surgery,and consequently improve effectiveness of treatment and quality of medical service.This paper discusses the composition and technical characteristics of medical operation visualization system based on mixed reality technology,and introduces some typical applications of mixed reality technology in medical operation visualization,which provides a new perspective for the application of mixed technology in medical surgery.

Application and Prospect of Mixed Reality Technology in Medical Field

Mixed reality(MR)technology is a new digital holographic image technology,which appears in the field of graphics after virtual reality(VR)and augmented reality(AR)technology,a new interdisciplinary frontier.As a new generation of technology,MR has attracted great attention of clinicians in recent years.The emergence of MR will bring about revolutionary changes in medical education training,medical research,medical communication,and clinical treatment.At present,MR technology has become the popular frontline information technology for medical applications.With the popularization of digital technology in the medical field,the development prospects of MR are inestimable.The purpose of this review article is to introduce the application of MR technology in the medical field and prospect its trend in the future.

Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods

Background: Augmented reality(AR) technology is used to reconstruct three-dimensional(3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.Data Sources: The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the Pub Med database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles.Results: In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery,which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology.Conclusions: With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling,and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.

Mixed Reality-based Interactive Technology for Aircraft Cabin Assembly

Due to the narrowness of space and the complexity of structure, the assembly of aircraft cabin has become one of the major bottlenecks in the whole manufacturing process. To solve the problem, at the beginning of aircraft design, the different stages of the lifecycle of aircraft must be thought about, which include the trial manufacture, assembly, maintenance, recycling and destruction of the product. Recently, thanks to the development of the virtual reality and augmented reality, some low-cost and fast solutions are found for the product assembly. This paper presents a mixed reality-based interactive technology for the aircraft cabin assembly, which can enhance the efficiency of the assemblage in a virtual environment in terms of vision, information and operation. In the mixed reality-based assembly environment, the physical scene can be obtained by a camera and then generated by a computer. The virtual parts, the features of visual assembly, the navigation information, the physical parts and the physical assembly environment will be mixed and presented in the same assembly scene. The mixed or the augmented information will provide some assembling information as a detailed assembly instruction in the mixed reality-based assembly environment. Constraint proxy and its match rules help to reconstruct and visualize the restriction relationship among different parts, and to avoid the complex calculation of constraint＇s match. Finally, a desktop prototype system of virtual assembly has been built to assist the assembly verification and training with the virtual hand.

Mobile Web Augmented Reality in 5G and Beyond:Challenges, Opportunities, and Future Directions

The popularity of wearable devices and smartphones has fueled the development of Mobile Augmented Reality(MAR),which provides immersive experiences over the real world using techniques,such as computer vision and deep learning.However,the hardware-specific MAR is costly and heavy,and the App-based MAR requires an additional download and installation and it also lacks cross-platform ability.These limitations hamper the pervasive promotion of MAR.This paper argues that mobile Web AR(MWAR)holds the potential to become a practical and pervasive solution that can effectively scale to millions of end-users because MWAR can be developed as a lightweight,cross-platform,and low-cost solution for end-to-end delivery of MAR.The main challenges for making MWAR a reality lie in the low efficiency for dense computing in Web browsers,a large delay for real-time interactions over mobile networks,and the lack of standardization.The good news is that the newly emerging 5G and Beyond 5G(B5G)cellular networks can mitigate these issues to some extent via techniques such as network slicing,device-to-device communication,and mobile edge computing.In this paper,we first give an overview of the challenges and opportunities of MWAR in the 5G era.Then we describe our design and development of a generic service-oriented framework(called MWAR5)to provide a scalable,flexible,and easy to deploy MWAR solution.We evaluate the performance of our MWAR5 system in an actually deployed 5G trial network under the collaborative configurations,which shows encouraging results.Moreover,we also share the experiences and insights from our development and deployment,including some exciting future directions of MWAR over 5G and B5G networks.

Real-time in situ three-dimensional integral videography and surgical navigation using augmented reality: a pilot study

To evaluate the feasibility and accuracy of a three-dimensional augmented reality system incorporating integral videography for imaging oral and maxillofacial regions, based on preoperative computed tomography data. Three-dimensional surface models of the jawbones, based on the computed tomography data, were used to create the integral videography images of a subject＇s maxillofacial area. The three-dimensional augmented reality system （integral videography display, computed tomography, a position tracker and a computer） was used to generate a three-dimensional overlay that was projected on the surgical site via a half-silvered mirror. Thereafter, a feasibility study was performed on a volunteer. The accuracy of this system was verified on a solid model while simulating bone resection. Positional registration was attained by identifying and tracking the patient/surgical instrument＇s position. Thus, integral videography images of jawbones, teeth and the surgical tool were superimposed in the correct position. Stereoscopic images viewed from various angles were accurately displayed. Change in the viewing angle did not negatively affect the surgeon＇s ability to simultaneously observe the three-dimensional images and the patient, without special glasses. The difference in three-dimensional position of each measuring point on the solid model and augmented reality navigation was almost negligible （〈1 mm）; this indicates that the system was highly accurate. This augmented reality system was highly accurate and effective for surgical navigation and for overlaying a three-dimensional computed tomography image on a patient＇s surgical area, enabling the surgeon to understand the positional relationship between the preoperative image and the actual surgical site, with the naked eye.

An overview on the principle of inkjet printing technique and its application in micro-display for augmented/virtual realities

Augmented reality(AR)and virtual reality(VR)are two novel display technologies that are under updates.The essential feature of AR/VR is the full-color display that requires high pixel densities.To generate three-color pixels,the fluorescent color conversion layer inevitably includes green and red pixels.To fabricate such sort of display kits,inkjet printing is a promising way to position the color conversion layers.In this review article,the progress of AR/VR technologies is first reviewed,and in succession,the state of the art of inkjet printing,as well as two key issues-the optimization of ink and the reduction of coffee-ring effects,are introduced.Finally,some potential problems associated with the color converting layer are highlighted.

Mixed reality for visualization of orthopedic surgical anatomy

In the modern era,preoperative planning is substantially facilitated by artificial reality technologies,which permit a better understanding of patient anatomy,thus increasing the safety and accuracy of surgical interventions.In the field of orthopedic surgery,the increase in safety and accuracy improves treatment quality and orthopedic patient outcomes.Artificial reality technologies,which include virtual reality(VR),augmented reality(AR),and mixed reality(MR),use digital images obtained from computed tomography or magnetic resonance imaging.VR replaces the user’s physical environment with one that is computer generated.AR and MR have been defined as technologies that permit the fusing of the physical with the virtual environment,enabling the user to interact with both physical and virtual objects.MR has been defined as a technology that,in contrast to AR,enables users to visualize the depth and perspective of the virtual models.We aimed to shed light on the role that MR can play in the visualization of orthopedic surgical anatomy.The literature suggests that MR could be a valuable tool in orthopedic surgeon’s hands for visualization of the anatomy.However,we remark that confusion exists in the literature concerning the characteristics of MR.Thus,a more clear description of MR is needed in orthopedic research,so that the potential of this technology can be more deeply understood.

Rectification of magnetic force tracker using neural network inaugmented reality system

Nonlinear errors always exist in data obtained from tracker in augmented reality （AR）, which badly influence the effect of AR. This paper proposes to rectify the errors using BP neural network. As BP neural network is prone to getting into local extrema and convergence is slow, genetic algorithm is employed to optimize the initial weights and threshold of neural network. This paper discusses how to set the crucial parameters in the algorithm. Experimental results show that the method ensures that the neural network achieves global convergence quickly and correctly. Tracking precision of AR system is improved after the tracker is rectified, and the third dimension of AR system is enhanced.

Registration Based on ORB and FREAK Features for Augmented Reality Systems

This paper proposes a novel registration method for augmented reality (AR) systems based on Oriented FAST and Rotated BRIEF (ORB) and Fast Retina Keypoint (FREAK) natural features. In the proposed ORB-FREAK method, feature extraction is implemented based on the combination of ORB and FREAK, and the feature points are matched using Hamming distance. To get good matching points, cross-checks and least median squares are used to perform outlier filtration, and camera pose is estimated using the matched points. Finally, AR is rendered. Experiments show that the proposed method improves the speed of registration to be in real time; the proposed method can accurately register the target object under the circumstances of partial occlusion of the object; and it also can overcome the effects of rotation, scale change, ambient light and distance. © 2017, Tianjin University and Springer-Verlag Berlin Heidelberg.

Rise of Augmented Reality: Current and Future Application Areas

The massive technological advancements around the world have created significant challenging competition among companies where each of the companies tries to attract the customers using different techniques. One of the recent tech- niques is Augmented Reality (AR). The AR is a new technology which is capable of presenting possibilities that are difficult for other technologies to offer and meet. Nowadays, numerous augmented reality applications have been used in the industry of different kinds and disseminated all over the world. AR will really alter the way individuals view the world. The AR is yet in its initial phases of research and development at different colleges and high-tech institutes. Throughout the last years, AR apps became transportable and generally available on various devices. Besides, AR be- gins to occupy its place in our audio-visual media and to be used in various fields in our life in tangible and exciting ways such as news, sports and is used in many domains in our life such as electronic commerce, promotion, design, and business. In addition, AR is used to facilitate the learning whereas it enables students to access location-specific infor- mation provided through various sources. Such growth and spread of AR applications pushes organizations to compete one another, every one of them exerts its best to gain the customers. This paper provides a comprehensive study of AR including its history, architecture, applications, current challenges and future trends.