Application of Mixed Reality Technology in Visualization of Medical Operations

With the continuous progress of virtual simulation technology,medical surgery visualization system has been developed from two-dimensional to three-dimensional,from digital to network and intelligentization.The visualization system with mixed reality technology will also be used in all stage of medical surgery,such as case discussion,surgical planning,intraoperative guidance,post-operative evaluation,rehabilitation,so as to further promote high intelligence,high precision of medical surgery,and consequently improve effectiveness of treatment and quality of medical service.This paper discusses the composition and technical characteristics of medical operation visualization system based on mixed reality technology,and introduces some typical applications of mixed reality technology in medical operation visualization,which provides a new perspective for the application of mixed technology in medical surgery.

Application and Prospect of Mixed Reality Technology in Medical Field

Mixed reality(MR)technology is a new digital holographic image technology,which appears in the field of graphics after virtual reality(VR)and augmented reality(AR)technology,a new interdisciplinary frontier.As a new generation of technology,MR has attracted great attention of clinicians in recent years.The emergence of MR will bring about revolutionary changes in medical education training,medical research,medical communication,and clinical treatment.At present,MR technology has become the popular frontline information technology for medical applications.With the popularization of digital technology in the medical field,the development prospects of MR are inestimable.The purpose of this review article is to introduce the application of MR technology in the medical field and prospect its trend in the future.

Mixed Reality-based Interactive Technology for Aircraft Cabin Assembly

Due to the narrowness of space and the complexity of structure, the assembly of aircraft cabin has become one of the major bottlenecks in the whole manufacturing process. To solve the problem, at the beginning of aircraft design, the different stages of the lifecycle of aircraft must be thought about, which include the trial manufacture, assembly, maintenance, recycling and destruction of the product. Recently, thanks to the development of the virtual reality and augmented reality, some low-cost and fast solutions are found for the product assembly. This paper presents a mixed reality-based interactive technology for the aircraft cabin assembly, which can enhance the efficiency of the assemblage in a virtual environment in terms of vision, information and operation. In the mixed reality-based assembly environment, the physical scene can be obtained by a camera and then generated by a computer. The virtual parts, the features of visual assembly, the navigation information, the physical parts and the physical assembly environment will be mixed and presented in the same assembly scene. The mixed or the augmented information will provide some assembling information as a detailed assembly instruction in the mixed reality-based assembly environment. Constraint proxy and its match rules help to reconstruct and visualize the restriction relationship among different parts, and to avoid the complex calculation of constraint＇s match. Finally, a desktop prototype system of virtual assembly has been built to assist the assembly verification and training with the virtual hand.

Chinese expert consensus on laparoscopic hepatic segmentectomy and subsegmentectomy navigated by augmented-and mixed-reality technology combined with indocyanine green fluorescence imaging

Augmented-and mixed-reality technologies have pioneered the realization of real-time fusion and interactive projection for laparoscopic surgeries.Indocyanine green fluorescence imaging technology has enabled anatomical,functional,and radical hepatectomy through tumor identification and localization of target hepatic segments,driving a transformative shift in themanagement of hepatic surgical diseases,moving away from traditional,empirical diagnostic and treatment approaches toward digital,intelligent ones.The Hepatic Surgery Group of the Surgery Branch of the Chinese Medical Association,Digital Medicine Branch of the Chinese Medical Association,Digital Intelligent Surgery Committee of the Chinese Society of ResearchHospitals,and Liver Cancer Committee of the Chinese Medical Doctor Association organized the relevant experts in China to formulate this consensus.This consensus provides a comprehensive outline of the principles,advantages,processes,and key considerations associated with the application of augmented reality and mixed-reality technology combined with indocyanine green fluorescence imaging technology for hepatic segmental and subsegmental resection.The purpose is to streamline and standardize the application of these technologies.

Review of Digital Campus Construction Led by Virtual Reality Technology

In recent years,the government has issued a series of documents to promote the construction of digital campuses.This initiative serves to encourage the deep integration of information technology and intelligent technology education and digital reform,the combination of virtual reality and campus management is the need for innovative thinking and economic and social development,and then better change our learning style and living environment.The construction of the digital campus is based on virtual reality technology,BIM,GIS,and three-dimensional modeling technology to provide an immersive platform for students,promote the integration of virtual reality technology and education,help teachers,students,and parents to understand all kinds of education information and resources,to achieve their interoperability.From the off-campus environment to the school teaching equipment,teachers to teaching quality certification,and learning,to extracurricular entertainment,opening ceremonies to graduation parties,to bring more efficient,convenient,and safe campus life for teachers,students,and staff in school,and break the traditional information restrictions.

Application Strategies of Virtual Reality Technology in the Teaching Design of Vocational Courses from the Perspective of Learning Transfer Theory

With the rapid development of virtual reality technology,it has been widely used in the field of education.It can promote the development of learning transfer,which is an effective method for learners to learn effectively.Therefore,this paper describes how to use virtual reality technology to achieve learning transfer in order to achieve teaching goals and improve learning efficiency.

Design of Teaching System of Industrial Robots Using Mixed Reality Technology

Traditional teaching and learning about industrial robots uses abstract instructions,which are difficult for students to understand.Meanwhile,there are safety issues associated with the use of practical training equipment.To address these problems,this paper developed an instructional system based on mixed-reality(MR)technology for teaching about industrial robots.The Siasun T6A-series robots were taken as a case study,and the Microsoft MR device HoloLens-2 was used as the instructional platform.First,the parameters of the robots were analyzed based on their structural drawings.Then,the robot modules were decomposed,and 1:1 three-dimensional(3D)digital reproductions were created in Maya.Next,a library of digital models of the robot components was established,and a 3D spatial operation interface for the virtual instructional system was created in Unity.Subsequently,a C#code framework was established to satisfy the requirements of interactive functions and data transmission,and the data were saved in JSON format.In this way,a key technique that facilitates the understanding of spatial structures and a variety of human-machine interactions were realized.Finally,an instructional system based on HoloLens-2 was established for understanding the structures and principles of robots.The results showed that the instructional system developed in this study provides realistic 3D visualizations and a natural,efficient approach for human-machine interactions.This system could effectively improve the efficiency of knowledge transfer and the student’s motivation to learn.

Remote consultation based on mixed reality technology

Telemedicine includes remote teleradiology,remote ultrasound diagnostics,telesurgery,telemedicine consultation,and other forms.Telemedicine consultation is the most used form of telemedicine.However,the traditional telemedicine consultation is limited by a lack of communication and presentation methods,and its wide application is greatly limited.Mixed reality technology cuts through the boundaries between virtual reality and actual reality,bringing a new method of remote consultation.

Education Platform of Congenital Heart Disease Based on Mixed Reality Technology

Recently the Mixed Reality (MR) technology has a rapid development and strong application prospect in various fields. This technology has been successfully applied in clinical work by means of surgical navigation and puncture positioning. However, there is few reports about MR technology’s applications in China. Therefore, based on the MR technology, a medical education platform was designed for related professions through the reconstruction of 3D heart model. Because of the various types of congenital heart diseases and the comprehensive medical knowledge system involved, it will start from congenital heart disease and expand to other organs later. The teaching mode was enriched and basic and clinical teaching materials were provided. And students’ interests were motivated and learning efficiency was strengthened. An equal communication mechanism between teachers and students was constructed. Experimental results show that both teachers and students benefit from this proposed platform.

Application of Virtual Reality Technology in Constructing Virtual Tourism Environment

Virtual tourism is the application of virtual reality technology in tourism industry,in which the key is to construct lifelike virtual environment.This paper introduces several main methods in producing virtual environment,such as landform formation with the help of height field method,construction of complicated three-dimensional models in virtual environment through 3DS MAX,construction of lifelike tree models using Billboard technique and Cross Billboard technique,and adoption of texture mapping,LOD(level of detail),and collision testing technique to increase sense of reality in virtual environment.Finally a lifelike virtual tourism environment is realized under the context of Visual C++6.0 through OpenGL.

基于MR技术的计算机组装系统设计与实现

使用HoloLens2头盔和混合现实插件Mixed Reality Feature Tool开发出一款计算机组装混合现实交互应用。通过HoloLens2凝视、手部跟踪、UX构建基块、语音和听写系统、眼部跟踪等进行输入交互,并配合Unity3D中的Animation系统与UGUI系统,最终实现混合现实虚拟交互。Unity3D工程导出UWP,并生成VS工程,然后部署到HoloLens2中进行测试。测试结果表明,该系统实现了计算机组装的相关交互功能。研究为计算机组装的虚拟操作提供了一种混合现实解决方案。

Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods

Background: Augmented reality(AR) technology is used to reconstruct three-dimensional(3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.Data Sources: The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the Pub Med database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles.Results: In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery,which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology.Conclusions: With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling,and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.

基于MR混合现实的足球机器人目标跟踪仿真

机器人跟踪足球时,由于目标具有较强的运动随机性,且较小的目标尺度增大了跟踪框的自适应跟踪难度,目标跟踪时会将很多背景信息包含在内,导致目标模型的更新错误。为此,提出基于MR混合现实的足球机器人目标跟踪方法。构建MR混合现实平台,令足球机器人在虚拟现实场景下完成人机交互,在保证目标跟踪安全性的同时提升目标跟踪性能;采用帧差法获取目标模型,利用投影法对二值图像投影,并对差分图像滤波,以此获取足球机器人在目标模板区域及候选目标区域中的直方图。基于此,更新足球机器人目标模型,实现基于MR混合现实的足球机器人目标跟踪。实验结果表明,当跟踪误差阈值为0.8mm时,研究方法的目标跟踪准确率和成功率均可达95%以上,平均耗时为55.17ms。当足球目标出现遮挡问题时,该方法仍能完成高精度跟踪。

Mixed reality for visualization of orthopedic surgical anatomy

In the modern era,preoperative planning is substantially facilitated by artificial reality technologies,which permit a better understanding of patient anatomy,thus increasing the safety and accuracy of surgical interventions.In the field of orthopedic surgery,the increase in safety and accuracy improves treatment quality and orthopedic patient outcomes.Artificial reality technologies,which include virtual reality(VR),augmented reality(AR),and mixed reality(MR),use digital images obtained from computed tomography or magnetic resonance imaging.VR replaces the user’s physical environment with one that is computer generated.AR and MR have been defined as technologies that permit the fusing of the physical with the virtual environment,enabling the user to interact with both physical and virtual objects.MR has been defined as a technology that,in contrast to AR,enables users to visualize the depth and perspective of the virtual models.We aimed to shed light on the role that MR can play in the visualization of orthopedic surgical anatomy.The literature suggests that MR could be a valuable tool in orthopedic surgeon’s hands for visualization of the anatomy.However,we remark that confusion exists in the literature concerning the characteristics of MR.Thus,a more clear description of MR is needed in orthopedic research,so that the potential of this technology can be more deeply understood.

The Augmented Reality Application of Multimedia Technology in Aquatic Organisms Instruction

The paper elaborates the features and superiority of applying multimedia technology to aquatic organisms in a Science and Life Technology course. The augmented reality application of multimedia technology in aquatic organisms’ instruction has its own features, teaching information capacity, interactive user interface. The paper concludes the superiority of Unity and Vuforia AR instruction from five perspectives;meanwhile, the paper elaborates its disadvantages. An AR-based instruction attitude questionnaire was conducted at the end of the experiment to obtain learners’ perspectives of the system. Learners in AR approach demonstrate higher motivation and concentrate their attention on the learning tasks. Learners showed more positive attitude in willingness to use the AR-based instruction to improve learning interests. This study provided insights for better understanding the design, theory and practice of e-learning through augmented reality technology.

Application value of mixed reality in hepatectomy for hepatocellular carcinoma

BACKGROUND As a new digital holographic imaging technology,mixed reality(MR)technology has unique advantages in determining the liver anatomy and location of tumor lesions.With the popularization of 5 G communication technology,MR shows great potential in preoperative planning and intraoperative navigation,making hepatectomy more accurate and safer.AIM To evaluate the application value of MR technology in hepatectomy for hepatocellular carcinoma(HCC).METHODS The clinical data of 95 patients who underwent open hepatectomy surgery for HCC between June 2018 and October 2020 at our hospital were analyzed retrospectively.We selected 95 patients with HCC according to the inclusion criteria and exclusion criteria.In 38 patients,hepatectomy was assisted by MR(Group A),and an additional 57 patients underwent traditional hepatectomy without MR(Group B).The perioperative outcomes of the two groups were collected and compared to evaluate the application value of MR in hepatectomy for patients with HCC.RESULTS We summarized the technical process of MR-assisted hepatectomy in the treatment of HCC.Compared to traditional hepatectomy in Group B,MR-assisted hepatectomy in Group A yielded a shorter operation time(202.86±46.02 min vs 229.52±57.13 min,P=0.003),less volume of bleeding(329.29±97.31 mL vs 398.23±159.61 mL,P=0.028),and shorter obstructive time of the portal vein(17.71±4.16 min vs 21.58±5.24 min,P=0.019).Group A had lower alanine aminotransferas and higher albumin values on the third day after the operation(119.74±29.08 U/L vs 135.53±36.68 U/L,P=0.029 and 33.60±3.21 g/L vs 31.80±3.51 g/L,P=0.014,respectively).The total postoperative complications and hospitalization days in Group A were significantly less than those in Group B[14(37.84%)vs 35(60.34%),P=0.032 and 12.05±4.04 d vs 13.78±4.13 d,P=0.049,respectively].CONCLUSION MR has some application value in three-dimensional visualization of the liver,surgical planning,and intraoperative navigation during hepatectomy,and it significantly improves the perioperative outcomes of hepatectomy for HCC.

A first attempt of inferior vena cava filter successfully guided by a mixed-reality system: a case report

Pulmonary embolism(PE)is one of the fatal heart attacks,and lower limbs deep vein thrombosis(DVT)is the most common reason for PE.Inferior vena cava filter(IVCF)implantation is a most prevention for PE.But it may carry a high risk of injury because of the radiation and contrast agent.Patients with nephrotic syndrome(NS)or some other renal diseases may prone to thrombosis due to the excretion of protein C and protein S overmuch.So,it is necessary to develop a new therapy without contrast agent.Mixed-reality(MR)is a new technology as a guidance of inferior vena cava filter implantation exposed under no X-ray and required no contrast agent.

Uses of a dedicated 3D reconstruction software with augmented and mixed reality in planning and performing advanced liver surgery and living donor liver transplantation(with videos)

The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery simulation and using the created scenarios in real-time surgery using mixed reality.In this article,we described our experience on developing a dedicated 3 dimensional visualization and reconstruction software for surgeons to be used in advanced liver surgery and living donor liver transplantation.Furthermore,we shared the recent developments in the field by explaining the outreach of the software from virtual reality to augmented reality and mixed reality.

Mixed reality breathes fresh energy into the development of modern surgery

With the rapid development of holographic imaging technology,virtual reality and augmented reality technology have been used in medicine for decades.However,due to the lack of compatibility between the virtual world and the real world and the lack of human-computer interaction,their widespread use is limited.As a research hotspot in recent years,the mixed reality technology has a huge prospect in the medical field by virtue of its unique advantages combined with the virtual world and the real world,as well as better human-machine realtime interaction and precise matching.This article will focus on the development of this technology in recent years,and introduce the application of mixed reality technology in medical education and clinical application.

Training on LSA lifeboat operation using Mixed Reality

Background This work aims to provide an overview of the Mixed Reality(MR)technology’s use in maritime industry for training purposes.Current training procedures cover a broad range of procedural operations for Life-Saving Appliances(LSA)lifeboats;however,several gaps and limitations have been identified related to the practical training that can be addressed through the use of MR.Augmented,Virtual and Mixed Reality applications are already used in various fields in maritime industry,but their full potential have not been yet exploited.SafePASS project aims to exploit MR advantages in the maritime training by introducing a relevant application focusing on use and maintenance of LSA lifeboats.Methods An MR Training application is proposed supporting the training of crew members in equipment usage and operation,as well as in maintenance activities and procedures.The application consists of the training tool that trains crew members on handling lifeboats,the training evaluation tool that allows trainers to assess the performance of trainees,and the maintenance tool that supports crew members to perform maintenance activities and procedures on lifeboats.For each tool,an indicative session and scenario workflow are implemented,along with the main supported interactions of the trainee with the equipment.Results The application has been tested and validated both in lab environment and using a real LSA lifeboat,resulting to improved experience for the users that provided feedback and recommendations for further development.The application has also been demonstrated onboard a cruise ship,showcasing the supported functionalities to relevant stakeholders that recognized the added value of the application and suggested potential future exploitation areas.Conclusions The MR Training application has been evaluated as very promising in providing a user-friendly training environment that can support crew members in LSA lifeboat operation and maintenance,while it is still subject to improvement and further expansion.