Arthroscopic scene segmentation using multispectral reconstructed frames and deep learning

Background Knee arthroscopy is one of the most complex minimally invasive surgeries,and it is routinelyperformed to treat a range of ailments and injuries to the knee joint.Its complex ergonomic design imposesvisualization and navigation constraints,consequently leading to unintended tissue damage and a steep learningcurve before surgeons gain proficiency.The lack of robust visual texture and landmark frame features furtherlimits the success of image-guided approaches to knee arthroscopy Feature-and texture-less tissue structures ofknee anatomy,lighting conditions,noise,blur,debris,lack of accurate ground-truth label,tissue degeneration,and injury make semantic segmentation an extremely challenging task.To address this complex research problem,this study reported the utility of reconstructed surface reflectance as a viable piece of information that could beused with cutting-edge deep learning technique to achieve highly accurate segmented scenes.Methods We proposed an intraoperative,two-tier deep learning method that makes full use of tissue reflectanceinformation present within an RGB frame to segment texture-less images into multiple tissue types from kneearthroscopy video frames.This study included several cadaver knees experiments at the Medical and EngineeringResearch Facility,located within the Prince Charles Hospital campus,Brisbane Queensland.Data were collectedfrom a total of five cadaver knees,three were males and one from a female.The age range of the donors was 56–93 years.Aging-related tissue degeneration and some anterior cruciate ligament injury were observed in mostcadaver knees.An arthroscopic image dataset was created and subsequently labeled by clinical experts.Thisstudy also included validation of a prototype stereo arthroscope,along with conventional arthroscope,to attainlarger field of view and stereo vision.We reconstructed surface reflectance from camera responses that exhibiteddistinct spatial features at different wavelengths ranging from 380 to 730 nm in the RGB spectrum.Toward theaim to segment texture-less tissue types,this data was used within a two-stage deep learning model.Results The accuracy of the network was measured using dice coefficient score.The average segmentationaccuracy for the tissue-type articular cruciate ligament(ACL)was 0.6625,for the tissue-type bone was 0.84,and for the tissue-type meniscus was 0.565.For the analysis,we excluded extremely poor quality of frames.Here,a frame is considered extremely poor quality when more than 50%of any tissue structures are over-orunderexposed due to nonuniform light exposure.Additionally,when only high quality of frames was consideredduring the training and validation stage,the average bone segmentation accuracy improved to 0.92 and theaverage ACL segmentation accuracy reached 0.73.These two tissue types,namely,femur bone and ACL,have ahigh importance in arthroscopy for tissue tracking.Comparatively,the previous work based on RGB data achieveda much lower average accuracy for femur,tibia,ACL,and meniscus of 0.78,0.50,0.41,and 0.43 using U-Net and0.79,0.50,0.51,and 0.48 using U-Net++.From this analysis,it is clear that our multispectral method outperformsthe previously proposed methods and delivers a much better solution in achieving automatic arthroscopic scenesegmentation.Conclusion The method was based on the deep learning model and requires a reconstructed surface reflectance.It could provide tissue awareness in an intraoperative manner that has a high potential to improve surgicalprecisions.It could be applied to other minimally invasive surgeries as an online segmentation tool for training,aiding,and guiding the surgeons as well as image-guided surgeries.