Development of Multi-Agent-Based Indoor 3D Reconstruction

Large-scale indoor 3D reconstruction with multiple robots faces challenges in core enabling technologies.This work contributes to a framework addressing localization,coordination,and vision processing for multi-agent reconstruction.A system architecture fusing visible light positioning,multi-agent path finding via reinforcement learning,and 360°camera techniques for 3D reconstruction is proposed.Our visible light positioning algorithm leverages existing lighting for centimeter-level localization without additional infrastructure.Meanwhile,a decentralized reinforcement learning approach is developed to solve the multi-agent path finding problem,with communications among agents optimized.Our 3D reconstruction pipeline utilizes equirectangular projection from 360°cameras to facilitate depth-independent reconstruction from posed monocular images using neural networks.Experimental validation demonstrates centimeter-level indoor navigation and 3D scene reconstruction capabilities of our framework.The challenges and limitations stemming from the above enabling technologies are discussed at the end of each corresponding section.In summary,this research advances fundamental techniques for multi-robot indoor 3D modeling,contributing to automated,data-driven applications through coordinated robot navigation,perception,and modeling.

Multi-View Image-Based 3D Reconstruction in Indoor Scenes:A Survey

Three-dimensional reconstruction technology plays an important role in indoor scenes by converting objects and structures in indoor environments into accurate 3D models using multi-view RGB images.It offers a wide range of applications in fields such as virtual reality,augmented reality,indoor navigation,and game development.Existing methods based on multi-view RGB images have made significant progress in 3D reconstruction.These image-based reconstruction methods not only possess good expressive power and generalization performance,but also handle complex geometric shapes and textures effectively.Despite facing challenges such as lighting variations,occlusion,and texture loss in indoor scenes,these challenges can be effectively addressed through deep neural networks,neural implicit surface representations,and other techniques.The technology of indoor 3D reconstruction based on multi-view RGB images has a promising future.It not only provides immersive and interactive virtual experiences but also brings convenience and innovation to indoor navigation,interior design,and virtual tours.As the technology evolves,these image-based reconstruction methods will be further improved to provide higher quality and more accurate solutions to indoor scene reconstruction.

SAR regional all-azimuth observation orbit design for target 3D reconstruction

Three-dimensional(3D) synthetic aperture radar(SAR)extends the conventional 2D images into 3D features by several acquisitions in different aspects. Compared with 3D techniques via multiple observations in elevation, e.g. SAR interferometry(InSAR) and SAR tomography(TomoSAR), holographic SAR can retrieve 3D structure by observations in azimuth. This paper focuses on designing a novel type of orbit to achieve SAR regional all-azimuth observation(AAO) for embedded targets detection and holographic 3D reconstruction. The ground tracks of the AAO orbit separate the earth surface into grids. Target in these grids can be accessed with an azimuth angle span of360°, which is similar to the flight path of airborne circular SAR(CSAR). Inspired from the successive coverage orbits of optical sensors, several optimizations are made in the proposed method to ensure favorable grazing angles, the performance of 3D reconstruction, and long-term supervision for SAR sensors. Simulation experiments show the regional AAO can be completed within five hours. In addition, a second AAO of the same area can be duplicated in two days. Finally, an airborne SAR data process result is presented to illustrate the significance of AAO in 3D reconstruction.

Accurate resection of hilar cholangiocarcinoma using eOrganmap 3D reconstruction and full quantization technique

BACKGROUND For treatment of hilar cholangiocarcinoma(HCCA),the rate of radical resection is low and prognosis is poor,and preoperative evaluation is not sufficiently accurate.3D visualization has the advantage of giving a stereoscopic view,which makes accurate resection of HCCA possible.AIM To establish precise resection of HCCA based on eOrganmap 3D reconstruction and full quantification technology.METHODS We retrospectively analyzed the clinical data of 73 patients who underwent HCCA surgery.All patients were assigned to two groups.The traditional group received traditional 2D imaging planning before surgery(n=35).The eOrganmap group underwent 3D reconstruction and full quantitative technical planning before surgery(n=38).The preoperative evaluation,anatomical classification of hilar hepatic vessels,indicators associated with surgery,postoperative complications,liver function,and stress response indexes were compared between the groups.RESULTS Compared with the traditional group,the amount of intraoperative blood loss in the eOrganmap group was lower,the operating time and postoperative intestinal ventilation time were shorter,and R0 resection rate and lymph node dissection number were higher(P<0.05).The total complication rate in the eOrganmap group was 21.05%compared with 25.71%in the traditional group(P>0.05).The levels of total bilirubin,Albumin(ALB),aspartate transaminase,and alanine transaminase in the eOrganmap group were significantly different from those in the traditional group(intergroup effect:F=450.400,79.120,95.730,and 13.240,respectively;all P<0.001).Total bilirubin,aspartate transaminase,and alanine transaminase in both groups showed a decreasing trend with time(time effect:F=30.270,17.340,and 13.380,respectively;all P<0.001).There was an interaction between patient group and time(interaction effect:F=3.072,2.965,and 2.703,respectively;P=0.0282,0.032,and 0.046,respectively);ALB levels in both groups tended to increase with time(time effect:F=22.490,P<0.001),and there was an interaction effect between groups and time(interaction effect:F=4.607,P=0.004).In the eOrganmap group,there was a high correlation between the actual volume of intraoperative liver specimen resection and the volume of preoperative virtual liver resection(t=0.916,P<0.001).CONCLUSION The establishment of accurate laparoscopic resection of hilar cholangiocarcinoma based on preoperative eOrganmap 3D reconstruction and full quantization technology can make laparoscopic resection of hilar cholangiocarcinoma more accurate and safe.

Preoperative 3D reconstruction and fluorescent indocyanine green for laparoscopic duodenum preserving pancreatic head resection:A case report

BACKGROUND Duodenum-preserving pancreatic head resection(DPPHR)is the choice of surgery for benign or low-grade malignant tumors of the pancreatic head.Laparoscopic DPPHR(LDPPHR)procedure can be improved by preoperative 3D model reconstruction and the use of intravenous indocyanine green fluorescent before surgery for real-time navigation with fluorescent display to guide the surgical dissection and prevention of from injury to vessels and biliary tract.CASE SUMMARY Here we report the successful short-and long-term outcomes after one year following LDPPHR for a 60-year lady who had an uneventful recovery and was discharged home one week after the surgery.CONCLUSION There was no bile leakage or pancreatic leakage or delayed gastric emptying.The histopathology report showed multiple cysts in the pancreatic head and localized pancreatic intraepithelial tumor lesions.The resected margin was free of tumor.

Fast Estimation of Loader’s Shovel Load Volume by 3D Reconstruction of Material Piles

Fast and accurate measurement of the volume of earthmoving materials is of great signifcance for the real-time evaluation of loader operation efciency and the realization of autonomous operation. Existing methods for volume measurement, such as total station-based methods, cannot measure the volume in real time, while the bucket-based method also has the disadvantage of poor universality. In this study, a fast estimation method for a loader’s shovel load volume by 3D reconstruction of material piles is proposed. First, a dense stereo matching method (QORB–MAPM) was proposed by integrating the improved quadtree ORB algorithm (QORB) and the maximum a posteriori probability model (MAPM), which achieves fast matching of feature points and dense 3D reconstruction of material piles. Second, the 3D point cloud model of the material piles before and after shoveling was registered and segmented to obtain the 3D point cloud model of the shoveling area, and the Alpha-shape algorithm of Delaunay triangulation was used to estimate the volume of the 3D point cloud model. Finally, a shovel loading volume measurement experiment was conducted under loose-soil working conditions. The results show that the shovel loading volume estimation method (QORB–MAPM VE) proposed in this study has higher estimation accuracy and less calculation time in volume estimation and bucket fll factor estimation, and it has signifcant theoretical research and engineering application value.

3D Reconstruction for Motion Blurred Images Using Deep Learning-Based Intelligent Systems

The 3D reconstruction using deep learning-based intelligent systems can provide great help for measuring an individual’s height and shape quickly and accurately through 2D motion-blurred images.Generally,during the acquisition of images in real-time,motion blur,caused by camera shaking or human motion,appears.Deep learning-based intelligent control applied in vision can help us solve the problem.To this end,we propose a 3D reconstruction method for motion-blurred images using deep learning.First,we develop a BF-WGAN algorithm that combines the bilateral filtering(BF)denoising theory with a Wasserstein generative adversarial network(WGAN)to remove motion blur.The bilateral filter denoising algorithm is used to remove the noise and to retain the details of the blurred image.Then,the blurred image and the corresponding sharp image are input into the WGAN.This algorithm distinguishes the motion-blurred image from the corresponding sharp image according to the WGAN loss and perceptual loss functions.Next,we use the deblurred images generated by the BFWGAN algorithm for 3D reconstruction.We propose a threshold optimization random sample consensus(TO-RANSAC)algorithm that can remove the wrong relationship between two views in the 3D reconstructed model relatively accurately.Compared with the traditional RANSAC algorithm,the TO-RANSAC algorithm can adjust the threshold adaptively,which improves the accuracy of the 3D reconstruction results.The experimental results show that our BF-WGAN algorithm has a better deblurring effect and higher efficiency than do other representative algorithms.In addition,the TO-RANSAC algorithm yields a calculation accuracy considerably higher than that of the traditional RANSAC algorithm.

3D reconstruction of worn parts for flexible remanufacture based on robotic arc welding

3D reconstruction of worn parts is the foundation for remanufacturing system based on robotic arc welding, because it can provide 3D geometric information for robot task plan. In this investigation, a novel 3D reconstruction system based on linear structured light vision sensing is developed. This system hardware consists of a MTC368-CB CCD camera, a MLH-645 laser projector and a DH-CG300 image grabbing card. This system software is developed to control the image data capture. In order to reconstruct the 3D geometric information from the captured image, a two steps rapid calibration algorithm is proposed. The 3D reconstruction experiment shows a satisfactory result.

Uses of a dedicated 3D reconstruction software with augmented and mixed reality in planning and performing advanced liver surgery and living donor liver transplantation(with videos)

The development of digital intelligent diagnostic and treatment technology has opened countless new opportunities for liver surgery from the era of digital anatomy to a new era of digital diagnostics,virtual surgery simulation and using the created scenarios in real-time surgery using mixed reality.In this article,we described our experience on developing a dedicated 3 dimensional visualization and reconstruction software for surgeons to be used in advanced liver surgery and living donor liver transplantation.Furthermore,we shared the recent developments in the field by explaining the outreach of the software from virtual reality to augmented reality and mixed reality.

Deep-Learning-Empowered 3D Reconstruction for Dehazed Images in IoT-Enhanced Smart Cities

With increasingly more smart cameras deployed in infrastructure and commercial buildings,3D reconstruction can quickly obtain cities’information and improve the efficiency of government services.Images collected in outdoor hazy environments are prone to color distortion and low contrast;thus,the desired visual effect cannot be achieved and the difficulty of target detection is increased.Artificial intelligence(AI)solutions provide great help for dehazy images,which can automatically identify patterns or monitor the environment.Therefore,we propose a 3D reconstruction method of dehazed images for smart cities based on deep learning.First,we propose a fine transmission image deep convolutional regression network(FT-DCRN)dehazing algorithm that uses fine transmission image and atmospheric light value to compute dehazed image.The DCRN is used to obtain the coarse transmission image,which can not only expand the receptive field of the network but also retain the features to maintain the nonlinearity of the overall network.The fine transmission image is obtained by refining the coarse transmission image using a guided filter.The atmospheric light value is estimated according to the position and brightness of the pixels in the original hazy image.Second,we use the dehazed images generated by the FT-DCRN dehazing algorithm for 3D reconstruction.An advanced relaxed iterative fine matching based on the structure from motion(ARI-SFM)algorithm is proposed.The ARISFM algorithm,which obtains the fine matching corner pairs and reduces the number of iterations,establishes an accurate one-to-one matching corner relationship.The experimental results show that our FT-DCRN dehazing algorithm improves the accuracy compared to other representative algorithms.In addition,the ARI-SFM algorithm guarantees the precision and improves the efficiency.

Application of computer tomography-based 3D reconstruction technique in hernia repair surgery

BACKGROUND Hernia is a common condition requiring abdominal surgery.The current standard treatment for hernia is tension-free repair using meshes.Globally,more than 200 new types of meshes are licensed each year.However,their clinical applications are associated with a series of complications,such as recurrence(10%-24%)and infection(0.5%-9.0%).In contrast,3D-printed meshes have significantly reduced the postoperative complications in patients.They have also shortened operating time and minimized the loss of mesh materials.In this study,we used the myopectineal orifice(MPO)data obtained from preoperative computer tomography(CT)-based 3D reconstruction for the production of 3D-printed biologic meshes.AIM To investigate the application of multislice spiral CT-based 3D reconstruction technique in 3D-printed biologic mesh for hernia repair surgery.METHODS We retrospectively analyzed 60 patients who underwent laparoscopic tension-free repair for inguinal hernia in the Department of General Surgery of the First Hospital of Shanxi Medical University from September 2019 to December 2019.This study included 30 males and 30 females,with a mean age of 40±5.6 years.Data on the MPO were obtained from preoperative CT-based 3D reconstruction as well as from real-world intraoperative measurements for all patients.Anatomic points were set for the purpose of measurement based on the definition of MPO:A:The pubic tubercle;B:Intersection of the horizontal line extending from the summit of the inferior edge of the internal oblique and transversus abdominis and the outer edge of the rectus abdominis,C:Intersection of the horizontal line extending from the summit of the inferior edge of the internal oblique and transversus abdominis and the inguinal ligament,D:Intersection of the iliopsoas muscle and the inguinal ligament,and E:Intersection of the iliopsoas muscle and the superior pubic ramus.The distance between the points was measured.All preoperative and intraoperative data were analyzed using the t test.Differences with P<0.05 were considered significant in comparative analysis.RESULTS The distance between points AB,AC,BC,DE,and AE based on preoperative and intraoperative data was 7.576±0.212 cm vs 7.573±0.266 cm,7.627±0.212 cm vs 7.627±0.212 cm,7.677±0.229 cm vs 7.567±0.786 cm,7.589±0.204 cm vs 7.512±0.21 cm,and 7.617±0.231 cm vs 7.582±0.189 cm,respectively.All differences were not statistically significant(P>0.05).CONCLUSION The use of multislice spiral CT-based 3D reconstruction technique before hernia repair surgery allows accurate measurement of data and relationships of different anatomic sites in the MPO region.This technique can provide precise data for the production of 3D-printed biologic meshes.

MULTIPARAMETER MEASUREMENT FOR RACEWAY GROOVE OF BEARING BASED ON 3D RECONSTRUCTION WITH DIGITAL STRUCTURED LIGHT

A fast 3D reconstruction method based on structured light to measure various parameters of the raceway groove is presented. Digital parallel grating stripes distributed with sine density are projected onto the raceway groove by a DLP projector, and distorting of stripes is happened on the raceway. Simultaneously, aided by three-step phase-shifting approach, three images covered by different stripes are obtained by a high-resolution CCD camera at the same location, thus a more accuracy local topography can be obtained. And then the bearing is rotated on a high precision computer controlled rotational stage. Three images are also obtained as the former step at next planned location triggered by the motor. After one cycle, all images information is combined through the mosaics. As a result, the 3D information of raceway groove can be gained. Not only geometric properties but also surface flaws can be extracted by software. A preliminary hardware system has been built, with which some geometric parameters have been extracted from reconstructed local topography.

3D Reconstruction by Kinect Sensor:A Brief Review

While Kinect was originally designed as a motion sensing input device of the gaming console Microsoft Xbox 360 for gaming purposes, it＇s easy-to-use, low-cost, reliability, speed of the depth measurement and relatively high quality of depth measurement make it can be used for 3D reconstruction. It could make 3D scanning technology more accessible to everyday users and turn 3D reconstruction models into much widely used asset for many applications. In this paper, we focus on Kinect 3D reconstruction.

3D reconstruction method based on contour features

To guarantee the accuracy and real-time of the 3D reconstruction method for outdoor scene,an algorithm based on region segmentation and matching was proposed.Firstly,on the basis of morphological gradient information,obtained by comparing color weight gradient images and proposing a multi-threshold segmentation,scene contour features were extracted by a watershed algorithm and a fuzzy c-means clustering algorithm.Secondly,to reduce the search area,increase the correct matching ratio and accelerate the matching speed,the region constraint was established according to a region＇s local position,area and gray characteristics,the edge pixel constraint was established according to the epipolar constraint and the continuity constraint.Finally,by using the stereo matching edge pixel pairs,their 3D coordinates were estimated according to the binocular stereo vision imaging model.Experimental results show that the proposed method can yield a high stereo matching ratio and reconstruct a 3D scene quickly and efficiently.

Improved Lightweight Deep Learning Algorithm in 3D Reconstruction

The three-dimensional(3D)reconstruction technology based on structured light has been widely used in the field of industrial measurement due to its many advantages.Aiming at the problems of high mismatch rate and poor real-time performance caused by factors such as system jitter and noise,a lightweight stripe image feature extraction algorithm based on You Only Look Once v4(YOLOv4)network is proposed.First,Mobilenetv3 is used as the backbone network to effectively extract features,and then the Mish activation function and Complete Intersection over Union(CIoU)loss function are used to calculate the improved target frame regression loss,which effectively improves the accuracy and real-time performance of feature detection.Simulation experiment results show that the model size after the improved algorithm is only 52 MB,the mean average accuracy(mAP)of fringe image data reconstruction reaches 82.11%,and the 3D point cloud restoration rate reaches 90.1%.Compared with the existing model,it has obvious advantages and can satisfy the accuracy and real-time requirements of reconstruction tasks in resource-constrained equipment.

Vertebrae CT Images 3D Reconstruction with Improve Marching Cubes Algorithm

Medical images 3D reconstruction is an important part in medical image analysis and processing. Although lots of algorithms have been proposed continuously, speed and accuracy cannot conform to actual needs, which has always been the focus topic. In this paper, we propose an Improved Marching Cubes algorithm ( I-MC) based on the surface rendering theory, which implements 3D reconstruction of the vertebrae. Firstly, we preprocessed the original 2D vertebrae CT images with the bilateral-filter denoising algorithm. Secondly, on the basis of the traditional Marching Cubes algorithm, the seed voxels were extracted and the Region Growing algorithm was used to determine all voxels that contain isosurfaces. Then, the Golden Section instead of the traditional linear interpolation was used to calculate the equivalent point, and this method reduced the calculations of public edges. VTK and OpenGL implemented 3D reconstruction of the vertebrae on GPU quickly and accurately. The experimental results show that when compared with the traditional Marching Cubes algorithm and Mesh Simplification Marching Cubes algorithm, the improved algorithm achieves a significant improvement of reconstruction speed while preserving the accurate results. The efficiency of algorithm is improved dramatically. This method is real-time and achieves the goal of efficient 3D reconstruction of vertebrae CT images.

3D Reconstruction of Fruit Shape based on Vision and Edge Sections

The fruit industry has been known as one of the largest businesses in Malaysia,where most of the fruits pass through the peeling process well in advance before the final product as juice in a bottle or slices in a can.The current industrial fruit peeling techniques are passive and inefficient by cutting parts of the pulp of the fruit with peels leading to losses.To avoid this issue,a multi-axis CNC fruit peeler can be used to precisely peel the outer layer with the guidance of a 3D virtual model of fruit.In this work,a new cost-effective method of 3D image reconstruction was developed to convert 36 fruit images captured by a normal RGB camera to a 3D model by capturing a single image every 10 degrees of fruit rotation along a fixed axis.The point cloud data extracted with edge detection were passed to Blender 3D software for meshing in different approaches.The vertical link frame meshing method developed in this research proved a qualitative similarity between the output result and the scanned fruit in a processing time of less than 50 seconds.

3D reconstruction of complex curved objects from line drawings

An active research topic in computer vision and graphics is developing algorithms that can reconstruct the 3D surface of curved objects from line drawings. There are a number of algorithms have been dedicated to solve this problem, but they can＇t solve this problem when the geometric structure of a curved object becomes complex. This paper proposes a novel approach to reconstructing a complex curved 3D object from single 2D line drawings. Our approach has three steps： （1） decomposing a complex line drawing into several simpler line drawings and transforming them into polyhedron; （2） reconstructing the 3D wireframe of curved object from these simpler line drawings and generating the curved faces; （3） combining the 3D objects into the complete objects. A number of examples are given to demonstrate the ability of our approach to successfully perform reconstruction of curved objects which are more complex than previous methods.

3D Reconstruction from 2D Line Drawings only with Visible Vertices and Edges

The human vision system can reconstruct a 3D object easily from single 2D line drawings even if the hidden lines of the object are invisible. Now, there are many methods have emulated this ability, but when the hidden lines of the object are invisible, these methods cannot reconstruct a complete 3D object. Therefore, we put forward a new algorithm to settle this hard problem. Our approach consists of two steps： （1） infer the invisible vertices and edges to complete the line drawing, （2） propose a vertex-based optimization method to reconstruct a 3D object.

Application of generalized regression neural network on fast 3D reconstruction

In robot-assisted surgery projects,researchers should be able to make fast 3D reconstruction. Usually 2D images acquired with common diagnostic equipments such as UT, CT and MRI are not enough and complete for an accurate 3D reconstruction. There are some interpolation methods for approximating non value voxels which consume large execution time. A novel algorithm is introduced based on generalized regression neural network (GRNN) which can interpolate unknown voxles fast and reliable. The GRNN interpolation is used to produce new 2D images between each two succeeding ultrasonic images. It is shown that the composition of GRNN with image distance transformation can produce higher quality 3D shapes. The results of this method are compared with other interpolation methods practically. It shows this method can decrease overall time consumption on online 3D reconstruction.