From text to image:challenges in integrating vision into ChatGPT for medical image interpretation

Large language models(LLMs),such as ChatGPT developed by OpenAI,represent a significant advancement in artificial intelligence(AI),designed to understand,generate,and interpret human language by analyzing extensive text data.Their potential integration into clinical settings offers a promising avenue that could transform clinical diagnosis and decision-making processes in the future(Thirunavukarasu et al.,2023).This article aims to provide an in-depth analysis of LLMs’current and potential impact on clinical practices.Their ability to generate differential diagnosis lists underscores their potential as invaluable tools in medical practice and education(Hirosawa et al.,2023;Koga et al.,2023).

Using restored two-dimensional X-ray images to reconstruct the three-dimensional magnetopause

Astronomical imaging technologies are basic tools for the exploration of the universe,providing basic data for the research of astronomy and space physics.The Soft X-ray Imager(SXI)carried by the Solar wind Magnetosphere Ionosphere Link Explorer(SMILE)aims to capture two-dimensional(2-D)images of the Earth’s magnetosheath by using soft X-ray imaging.However,the observed 2-D images are affected by many noise factors,destroying the contained information,which is not conducive to the subsequent reconstruction of the three-dimensional(3-D)structure of the magnetopause.The analysis of SXI-simulated observation images shows that such damage cannot be evaluated with traditional restoration models.This makes it difficult to establish the mapping relationship between SXIsimulated observation images and target images by using mathematical models.We propose an image restoration algorithm for SXIsimulated observation images that can recover large-scale structure information on the magnetosphere.The idea is to train a patch estimator by selecting noise–clean patch pairs with the same distribution through the Classification–Expectation Maximization algorithm to achieve the restoration estimation of the SXI-simulated observation image,whose mapping relationship with the target image is established by the patch estimator.The Classification–Expectation Maximization algorithm is used to select multiple patch clusters with the same distribution and then train different patch estimators so as to improve the accuracy of the estimator.Experimental results showed that our image restoration algorithm is superior to other classical image restoration algorithms in the SXI-simulated observation image restoration task,according to the peak signal-to-noise ratio and structural similarity.The restoration results of SXI-simulated observation images are used in the tangent fitting approach and the computed tomography approach toward magnetospheric reconstruction techniques,significantly improving the reconstruction results.Hence,the proposed technology may be feasible for processing SXI-simulated observation images.

The Soft X-ray Imager(SXI)on the SMILE Mission

The Soft X-ray Imager(SXI)is part of the scientific payload of the Solar wind Magnetosphere Ionosphere Link Explorer(SMILE)mission.SMILE is a joint science mission between the European Space Agency(ESA)and the Chinese Academy of Sciences(CAS)and is due for launch in 2025.SXI is a compact X-ray telescope with a wide field-of-view(FOV)capable of encompassing large portions of Earth’s magnetosphere from the vantage point of the SMILE orbit.SXI is sensitive to the soft X-rays produced by the Solar Wind Charge eXchange(SWCX)process produced when heavy ions of solar wind origin interact with neutral particles in Earth’s exosphere.SWCX provides a mechanism for boundary detection within the magnetosphere,such as the position of Earth’s magnetopause,because the solar wind heavy ions have a very low density in regions of closed magnetic field lines.The sensitivity of the SXI is such that it can potentially track movements of the magnetopause on timescales of a few minutes and the orbit of SMILE will enable such movements to be tracked for segments lasting many hours.SXI is led by the University of Leicester in the United Kingdom(UK)with collaborating organisations on hardware,software and science support within the UK,Europe,China and the United States.

SMILE soft X-ray Imager flight model CCD370 pre-flight device characterisation

Throughout the SMILE mission the satellite will be bombarded by radiation which gradually damages the focal plane devices and degrades their performance.In order to understand the changes of the CCD370s within the soft X-ray Imager,an initial characterisation of the devices has been carried out to give a baseline performance level.Three CCDs have been characterised,the two flight devices and the flight spa re.This has been carried out at the Open University in a bespo ke cleanroom measure ment facility.The results show that there is a cluster of bright pixels in the flight spa re which increases in size with tempe rature.However at the nominal ope rating tempe rature(-120℃) it is within the procure ment specifications.Overall,the devices meet the specifications when ope rating at -120℃ in 6 × 6 binned frame transfer science mode.The se rial charge transfer inefficiency degrades with temperature in full frame mode.However any charge losses are recovered when binning/frame transfer is implemented.

Automated Algorithms for Detecting and Classifying X-Ray Images of Spine Fractures

This paper emphasizes a faster digital processing time while presenting an accurate method for identifying spinefractures in X-ray pictures. The study focuses on efficiency by utilizing many methods that include picturesegmentation, feature reduction, and image classification. Two important elements are investigated to reducethe classification time: Using feature reduction software and leveraging the capabilities of sophisticated digitalprocessing hardware. The researchers use different algorithms for picture enhancement, including theWiener andKalman filters, and they look into two background correction techniques. The article presents a technique forextracting textural features and evaluates three picture segmentation algorithms and three fractured spine detectionalgorithms using transformdomain, PowerDensity Spectrum(PDS), andHigher-Order Statistics (HOS) for featureextraction.With an emphasis on reducing digital processing time, this all-encompassing method helps to create asimplified system for classifying fractured spine fractures. A feature reduction program code has been built toimprove the processing speed for picture classification. Overall, the proposed approach shows great potential forsignificantly reducing classification time in clinical settings where time is critical. In comparison to other transformdomains, the texture features’ discrete cosine transform (DCT) yielded an exceptional classification rate, and theprocess of extracting features from the transform domain took less time. More capable hardware can also result inquicker execution times for the feature extraction algorithms.

Simulation of the SMILE Soft X-ray Imager response to a southward interplanetary magnetic field turning

The Solar wind Magnetosphere Ionosphere Link Explorer(SMILE)Soft X-ray Imager(SXI)will shine a spotlight on magnetopause dynamics during magnetic reconnection.We simulate an event with a southward interplanetary magnetic field turning and produce SXI count maps with a 5-minute integration time.By making assumptions about the magnetopause shape,we find the magnetopause standoff distance from the count maps and compare it with the one obtained directly from the magnetohydrodynamic(MHD)simulation.The root mean square deviations between the reconstructed and MHD standoff distances do not exceed 0.2 RE(Earth radius)and the maximal difference equals 0.24 RE during the 25-minute interval around the southward turning.

Robust Machine Learning Technique to Classify COVID-19 Using Fusion of Texture and Vesselness of X-Ray Images

Manual investigation of chest radiography(CXR)images by physicians is crucial for effective decision-making in COVID-19 diagnosis.However,the high demand during the pandemic necessitates auxiliary help through image analysis and machine learning techniques.This study presents a multi-threshold-based segmentation technique to probe high pixel intensity regions in CXR images of various pathologies,including normal cases.Texture information is extracted using gray co-occurrence matrix(GLCM)-based features,while vessel-like features are obtained using Frangi,Sato,and Meijering filters.Machine learning models employing Decision Tree(DT)and RandomForest(RF)approaches are designed to categorize CXR images into common lung infections,lung opacity(LO),COVID-19,and viral pneumonia(VP).The results demonstrate that the fusion of texture and vesselbased features provides an effective ML model for aiding diagnosis.The ML model validation using performance measures,including an accuracy of approximately 91.8%with an RF-based classifier,supports the usefulness of the feature set and classifier model in categorizing the four different pathologies.Furthermore,the study investigates the importance of the devised features in identifying the underlying pathology and incorporates histogrambased analysis.This analysis reveals varying natural pixel distributions in CXR images belonging to the normal,COVID-19,LO,and VP groups,motivating the incorporation of additional features such as mean,standard deviation,skewness,and percentile based on the filtered images.Notably,the study achieves a considerable improvement in categorizing COVID-19 from LO,with a true positive rate of 97%,further substantiating the effectiveness of the methodology implemented.

Pulmonary Edema and Pleural Effusion Detection Using Efficient Net-V1-B4 Architecture and AdamW Optimizer from Chest X-Rays Images

This paper presents a novelmulticlass systemdesigned to detect pleural effusion and pulmonary edema on chest Xray images,addressing the critical need for early detection in healthcare.A new comprehensive dataset was formed by combining 28,309 samples from the ChestX-ray14,PadChest,and CheXpert databases,with 10,287,6022,and 12,000 samples representing Pleural Effusion,Pulmonary Edema,and Normal cases,respectively.Consequently,the preprocessing step involves applying the Contrast Limited Adaptive Histogram Equalization(CLAHE)method to boost the local contrast of the X-ray samples,then resizing the images to 380×380 dimensions,followed by using the data augmentation technique.The classification task employs a deep learning model based on the EfficientNet-V1-B4 architecture and is trained using the AdamW optimizer.The proposed multiclass system achieved an accuracy(ACC)of 98.3%,recall of 98.3%,precision of 98.7%,and F1-score of 98.7%.Moreover,the robustness of the model was revealed by the Receiver Operating Characteristic(ROC)analysis,which demonstrated an Area Under the Curve(AUC)of 1.00 for edema and normal cases and 0.99 for effusion.The experimental results demonstrate the superiority of the proposedmulti-class system,which has the potential to assist clinicians in timely and accurate diagnosis,leading to improved patient outcomes.Notably,ablation-CAM visualization at the last convolutional layer portrayed further enhanced diagnostic capabilities with heat maps on X-ray images,which will aid clinicians in interpreting and localizing abnormalities more effectively.

Projection registration of X-ray image and CT image

A methodology for alignment of an X-ray image and a CT image, based on the Chamfer 3-4 distance transform and simulated annealing optimization algorithm is presented. Firstly, an initial transformation matrix is constructed. For the convenience of computing, geometric models of the X-ray device to reconstruct the calibration matrix are used. Then, by defining the distance between the 3-D protective and the 2-D object image, we optimize this distance matching problem, using the simulated annealing algorithm. This method is also integrated into medical intra-operation, dealing with the data set acquired from 3-D image workstation and active navigation.

Investigation of prior image constrained compressed sensing-based spectral X-ray CT image reconstruction

To improve spectral X-ray CT reconstructed image quality, the energy-weighted reconstructed image xbins^W and the separable paraboloidal surrogates（SPS） algorithm are proposed for the prior image constrained compressed sensing（PICCS）-based spectral X-ray CT image reconstruction. The PICCS-based image reconstruction takes advantage of the compressed sensing theory, a prior image and an optimization algorithm to improve the image quality of CT reconstructions.To evaluate the performance of the proposed method, three optimization algorithms and three prior images are employed and compared in terms of reconstruction accuracy and noise characteristics of the reconstructed images in each energy bin.The experimental simulation results show that the image xbins^W is the best as the prior image in general with respect to the three optimization algorithms; and the SPS algorithm offers the best performance for the simulated phantom with respect to the three prior images. Compared with filtered back-projection（FBP）, the PICCS via the SPS algorithm and xbins^W as the prior image can offer the noise reduction in the reconstructed images up to 80. 46%, 82. 51%, 88. 08% in each energy bin,respectively. M eanwhile, the root-mean-squared error in each energy bin is decreased by 15. 02%, 18. 15%, 34. 11% and the correlation coefficient is increased by 9. 98%, 11. 38%,15. 94%, respectively.

X-ray image distortion correction based on SVR

X-ray image has been widely used in many fields such as medical diagnosis,industrial inspection,and so on.Unfortunately,due to the physical characteristics of X-ray and imaging system,distortion of the projected image will happen,which restrict the application of X-ray image,especially in high accuracy fields.Distortion correction can be performed using algorithms that can be classified as global or local according to the method used,both having specific advantages and disadvantages.In this paper,a new global method based on support vector regression（SVR）machine for distortion correction is proposed.In order to test the presented method,a calibration phantom is specially designed for this purpose.A comparison of the proposed method with the traditional global distortion correction techniques is performed.The experimental results show that the proposed correction method performs better than the traditional global one.

Dendritic Learning-Incorporated Vision Transformer for Image Recognition

Dear Editor,This letter proposes to integrate dendritic learnable network architecture with Vision Transformer to improve the accuracy of image recognition.In this study,based on the theory of dendritic neurons in neuroscience,we design a network that is more practical for engineering to classify visual features.Based on this,we propose a dendritic learning-incorporated vision Transformer(DVT),which out-performs other state-of-the-art methods on three image recognition benchmarks.

Infrared and Visible Image Fusion Based on Res2Net-Transformer Automatic Encoding and Decoding

A novel image fusion network framework with an autonomous encoder and decoder is suggested to increase thevisual impression of fused images by improving the quality of infrared and visible light picture fusion. The networkcomprises an encoder module, fusion layer, decoder module, and edge improvementmodule. The encoder moduleutilizes an enhanced Inception module for shallow feature extraction, then combines Res2Net and Transformerto achieve deep-level co-extraction of local and global features from the original picture. An edge enhancementmodule (EEM) is created to extract significant edge features. A modal maximum difference fusion strategy isintroduced to enhance the adaptive representation of information in various regions of the source image, therebyenhancing the contrast of the fused image. The encoder and the EEM module extract features, which are thencombined in the fusion layer to create a fused picture using the decoder. Three datasets were chosen to test thealgorithmproposed in this paper. The results of the experiments demonstrate that the network effectively preservesbackground and detail information in both infrared and visible images, yielding superior outcomes in subjectiveand objective evaluations.

Background removal from global auroral images:Data-driven dayglow modeling

Global images of auroras obtained by cameras on spacecraft are a key tool for studying the near-Earth environment.However,the cameras are sensitive not only to auroral emissions produced by precipitating particles,but also to dayglow emissions produced by photoelectrons induced by sunlight.Nightglow emissions and scattered sunlight can contribute to the background signal.To fully utilize such images in space science,background contamination must be removed to isolate the auroral signal.Here we outline a data-driven approach to modeling the background intensity in multiple images by formulating linear inverse problems based on B-splines and spherical harmonics.The approach is robust,flexible,and iteratively deselects outliers,such as auroral emissions.The final model is smooth across the terminator and accounts for slow temporal variations and large-scale asymmetries in the dayglow.We demonstrate the model by using the three far ultraviolet cameras on the Imager for Magnetopause-to-Aurora Global Exploration(IMAGE)mission.The method can be applied to historical missions and is relevant for upcoming missions,such as the Solar wind Magnetosphere Ionosphere Link Explorer(SMILE)mission.

Method for evaluation of geological strength index of carbonate cliff rocks:Coupled hyperspectral-digital borehole image technique

The deterioration of unstable rock mass raised interest in evaluating rock mass quality.However,the traditional evaluation method for the geological strength index(GSI)primarily emphasizes the rock structure and characteristics of discontinuities.It ignores the influence of mineral composition and shows a deficiency in assessing the integrity coefficient.In this context,hyperspectral imaging and digital panoramic borehole camera technologies are applied to analyze the mineral content and integrity of rock mass.Based on the carbonate mineral content and fissure area ratio,the strength reduction factor and integrity coefficient are calculated to improve the GSI evaluation method.According to the results of mineral classification and fissure identification,the strength reduction factor and integrity coefficient increase with the depth of rock mass.The rock mass GSI calculated by the improved method is mainly concentrated between 40 and 60,which is close to the calculation results of the traditional method.The GSI error rates obtained by the two methods are mostly less than 10%,indicating the rationality of the hyperspectral-digital borehole image coupled evaluation method.Moreover,the sensitivity of the fissure area ratio(Sr)to GSI is greater than that of the strength reduction factor(a),which means the proposed GSI is suitable for rocks with significant fissure development.The improved method reduces the influence of subjective factors and provides a reliable index for the deterioration evaluation of rock mass.

Deep learning-based inpainting of saturation artifacts in optical coherence tomography images

Limited by the dynamic range of the detector,saturation artifacts usually occur in optical coherence tomography(OCT)imaging for high scattering media.The available methods are difficult to remove saturation artifacts and restore texture completely in OCT images.We proposed a deep learning-based inpainting method of saturation artifacts in this paper.The generation mechanism of saturation artifacts was analyzed,and experimental and simulated datasets were built based on the mechanism.Enhanced super-resolution generative adversarial networks were trained by the clear–saturated phantom image pairs.The perfect reconstructed results of experimental zebrafish and thyroid OCT images proved its feasibility,strong generalization,and robustness.

Artificial Intelligence and Computer Vision during Surgery: Discussing Laparoscopic Images with ChatGPT4—Preliminary Results

Introduction: Ultrafast latest developments in artificial intelligence (ΑΙ) have recently multiplied concerns regarding the future of robotic autonomy in surgery. However, the literature on the topic is still scarce. Aim: To test a novel AI commercially available tool for image analysis on a series of laparoscopic scenes. Methods: The research tools included OPENAI CHATGPT 4.0 with its corresponding image recognition plugin which was fed with a list of 100 laparoscopic selected snapshots from common surgical procedures. In order to score reliability of received responses from image-recognition bot, two corresponding scales were developed ranging from 0 - 5. The set of images was divided into two groups: unlabeled (Group A) and labeled (Group B), and according to the type of surgical procedure or image resolution. Results: AI was able to recognize correctly the context of surgical-related images in 97% of its reports. For the labeled surgical pictures, the image-processing bot scored 3.95/5 (79%), whilst for the unlabeled, it scored 2.905/5 (58.1%). Phases of the procedure were commented in detail, after all successful interpretations. With rates 4 - 5/5, the chatbot was able to talk in detail about the indications, contraindications, stages, instrumentation, complications and outcome rates of the operation discussed. Conclusion: Interaction between surgeon and chatbot appears to be an interesting frontend for further research by clinicians in parallel with evolution of its complex underlying infrastructure. In this early phase of using artificial intelligence for image recognition in surgery, no safe conclusions can be drawn by small cohorts with commercially available software. Further development of medically-oriented AI software and clinical world awareness are expected to bring fruitful information on the topic in the years to come.

Enhancing Dense Small Object Detection in UAV Images Based on Hybrid Transformer

Transformer-based models have facilitated significant advances in object detection.However,their extensive computational consumption and suboptimal detection of dense small objects curtail their applicability in unmanned aerial vehicle(UAV)imagery.Addressing these limitations,we propose a hybrid transformer-based detector,H-DETR,and enhance it for dense small objects,leading to an accurate and efficient model.Firstly,we introduce a hybrid transformer encoder,which integrates a convolutional neural network-based cross-scale fusion module with the original encoder to handle multi-scale feature sequences more efficiently.Furthermore,we propose two novel strategies to enhance detection performance without incurring additional inference computation.Query filter is designed to cope with the dense clustering inherent in drone-captured images by counteracting similar queries with a training-aware non-maximum suppression.Adversarial denoising learning is a novel enhancement method inspired by adversarial learning,which improves the detection of numerous small targets by counteracting the effects of artificial spatial and semantic noise.Extensive experiments on the VisDrone and UAVDT datasets substantiate the effectiveness of our approach,achieving a significant improvement in accuracy with a reduction in computational complexity.Our method achieves 31.9%and 21.1%AP on the VisDrone and UAVDT datasets,respectively,and has a faster inference speed,making it a competitive model in UAV image object detection.

Two-Staged Method for Ice Channel Identification Based on Image Segmentation and Corner Point Regression

Identification of the ice channel is the basic technology for developing intelligent ships in ice-covered waters,which is important to ensure the safety and economy of navigation.In the Arctic,merchant ships with low ice class often navigate in channels opened up by icebreakers.Navigation in the ice channel often depends on good maneuverability skills and abundant experience from the captain to a large extent.The ship may get stuck if steered into ice fields off the channel.Under this circumstance,it is very important to study how to identify the boundary lines of ice channels with a reliable method.In this paper,a two-staged ice channel identification method is developed based on image segmentation and corner point regression.The first stage employs the image segmentation method to extract channel regions.In the second stage,an intelligent corner regression network is proposed to extract the channel boundary lines from the channel region.A non-intelligent angle-based filtering and clustering method is proposed and compared with corner point regression network.The training and evaluation of the segmentation method and corner regression network are carried out on the synthetic and real ice channel dataset.The evaluation results show that the accuracy of the method using the corner point regression network in the second stage is achieved as high as 73.33%on the synthetic ice channel dataset and 70.66%on the real ice channel dataset,and the processing speed can reach up to 14.58frames per second.

An Intelligent Sensor Data Preprocessing Method for OCT Fundus Image Watermarking Using an RCNN

Watermarks can provide reliable and secure copyright protection for optical coherence tomography(OCT)fundus images.The effective image segmentation is helpful for promoting OCT image watermarking.However,OCT images have a large amount of low-quality data,which seriously affects the performance of segmentationmethods.Therefore,this paper proposes an effective segmentation method for OCT fundus image watermarking using a rough convolutional neural network(RCNN).First,the rough-set-based feature discretization module is designed to preprocess the input data.Second,a dual attention mechanism for feature channels and spatial regions in the CNN is added to enable the model to adaptively select important information for fusion.Finally,the refinement module for enhancing the extraction power of multi-scale information is added to improve the edge accuracy in segmentation.RCNN is compared with CE-Net and MultiResUNet on 83 gold standard 3D retinal OCT data samples.The average dice similarly coefficient(DSC)obtained by RCNN is 6%higher than that of CE-Net.The average 95 percent Hausdorff distance(95HD)and average symmetric surface distance(ASD)obtained by RCNN are 32.4%and 33.3%lower than those of MultiResUNet,respectively.We also evaluate the effect of feature discretization,as well as analyze the initial learning rate of RCNN and conduct ablation experiments with the four different models.The experimental results indicate that our method can improve the segmentation accuracy of OCT fundus images,providing strong support for its application in medical image watermarking.