Super-resolution microscopy and its applications in neuroscience

Optical microscopy promises researchers to soe most tiny substances directly.However,the resolution of conventional microscopy is resticted by the diffraction limit.This makes it a challenge to observe subcellular processes happened in nanoscale.The development of super-resolution microscopy provides a solution to this challenge.Here,we briefly review several commonly used super-resolution techniques,explicating their basic principles and applications in biological science,especially in neuroscience.In addition,characteristics and limitations of each techrique are compared to provide a guidance for biologists to choose the most suitable tool.

Super-resolution microscopy based on parallel detection

Image scanning microscopy based on pixel reassignment can improve the confocal resolution limit without losing the image signal-to-noise ratio(SNR)greatly[C.J.R.Sheppard,"Super resolution in confocal imaging,"Optik 80(2)53-54(1988).C.B.Miller,E.Jorg,"Image scanning microscopy,"Phys.Reu.Lett.104(19)198101(2010).C.J.R.Sheppard,s.B.Mehta,R Heintzmann,"Superresolution by image scanning microscopy using pixel reassignment,"Opt.Lett.38(15)28892892(2013)].Here,we use a tailor-made optical fiber and 19 avalanche pho-todiodes(APDs)as parallel detectors to upgrade our existing confocal microscopy,termed as parallel-detection super resolution(PDSR)microscopy.In order to obtain the correct shift value,we use the normalized 2D cross correlation to calculate the shifting value of each image.We characterized our system performance by imaging fuorescence beads and applied this system to observing the 3D structure of biological specimen.

Temporal compressive super-resolution microscopy at frame rate of 1200 frames per second and spatial resolution of 100 nm

Various super-resolution microscopy techniques have been presented to explore fine structures of biological specimens.However,the super-resolution capability is often achieved at the expense of reducing imaging speed by either point scanning or multiframe computation.The contradiction between spatial resolution and imaging speed seriously hampers the observation of high-speed dynamics of fine structures.To overcome this contradiction,here we propose and demonstrate a temporal compressive super-resolution microscopy(TCSRM)technique.This technique is to merge an enhanced temporal compressive microscopy and a deep-learning-based super-resolution image reconstruction,where the enhanced temporal compressive microscopy is utilized to improve the imaging speed,and the deep-learning-based super-resolution image reconstruction is used to realize the resolution enhancement.The high-speed super-resolution imaging ability of TCSRM with a frame rate of 1200 frames per second(fps)and spatial resolution of 100 nm is experimentally demonstrated by capturing the flowing fluorescent beads in microfluidic chip.Given the outstanding imaging performance with high-speed super-resolution,TCSRM provides a desired tool for the studies of high-speed dynamical behaviors in fine structures,especially in the biomedical field.

Shear Let Transform Residual Learning Approach for Single-Image Super-Resolution

Super-resolution techniques are employed to enhance image resolution by reconstructing high-resolution images from one or more low-resolution inputs.Super-resolution is of paramount importance in the context of remote sensing,satellite,aerial,security and surveillance imaging.Super-resolution remote sensing imagery is essential for surveillance and security purposes,enabling authorities to monitor remote or sensitive areas with greater clarity.This study introduces a single-image super-resolution approach for remote sensing images,utilizing deep shearlet residual learning in the shearlet transform domain,and incorporating the Enhanced Deep Super-Resolution network(EDSR).Unlike conventional approaches that estimate residuals between high and low-resolution images,the proposed approach calculates the shearlet coefficients for the desired high-resolution image using the provided low-resolution image instead of estimating a residual image between the high-and low-resolution image.The shearlet transform is chosen for its excellent sparse approximation capabilities.Initially,remote sensing images are transformed into the shearlet domain,which divides the input image into low and high frequencies.The shearlet coefficients are fed into the EDSR network.The high-resolution image is subsequently reconstructed using the inverse shearlet transform.The incorporation of the EDSR network enhances training stability,leading to improved generated images.The experimental results from the Deep Shearlet Residual Learning approach demonstrate its superior performance in remote sensing image recovery,effectively restoring both global topology and local edge detail information,thereby enhancing image quality.Compared to other networks,our proposed approach outperforms the state-of-the-art in terms of image quality,achieving an average peak signal-to-noise ratio of 35 and a structural similarity index measure of approximately 0.9.

Faster split-based feedback network for image super-resolution

Although most of the existing image super-resolution(SR)methods have achieved superior performance,contrastive learning for high-level tasks has not been fully utilized in the existing image SR methods based on deep learning.This work focuses on two well-known strategies developed for lightweight and robust SR,i.e.,contrastive learning and feedback mechanism,and proposes an integrated solution called a split-based feedback network(SPFBN).The proposed SPFBN is based on a feedback mechanism to learn abstract representations and uses contrastive learning to explore high information in the representation space.Specifically,this work first uses hidden states and constraints in recurrent neural network(RNN)to implement a feedback mechanism.Then,use contrastive learning to perform representation learning to obtain high-level information by pushing the final image to the intermediate images and pulling the final SR image to the high-resolution image.Besides,a split-based feedback block(SPFB)is proposed to reduce model redundancy,which tolerates features with similar patterns but requires fewer parameters.Extensive experimental results demonstrate the superiority of the proposed method in comparison with the state-of-the-art methods.Moreover,this work extends the experiment to prove the effectiveness of this method and shows better overall reconstruction quality.

Deep-learning-based methods for super-resolution fluorescence microscopy

The algorithm used for reconstruction or resolution enhancement is one of the factors affectingthe quality of super-resolution images obtained by fluorescence microscopy.Deep-learning-basedalgorithms have achieved stateof-the-art performance in super-resolution fluorescence micros-copy and are becoming increasingly attractive.We firstly introduce commonly-used deep learningmodels,and then review the latest applications in terms of the net work architectures,the trainingdata and the loss functions.Additionally,we discuss the challenges and limits when using deeplearning to analyze the fluorescence microscopic data,and suggest ways to improve the reliability and robustness of deep learning applications.

Improved spatiotemporal resolution of anti-scattering super-resolution label-free microscopy via synthetic wave 3D metalens imaging

Super-resolution(SR)microscopy has dramatically enhanced our understanding of biological processes.However,scattering media in thick specimens severely limits the spatial resolution,often rendering the images unclear or indistinguishable.Additionally,live-cell imaging faces challenges in achieving high temporal resolution for fast-moving subcellular structures.Here,we present the principles of a synthetic wave microscopy(SWM)to extract three-dimensional information from thick unlabeled specimens,where photobleaching and phototoxicity are avoided.SWM exploits multiple-wave interferometry to reveal the specimen’s phase information in the area of interest,which is not affected by the scattering media in the optical path.SWM achieves~0.42λ/NA resolution at an imaging speed of up to 106 pixels/s.SWM proves better temporal resolution and sensitivity than the most conventional microscopes currently available while maintaining exceptional SR and anti-scattering capabilities.Penetrating through the scattering media is challenging for conventional imaging techniques.Remarkably,SWM retains its efficacy even in conditions of low signal-to-noise ratios.It facilitates the visualization of dynamic subcellular structures in live cells,encompassing tubular endoplasmic reticulum(ER),lipid droplets,mitochondria,and lysosomes.

SOFFLFM:Super-resolution optical fluctuation Fourierlight-field microscopy

Fourier light-field microscopy(FLFM)uses a microlens aray(MLA)to segment the Fourierplane of the microscopic objective lens to generate multiple two-dimensional perspective views,thereby reconstructing the threedimensional(3D)structure of the sample using 3D deconvo-lution calculation without scanning.However,the resolution of FLFM is stil limited by dif-fraction,and furthermore,it is dependent on the aperture division.In order to improve itsresolution,a super-resolution opticai fuctuation Fourier light-field microscopy(SOFFLFM)wasproposed here,in which the super-resolution optical fluctuation imaging(SOFI)with the abilityof super-resolution was introduced into FLFM.SOFFLFM uses higher-order cumulants statis-tical analysis on an image sequence collected by FLFM,and then carries out 3D deconvolutioncalculation to reconstruct the 3D structure of the sample.The theoretical basis of SOFFLFM onimproving resolution was explained and then verified with the simulations.Simulation resultsdemonstrated that SOFFLFM improved the lateral and axial resolution by more than V2 and 2times in the second-and fourth-order accumulations,compared with that of FLFM.

Spatiotemporal Isolation Based Super-Resolution Microscopy

Super-resolution microscopy(SRM)is a powerful imaging technique that overcomes the diffraction limit and allows imaging of the object structure in the nanoscale.However,SRM suffers from a tradeoff between spatial and temporal resolution,which prevents further exploration in scientific discoveries.In this review,we mainly focus on the development of improving spatiotemporal resolution of SRM,including 1)SRM based on physical and computational principles,2)physical and computational factors affecting SRM,from which we conclude some strategies for developing new types of SRM,3)the summary of the various types of SRM based on physical and computational principles,as well as,the analysis of the ordinary and developing SRM.Both SRMs based on physical principles and computational principles can be realized with spatial isolation and temporal isolation methods.We expect this review will offer some new ideas to improve the spatial and temporal resolution simultaneously,which may lead to more new discoveries in biology,chemistry,and materials science.

Rethinking resolution estimation in fluorescence microscopy: from theoretical resolution criteria to super-resolution microscopy

Resolution is undoubtedly the most important parameter in optical microscopy by providing an estimation on the maximum resolving power of a certain optical microscope. For centuries, the resolution of an optical microscope is generally considered to be limited only by the numerical aperture of the optical system and the wavelength of light. However, since the invention and popularity of various advanced fluorescence microscopy techniques, especially super-resolution fluorescence microscopy, many new methods have been proposed for estimating the resolution, leading to confusions for researchers who need to quantify the resolution of their fluorescence microscopes. In this paper, we firstly summarize the early concepts and criteria for predicting the resolution limit of an ideal optical system. Then, we discuss some important influence factors that deteriorate the resolution of a certain fluorescence microscope. Finally, we provide methods and examples on how to measure the resolution of a fluorescence microscope from captured fluorescence images. This paper aims to answer as best as possible the theoretical and practical issues regarding the resolution estimation in fluorescence microscopy.

Super-resolution microscopy reveals the distribution of targeted molecules in the complex matrix

With the support by the National Natural Science Foundation of China and the Chinese Academy of Sciences,the research team led by Prof.Li Junbai(李峻柏)at the CAS Key Lab of Colloid,Interface and Thermodynamics,Institute of Chemistry,Chinese Academy of Sciences,revealed the distribution of proteins in the transformation of inorganic/protein hybrid crystals by super-resolution microscopy,which

Hyperspectral Image Super-Resolution Meets Deep Learning:A Survey and Perspective

Hyperspectral image super-resolution,which refers to reconstructing the high-resolution hyperspectral image from the input low-resolution observation,aims to improve the spatial resolution of the hyperspectral image,which is beneficial for subsequent applications.The development of deep learning has promoted significant progress in hyperspectral image super-resolution,and the powerful expression capabilities of deep neural networks make the predicted results more reliable.Recently,several latest deep learning technologies have made the hyperspectral image super-resolution method explode.However,a comprehensive review and analysis of the latest deep learning methods from the hyperspectral image super-resolution perspective is absent.To this end,in this survey,we first introduce the concept of hyperspectral image super-resolution and classify the methods from the perspectives with or without auxiliary information.Then,we review the learning-based methods in three categories,including single hyperspectral image super-resolution,panchromatic-based hyperspectral image super-resolution,and multispectral-based hyperspectral image super-resolution.Subsequently,we summarize the commonly used hyperspectral dataset,and the evaluations for some representative methods in three categories are performed qualitatively and quantitatively.Moreover,we briefly introduce several typical applications of hyperspectral image super-resolution,including ground object classification,urban change detection,and ecosystem monitoring.Finally,we provide the conclusion and challenges in existing learning-based methods,looking forward to potential future research directions.

Contrastive Learning for Blind Super-Resolution via A Distortion-Specific Network

Previous deep learning-based super-resolution(SR)methods rely on the assumption that the degradation process is predefined(e.g.,bicubic downsampling).Thus,their performance would suffer from deterioration if the real degradation is not consistent with the assumption.To deal with real-world scenarios,existing blind SR methods are committed to estimating both the degradation and the super-resolved image with an extra loss or iterative scheme.However,degradation estimation that requires more computation would result in limited SR performance due to the accumulated estimation errors.In this paper,we propose a contrastive regularization built upon contrastive learning to exploit both the information of blurry images and clear images as negative and positive samples,respectively.Contrastive regularization ensures that the restored image is pulled closer to the clear image and pushed far away from the blurry image in the representation space.Furthermore,instead of estimating the degradation,we extract global statistical prior information to capture the character of the distortion.Considering the coupling between the degradation and the low-resolution image,we embed the global prior into the distortion-specific SR network to make our method adaptive to the changes of distortions.We term our distortion-specific network with contrastive regularization as CRDNet.The extensive experiments on synthetic and realworld scenes demonstrate that our lightweight CRDNet surpasses state-of-the-art blind super-resolution approaches.

Residual Feature Attentional Fusion Network for Lightweight Chest CT Image Super-Resolution

The diagnosis of COVID-19 requires chest computed tomography(CT).High-resolution CT images can provide more diagnostic information to help doctors better diagnose the disease,so it is of clinical importance to study super-resolution(SR)algorithms applied to CT images to improve the reso-lution of CT images.However,most of the existing SR algorithms are studied based on natural images,which are not suitable for medical images;and most of these algorithms improve the reconstruction quality by increasing the network depth,which is not suitable for machines with limited resources.To alleviate these issues,we propose a residual feature attentional fusion network for lightweight chest CT image super-resolution(RFAFN).Specifically,we design a contextual feature extraction block(CFEB)that can extract CT image features more efficiently and accurately than ordinary residual blocks.In addition,we propose a feature-weighted cascading strategy(FWCS)based on attentional feature fusion blocks(AFFB)to utilize the high-frequency detail information extracted by CFEB as much as possible via selectively fusing adjacent level feature information.Finally,we suggest a global hierarchical feature fusion strategy(GHFFS),which can utilize the hierarchical features more effectively than dense concatenation by progressively aggregating the feature information at various levels.Numerous experiments show that our method performs better than most of the state-of-the-art(SOTA)methods on the COVID-19 chest CT dataset.In detail,the peak signal-to-noise ratio(PSNR)is 0.11 dB and 0.47 dB higher on CTtest1 and CTtest2 at×3 SR compared to the suboptimal method,but the number of parameters and multi-adds are reduced by 22K and 0.43G,respectively.Our method can better recover chest CT image quality with fewer computational resources and effectively assist in COVID-19.

Application of super-resolution fluorescence microscopy in hematologic malignancies

Hematologic malignancies are one of the most common malignant tumors caused by the clonal proliferation and differentiation of hematopoietic and lymphoid stem cells.The examination of bone marrow cells combined with immunodeficiency typing is of great significance to the diagnostic type,treatment and prognosis of hematologic malignancies.Super-resolution fluorescence microscopy(SRM)is a special kind of optical microscopy technology,which breaks the resolution limit and was awarded the Nobel Prize in Chemistry in 2014.With the development of SRM,many related technologies have been applied to the diagnosis and treatment of clinical diseases.It was reported that a major type of SRM technique,single molecule localization microscopy(SMLM),is more sensitive than flow cytometry(FC)in detecting cell membrane antigens'expression,thus enabling better chances in detecting antigens on hematopoietic cells than traditional analytic tools.Furthermore,SRM may be applied to clinical pathology and may guide precision medicine and personalized medicine for clone hematopoietic cell diseases.In this paper,we mainly discuss the application of SRM in clone hematological malignancies.

Super-resolution imaging of low-contrast periodic nanoparticle arrays by microsphere-assisted microscopy

We use the label-free microsphere-assisted microscopy to image low-contrast hexagonally close-packed polystyrene nanoparticle arrays with diameters of 300 and 250 nm.When a nanoparticle array is directly placed on a glass slide,it cannot be distinguished.If a 30-nm-thick Ag film is deposited on the surface of a nanoparticle array,the nanoparticle array with nanoparticle diameters of 300 and 250 nm can be distinguished.In addition,the Talbot effect of the 300-nm-diameter nanoparticle array is also observed.If a nanoparticle sample is assembled on a glass slide deposited with a 30-nm-thick Ag film,an array of 300-nm-diameter nanoparticles can be discerned.We propose that in microsphere-assisted microscopy imaging,the resolution can be improved by the excitation of surface plasmon polaritons(SPPs) on the sample surface or at the sample/substrate interface,and a higher near-field intensity due to the excited SPPs would benefit the resolution improvement.Our study of label-free super-resolution imaging of low-contrast objects will promote the applications of microsphere-assisted microscopy in life sciences.

A Review of Clinical Applications for Super-resolution Ultrasound Localization Microscopy

Microvascular structure and hemodynamics are important indicators for the diagnosis and assessment of many diseases and pathologies.The structural and functional imaging of tissue microvasculature in vivo is a clinically significant objective for the development of many imaging modalities.Contrast-enhanced ultrasound(CEUS)is a popular clinical tool for characterizing tissue microvasculature,due to the moderate cost,wide accessibility,and absence of ionizing radiation of ultrasound.

Mirror-enhanced super-resolution microscopy

Axial excitation confinement beyond the diffraction limit is crucial to the development of next-generation,super-resolution microscopy.STimulated Emission Depletion(STED)nanoscopy offers lateral super-resolution using a donut-beam depletion,but its axial resolution is still over 500 nm.Total internal reflection fluorescence microscopy is widely used for single-molecule localization,but its ability to detect molecules is limited to within the evanescent field of~100 nm from the cell attachment surface.We find here that the axial thickness of the point spread function(PSF)during confocal excitation can be easily improved to 110 nm by replacing the microscopy slide with a mirror.The interference of the local electromagnetic field confined the confocal PSF to a 110-nm spot axially,which enables axial super-resolution with all laser-scanning microscopes.Axial sectioning can be obtained with wavelength modulation or by controlling the spacer between the mirror and the specimen.With no additional complexity,the mirror-assisted excitation confinement enhanced the axial resolution six-fold and the lateral resolution two-fold for STED,which together achieved 19-nm resolution to resolve the inner rim of a nuclear pore complex and to discriminate the contents of 120 nm viral filaments.The ability to increase the lateral resolution and decrease the thickness of an axial section using mirror-enhanced STED without increasing the laser power is of great importance for imaging biological specimens,which cannot tolerate high laser power.

Super-resolution parameter estimation of monopulse radar by wide-narrowband joint processing

The angular resolution of radar is of crucial signifi-cance to its tracking performance.In this paper,a super-resolu-tion parameter estimation algorithm based on wide-narrowband joint processing is proposed to improve the angular resolution of wideband monopulse radar.The range cells containing resolv-able scattering points are detected in the wideband mode,and these range cells are adopted to estimate part of the target parameters by algorithms of low computational requirement.Then,the likelihood function of the echo is constructed in the narrow-band mode to estimate the rest of the parameters,and the parameters estimated in the wideband mode are employed to reduce computation and enhance estimation accuracy.Simu-lation results demonstrate that the proposed algorithm has higher estimation accuracy and lower computational complexity than the current algorithm and can avoid the risk of model mis-match.

Accelerate Single Image Super-Resolution Using Object Detection Process

Image Super-Resolution(SR)research has achieved great success with powerful neural networks.The deeper networks with more parameters improve the restoration quality but add the computation complexity,which means more inference time would be cost,hindering image SR from practical usage.Noting the spatial distribution of the objects or things in images,a twostage local objects SR system is proposed,which consists of two modules,the object detection module and the SR module.Firstly,You Only Look Once(YOLO),which is efficient in generic object detection tasks,is selected to detect the input images for obtaining objects of interest,then put them into the SR module and output corresponding High-Resolution(HR)subimages.The computational power consumption of image SR is optimized by reducing the resolution of input images.In addition,we establish a dataset,TrafficSign500,for our experiment.Finally,the performance of the proposed system is evaluated under several State-Of-The-Art(SOTA)YOLOv5 and SISR models.Results show that our system can achieve a tremendous computation improvement in image SR.