Multi-prior physics-enhanced neural network enables pixel super-resolution and twin-imagefree phase retrieval from single-shot hologram

Digital in-line holographic microscopy(DIHM)is a widely used interference technique for real-time reconstruction of living cells’morphological information with large space-bandwidth product and compact setup.However,the need for a larger pixel size of detector to improve imaging photosensitivity,field-of-view,and signal-to-noise ratio often leads to the loss of sub-pixel information and limited pixel resolution.Additionally,the twin-image appearing in the reconstruction severely degrades the quality of the reconstructed image.The deep learning(DL)approach has emerged as a powerful tool for phase retrieval in DIHM,effectively addressing these challenges.However,most DL-based strategies are datadriven or end-to-end net approaches,suffering from excessive data dependency and limited generalization ability.Herein,a novel multi-prior physics-enhanced neural network with pixel super-resolution(MPPN-PSR)for phase retrieval of DIHM is proposed.It encapsulates the physical model prior,sparsity prior and deep image prior in an untrained deep neural network.The effectiveness and feasibility of MPPN-PSR are demonstrated by comparing it with other traditional and learning-based phase retrieval methods.With the capabilities of pixel super-resolution,twin-image elimination and high-throughput jointly from a single-shot intensity measurement,the proposed DIHM approach is expected to be widely adopted in biomedical workflow and industrial measurement.

Deep-learning-based methods for super-resolution fluorescence microscopy

The algorithm used for reconstruction or resolution enhancement is one of the factors affectingthe quality of super-resolution images obtained by fluorescence microscopy.Deep-learning-basedalgorithms have achieved stateof-the-art performance in super-resolution fluorescence micros-copy and are becoming increasingly attractive.We firstly introduce commonly-used deep learningmodels,and then review the latest applications in terms of the net work architectures,the trainingdata and the loss functions.Additionally,we discuss the challenges and limits when using deeplearning to analyze the fluorescence microscopic data,and suggest ways to improve the reliability and robustness of deep learning applications.

Spectrum-optimized direct image reconstruction of super-resolution structured illumination microscopy

Super-resolution structured illumination microscopy(SR-SIM)has become a widely used nanoscopy technique for rapid,long-term,and multi-color imaging of live cells.Precise but troublesome determination of the illumination pattern parameters is a prerequisite for Wiener-deconvolution-based SR-SIM image reconstruction.Here,we present a direct reconstruction SIM algorithm(direct-SIM)with an initial spatial-domain reconstruction followed by frequency-domain spectrum optimization.Without any prior knowledge of illumination patterns and bypassing the artifact-sensitive Wiener deconvolution procedures,resolution-doubled SR images could be reconstructed by direct-SIM free of common artifacts,even for the raw images with large pattern variance in the field of view(FOV).Direct-SIM can be applied to previously difficult scenarios such as very sparse samples,periodic samples,very small FOV imaging,and stitched large FOV imaging.

STED microscopy based on axially symmetric polarized vortex beams

A stimulated emission depletion （STED） microscopy scheme using axially symmetric polarized vortex beams is pro- posed based on unique focusing properties of such kinds of beams. The concept of axially symmetric polarized vortex beams is first introduced, and the basic principle about the scheme is described. Simulation results for several typical beams are then shown, including radially polarized vortex beams, azimuthally polarized vortex beams, and high-order axi- ally symmetric polarized vortex beams. The results indicate that sharper doughnut spots and thus higher resolutions can be achieved, showing more flexibility than previous schemes based on flexible modulation of both phase and polarization for incident beams.

Fusion of clathrin and caveolae endocytic vesicles revealed by line-switching dual-color STED microscopy

Clathrin-and caveolae-mediated endocytosis are the most commonly used pathways for the internalization of cell membrane receptors.However,due to their dimensions are within the diffraction limit,traditional fluorescence microscopy cannot distinguish them and little is known about their interactions underneath cell membrane.In this study,we proposed the line-switching scanning imaging mode for dual-color triplet-state relaxation(T-Rex)stimulated emission depletion(STED)super-resolution microscopy.With this line-switching mode,the cross-talk between the two channels,the side effects from pulse picker and image drift in frame scanning mode can be effectively eliminated.The dual-color super-resolution imaging results in mixed fluorescent beads validated the excellent performance.With this super-resolution microscope,not only the ring-shaped structure of clathrin and caveolae endocytic vesicles,but also their semi-fused structures underneath the cell membrane were distinguished clearly.The resultant infor-mation will greatly facilitate the study of clathrin-and caveolae-mediated receptor endocytosis and signaling process and also our home-built dual-color T-Rex STED microscope with this line-switching imaging mode provides a precise and convenient way to study subcellular-scale protein interactions.

Method of lateral image reconstruction in structured illumination microscopy with super resolution

The image reconstruction process in super-resolution structured illumination microscopy(SIM)is investigated.The structured pattern is generated by the interference of two Gaussian beams to encode undetectable spectra into detectable region of microscope.After parameters estimation of the structured pattern,the encoded spectra are computationally decoded and recombined in Fourier domain to equivalently increase the cut-off frequency of microscope,resulting in the extension of detectable spectra and a reconstructed image with about two-fold enhanced resolution.Three di®erent methods to estimate the initial phase of structured pattern are compared,verifying the auto-correlation algorithm a®ords the fast,most precise and robust measurement.The artifacts sources and detailed reconstruction°owchart for both linear and nonlinear SIM are also presented.

A Perspective on Data Processing in Super-resolution Fluorescence Microscopy Imaging

With super-resolution microscopy,we attempt to visualize(biological)structures and processes at the sub-cellular level(i.e.,nanoscale).To obtain this information,the samples are labeled with fluorophores that have a stochastic on/off switching of their emissions,which help to overcome the optical diffraction limit of around 250 nm,related to the use of optical micro-scopes.However,nowadays,research focuses on the imaging of live cells and thicker samples.These investigations require a high amount of simultaneously active fluorophores(i.e.,high-density imaging)and are challenging due to the collapse of the single-molecule localization techniques and the increased background in the image.Therefore,recent efforts have shifted towards the development of new ways to process the data.This publication gives an introduction to wide-field super-resolution fluorescence microscopy,explaining the concepts of the technique,and then gives an overview of the recently developed methods to provide super-resolution images for high-density data of live cells and ways to overcome the issues related to the imaging of these samples.

Super-resolution imaging reveals the subcellular distribution of dextran at the nanoscale in living cells

Theranostic visualization of dextran at the nanoscale is beneficial for understanding the bioregulatory mechanisms of this molecule. In this study, we applied structured illumination microscopy(SIM) to capture the distribution of Cy5-Dextran at different incubation periods in living cells. The results showed that Cy5-Dextran could be absorbed by He La cells. In addition, we clarified that Cy5-Dextran exhibited differential organelle distribution(lysosomal or mitochondrial) in a time-dependent manner. Moreover,lysosomal Cy5-Dextran localization was found to be independent of the autophagy process, while Cy5-Dextran localized to the mitochondria triggered a pro-apoptotic event, upregulating the levels of reactive oxygen species(ROS) to accelerate mitochondrial fragmentation. This work uses a visualized strategy to reveal the anti-tumor bioactivity of dextran, which was achieved by regulating apoptosis and autophagy.

Nonlinear scanning structured illumination microscopy based on nonsinusoidal modulation

Structured illumination microscopy(SIM)is an essential super-resolution microscopy technique that enhances resolution.Several images are required to reconstruct a super-resolution image.However,linear SIM resolution enhancement can only increase the spatial resolution of micros-copy by a factor of two at most because the frequency of the structured illumination pattern is limited by the cutoff frequency of the excitation point spread function.The frequency of the pattern generated by the nonlinear response in samples is not limited;therefore,nonlinear SIM(NL-SIM),in theory,has no inherent limit to the resolution.In the present study,we describe a two-photon nonlinear SIM(2P-SIM)technique using a multiple harmonics scanning pattern that employs a composite structured illumination pattern,which can produce a higher order harmonic pattern based on the fluorescence nonlinear response in a 2P process.The theoretical models of super-resolution imaging were established through our simulation,which describes the working mechanism of the multi-frequency structure of the nonsinusoidal function to improve the reso-lution.The simulation results predict that a 5-fold improvement in resolution in the 2P-SIM is possible.

3D fluorescence emission difference microscopy based on spatial light modulator

We report three-dimensional fluorescence emission difference(3D-FED)microscopy using a spatial light modulator(SLM).Zero phase,0–2
vortex phase and binary 0-pi phase are loaded on the SLM to generate the corresponding solid,doughnut and z-axis hollow excitation spot,respectively.Our technique achieves super-resolved image by subtracting three di®erently acquired images with proper subtractive factors.Detailed theoretical analysis and simulation tests are proceeded to testify the performance of 3D-FED.Also,the improvement of lateral and axial resolution is demonstrated by imaging 100 nm°uorescent beads.The experiment yields lateral resolution of 140 nm and axial resolution of approximate 380 nm.

Enhancing image resolution of confocal fluorescence microscopy with deep learning

Super-resolution optical imaging is crucial to the study of cellular processes.Current super-resolution fluorescence microscopy is restricted by the need of special fluorophores or sophisticated optical systems,or long acquisition and computational times.In this work,we present a deep-learning-based super-resolution technique of confocal microscopy.We devise a two-channel attention network(TCAN),which takes advantage of both spatial representations and frequency contents to learn a more precise mapping from low-resolution images to high-resolution ones.This scheme is robust against changes in the pixel size and the imaging setup,enabling the optimal model to generalize to different fluorescence microscopy modalities unseen in the training set.Our algorithm is validated on diverse biological structures and dual-color confocal images of actin-microtubules,improving the resolution from~230 nm to~110 nm.Last but not least,we demonstrate live-cell super-resolution imaging by revealing the detailed structures and dynamic instability of microtubules.

Nucleic Acid Probes for Single-Molecule Localization Imaging of Cellular Biomolecules

Endogenous biomolecules in cells are the basis of all life activities.Directly visualizing the structural characteristics and dynamic behaviors of cellular biomolecules is signiffcant for understanding the molecular mechanisms in various biological processes.Singlemolecule localization microscopy(SMLM)can circumvent the optical diffraction limit,achieving analysis of the ffne structures and biological processes in living cells with nanoscale resolution.However,the large size of traditional imaging probes prevents SMLM from accurately locating ffne structures and densely distributed biomolecules within cells.In recent years,nucleic acid probes have emerged as potential tools to replace conventional SMLM probes by virtue of their small size and high speciffcity.In addition,due to their programmability,nucleic acid probes with different conformations can be constructed via sequence design,further extending the application of SMLM in bioanalysis.Here,we discuss the design concepts of different conformational nucleic acid probes for SMLM and summarize the application of SMLM based on nucleic acid probes in the ffeld of biomolecules.Furthermore,we provide a summary and future perspectives of the nucleic acid probe-based SMLM technology,aiming to provide guidance for the acquisition of nanoscale information about cellular biological processes.

SUPPRESSION OF SIDELOBES IN SINGLE-PHOTON 4PI CONFOCAL MICROSCOPY BY FORSTER RESONANT ENERGY TRANSFER

Recently,we theoretically demonstrate that utilization of silica nanobeads co-doped with Cy3 and Cy5 molecules instead of single dye molecules asfluorescent labels can enable optical resolutions far beyond the diffraction-limit.Here,we show that by combining the 4Pi microscopy and the novelfluorescent label,it is possible to completely suppress the sidelobes in 4Pi focal spot and significantly enhance the optical resolution in the axial direction.

Fluorescence interference structured illumination microscopy for 3D morphology imaging with high axial resolution

Imaging three-dimensional,subcellular structures with high axial resolution has always been the core purpose of fluorescence microscopy.However,trade-offs exist between axial resolution and other important technical indicators,such as temporal resolution,optical power density,and imaging process complexity.We report a new imaging modality,fluorescence interference structured illumination microscopy(FI-SIM),which is based on three-dimensional structured illumination microscopy for wide-field lateral imaging and fluorescence interference for axial reconstruction.FI-SIM can acquire images quickly within the order of hundreds of milliseconds and exhibit even 30 nm axial resolution in half the wavelength depth range without z-axis scanning.Moreover,the relatively low laser power density relaxes the requirements for dyes and enables a wide range of applications for observing fixed and live subcellular structures.

Optical super-resolution microscopy and its applications in nano-catalysis

The resolution of conventional optical microscopy is only -200 nm, which is becoming less and less sufficient for a variety of applications. In order to surpass the diffraction limited resolution, super-resolution microscopy （SRM） has been developed to achieve a high resolution of one to tens of nanometers. The techniques involved in SRM can be assigned into two broad categories, namely ＂true＂ super-resolution techniques and ＂functional＂ super-resolution techniques. In ＂functional＂ super-resolution techniques, stochastic super-resolution microscopy （SSRM） is widely used due to its low expense, simple operation, and high resolution. The principle process in SSRM is to accumulate the coordinates of many diffraction-limited emitters （e.g., single fluorescent molecules） on the object by localizing the centroids of the point spread functions （PSF）, and then reconstruct the image of the object using these coordinates. When the diffraction-limited emitters take part in a catalytic reaction, the activity distribution and kinetic information about the catalysis by nanoparticles can be obtained by SSRM. SSRM has been applied and exhibited outstanding advantages in several fields of catalysis, such as metal nanoparticle catalysis, molecular sieve catalysis, and photocatalysis. Since SSRM is able to resolve the catalytic activity within one nanoparticle, it promises to accelerate the development and discovery of new and better catalysts. This review will present a brief introduction to SRM, and a detailed description of SSRM and its applications in nano-catalysis.

Rethinking resolution estimation in fluorescence microscopy: from theoretical resolution criteria to super-resolution microscopy

Resolution is undoubtedly the most important parameter in optical microscopy by providing an estimation on the maximum resolving power of a certain optical microscope. For centuries, the resolution of an optical microscope is generally considered to be limited only by the numerical aperture of the optical system and the wavelength of light. However, since the invention and popularity of various advanced fluorescence microscopy techniques, especially super-resolution fluorescence microscopy, many new methods have been proposed for estimating the resolution, leading to confusions for researchers who need to quantify the resolution of their fluorescence microscopes. In this paper, we firstly summarize the early concepts and criteria for predicting the resolution limit of an ideal optical system. Then, we discuss some important influence factors that deteriorate the resolution of a certain fluorescence microscope. Finally, we provide methods and examples on how to measure the resolution of a fluorescence microscope from captured fluorescence images. This paper aims to answer as best as possible the theoretical and practical issues regarding the resolution estimation in fluorescence microscopy.

Temporal compressive super-resolution microscopy at frame rate of 1200 frames per second and spatial resolution of 100 nm

Various super-resolution microscopy techniques have been presented to explore fine structures of biological specimens.However,the super-resolution capability is often achieved at the expense of reducing imaging speed by either point scanning or multiframe computation.The contradiction between spatial resolution and imaging speed seriously hampers the observation of high-speed dynamics of fine structures.To overcome this contradiction,here we propose and demonstrate a temporal compressive super-resolution microscopy(TCSRM)technique.This technique is to merge an enhanced temporal compressive microscopy and a deep-learning-based super-resolution image reconstruction,where the enhanced temporal compressive microscopy is utilized to improve the imaging speed,and the deep-learning-based super-resolution image reconstruction is used to realize the resolution enhancement.The high-speed super-resolution imaging ability of TCSRM with a frame rate of 1200 frames per second(fps)and spatial resolution of 100 nm is experimentally demonstrated by capturing the flowing fluorescent beads in microfluidic chip.Given the outstanding imaging performance with high-speed super-resolution,TCSRM provides a desired tool for the studies of high-speed dynamical behaviors in fine structures,especially in the biomedical field.

Superresolution imaging of telomeres with continuous wave stimulated emission depletion (STED) microscope

The significant role of telomeres in cells has attracted much attention since they were discovered.Fluorescence imaging is an effective method to study subcellular structures like telomeres.However,the diffraction limit of traditional optical microscope hampers further investigation on them.Recent progress on superresolution fluorescence microscopy has broken this limit.In this work,we used stimulated emission depletion(STED) microscope to observe fluorescence-labeled telomeres in interphase cell nuclei.The results showed that the size of fluorescent puncta representing telomeres under the STED microscope was much smaller than that under the confocal microscope.Two adjacent telomeres were clearly separated via STED imaging,which could hardly be discriminated by confocal microscopy due to the diffraction limit.We conclude that STED microscope is a more powerful tool that enable us to obtain detailed information about telomeres.

Super-resolution visible photoactivated atomic force microscopy

Imaging the intrinsic optical absorption properties of nanomaterials with optical microscopy(OM)is hindered by the optical diffraction limit and intrinsically poor sensitivity.Thus,expensive and destructive electron microscopy(EM)has been commonly used to examine the morphologies of nanostructures.Further,while nanoscale fluorescence OM has become crucial for investigating the morphologies and functions of intracellular specimens,this modality is not suitable for imaging optical absorption and requires the use of possibly undesirable exogenous fluorescent molecules for biological samples.Here we demonstrate super-resolution visible photoactivated atomic force microscopy(pAFM),which can sense intrinsic optical absorption with~8 nm resolution.Thus,the resolution can be improved down to~8 nm.This system can detect not only the first harmonic response,but also the higher harmonic response using the nonlinear effect.The thermoelastic effects induced by pulsed laser irradiation allow us to obtain visible pAFM images of single gold nanospheres,various nanowires,and biological cells,all with nanoscale resolution.Unlike expensive EM,the visible pAFM system can be simply implemented by adding an optical excitation sub-system to a commercial atomic force microscope.

Distinguishing Single-Metal Nanoparticles with Subdiffraction Spatial Resolution Using Variable-Polarization Fourier Transform Nonlinear Optical Microscopy

The development and use of interferometric variablepolarization Fourier transform nonlinear optical(vpFT-NLO)imaging to distinguish colloidal nanoparticles colocated within the optical diffraction limit is described.Using a collinear train of phase-stabilized pulse pairs with orthogonal electric ffeld vectors,the polarization of nonlinear excitation ffelds are controllably modulated between linear,circular,and various elliptical states.Polarization modulation is achieved by precise control over the time delay separating the orthogonal pulse pairs to within hundreds of attoseconds.The resultant emission from gold nanorods is imaged to a 2D array detector and correlated to the excitation ffeld polarization and plasmon resonance frequency by Fourier transformation.Gold nanorods with length-to-diameter aspect ratios of 2 support a longitudinal surface plasmon resonance at approximately 800 nm,which is resonant with the excitation fundamental carrier wavelength.Differences in the intrinsic linear and circular dichroism resulting from variation in their relative alignment with respect to the laboratory frame enable optical differentiation of nanorods separated within 50 nm,which is an approximate 5-fold improvement over the diffraction limit of the microscope.The experimental results are supported by analytical simulations.In addition to subdiffraction spatial resolution,the vpFT-NLO method intrinsically provides the polarization-and frequency-dependent resonance response of the nanoparticles�providing spectroscopic information content along with super-resolution imaging capabilities.