Three-dimensional nanoscale reduced-angle ptycho-tomographic imaging with deep learning(RAPID)

X-ray ptychographic tomography is a nondestructive method for three dimensional(3D)imaging with nanometer-sized resolvable features.The size of the volume that can be imaged is almost arbitrary,limited only by the penetration depth and the available scanning time.Here we present a method that rapidly accelerates the imaging operation over a given volume through acquiring a limited set of data via large angular reduction and compensating for the resulting ill-posedness through deeply learned priors.The proposed 3D reconstruction method“RAPID”relies initially on a subset of the object measured with the nominal number of required illumination angles and treats the reconstructions from the conventional two-step approach as ground truth.It is then trained to reproduce equal fidelity from much fewer angles.After training,it performs with similar fidelity on the hitherto unexamined portions of the object,previously not shown during training,with a limited set of acquisitions.In our experimental demonstration,the nominal number of angles was 349 and the reduced number of angles was 21,resulting in a×140 aggregate speedup over a volume of 4.48×93.18×3.92μm^(3) and with(14 nm)^(3) feature size,i.e.-10^(8) voxels.RAPID’s key distinguishing feature over earlier attempts is the incorporation of atrous spatial pyramid pooling modules into the deep neural network framework in an anisotropic way.We found that adjusting the atrous rate improves reconstruction fidelity because it expands the convolutional kernels’range to match the physics of multi-slice ptychography without significantly increasing the number of parameters.

Attentional Ptycho-Tomography(APT)for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time

Noninvasive X-ray imaging of nanoscale three-dimensional objects,such as integrated circuits(ICs),generally requires two types of scanning:ptychographic,which is translational and returns estimates of the complex electromagnetic field through the IC;combined with a tomographic scan,which collects these complex field projections from multiple angles.Here,we present Attentional Ptycho-Tomography(APT),an approach to drastically reduce the amount of angular scanning,and thus the total acquisition time.APT is machine learning-based,utilizing axial self-Attention for Ptycho-Tomographic reconstruction.APT is trained to obtain accurate reconstructions of the ICs,despite the incompleteness of the measurements.The training process includes regularizing priors in the form of typical patterns found in IC interiors,and the physics of X-ray propagation through the IC.We show that APT with×12 reduced angles achieves fidelity comparable to the gold standard Simultaneous Algebraic Reconstruction Technique(SART)with the original set of angles.When using the same set of reduced angles,then APT also outperforms Filtered Back Projection(FBP),Simultaneous Iterative Reconstruction Technique(SIRT)and SART.The time needed to compute the reconstruction is also reduced,because the trained neural network is a forward operation,unlike the iterative nature of these alternatives.Our experiments show that,without loss in quality,for a 4.48×93.2×3.92µm^(3) IC(≃6×10^(8) voxels),APT reduces the total data acquisition and computation time from 67.96 h to 38 min.We expect our physics-assisted and attention-utilizing machine learning framework to be applicable to other branches of nanoscale imaging,including materials science and biological imaging.

Learning to synthesize:robust phase retrieval at low photon counts

The quality of inverse problem solutions obtained through deep learning is limited by the nature of the priors learned from examples presented during the training phase.Particularly in the case of quantitative phase retrieval,spatial frequencies that are underrepresented in the training database,most often at the high band,tend to be suppressed in the reconstruction.Ad hoc solutions have been proposed,such as pre-amplifying the high spatial frequencies in the examples;however,while that strategy improves the resolution,it also leads to high-frequency artefacts,as well as low-frequency distortions in the reconstructions.Here,we present a new approach that learns separately how to handle the two frequency bands,low and high,and learns how to synthesize these two bands into full-band reconstructions.We show that this“learning to synthesize”(LS)method yields phase reconstructions of high spatial resolution and without artefacts and that it is resilient to high-noise conditions,e.g.,in the case of very low photon flux.In addition to the problem of quantitative phase retrieval,the LS method is applicable,in principle,to any inverse problem where the forward operator treats different frequency bands unevenly,i.e.,is ill-posed.

Dynamical machine learning volumetric reconstruction of objects’ interiors from limited angular views

Limited-angle tomography of an interior volume is a challenging, highly ill-posed problem with practical implications in medical and biological imaging, manufacturing, automation, and environmental and food security. Regularizing priors are necessary to reduce artifacts by improving the condition of such problems. Recently, it was shown that one effective way to learn the priors for strongly scattering yet highly structured 3D objects, e.g. layered and Manhattan, is by a static neural network [Goy et al. Proc. Natl. Acad. Sci. 116, 19848–19856 (2019)]. Here, we present a radically different approach where the collection of raw images from multiple angles is viewed analogously to a dynamical system driven by the object-dependent forward scattering operator. The sequence index in the angle of illumination plays the role of discrete time in the dynamical system analogy. Thus, the imaging problem turns into a problem of nonlinear system identification, which also suggests dynamical learning as a better fit to regularize the reconstructions. We devised a Recurrent Neural Network (RNN) architecture with a novel Separable-Convolution Gated Recurrent Unit (SC-GRU) as the fundamental building block. Through a comprehensive comparison of several quantitative metrics, we show that the dynamic method is suitable for a generic interior-volumetric reconstruction under a limited-angle scheme. We show that this approach accurately reconstructs volume interiors under two conditions: weak scattering, when the Radon transform approximation is applicable and the forward operator well defined;and strong scattering, which is nonlinear with respect to the 3D refractive index distribution and includes uncertainty in the forward operator.