Machine learning for automated experimentation in scanning transmission electron microscopy

Machine learning(ML)has become critical for post-acquisition data analysis in(scanning)transmission electron microscopy,(S)TEM,imaging and spectroscopy.An emerging trend is the transition to real-time analysis and closed-loop microscope operation.The effective use of ML in electron microscopy now requires the development of strategies for microscopy-centric experiment workflow design and optimization.Here,we discuss the associated challenges with the transition to active ML,including sequential data analysis and out-of-distribution drift effects,the requirements for edge operation,local and cloud data storage,and theory in the loop operations.Specifically,we discuss the relative contributions of human scientists and ML agents in the ideation,orchestration,and execution of experimental workflows,as well as the need to develop universal hyper languages that can apply across multiple platforms.These considerations will collectively inform the operationalization of ML in next-generation experimentation.

Ensemble learning-iterative training machine learning for uncertainty quantification and automated experiment in atom-resolved microscopy

Deep learning has emerged as a technique of choice for rapid feature extraction across imaging disciplines,allowing rapid conversion of the data streams to spatial or spatiotemporal arrays of features of interest.However,applications of deep learning in experimental domains are often limited by the out-of-distribution drift between the experiments,where the network trained for one set of imaging conditions becomes sub-optimal for different ones.This limitation is particularly stringent in the quest to have an automated experiment setting,where retraining or transfer learning becomes impractical due to the need for human intervention and associated latencies.Here we explore the reproducibility of deep learning for feature extraction in atom-resolved electron microscopy and introduce workflows based on ensemble learning and iterative training to greatly improve feature detection.This approach allows incorporating uncertainty quantification into the deep learning analysis and also enables rapid automated experimental workflows where retraining of the network to compensate for out-of-distribution drift due to subtle change in imaging conditions is substituted for human operator or programmatic selection of networks from the ensemble.This methodology can be further applied to machine learning workflows in other imaging areas including optical and chemical imaging.

Deep learning ferroelectric polarization distributions from STEM data via with and without atom finding

Over the last decade,scanning transmission electron microscopy(STEM)has emerged as a powerful tool for probing atomic structures of complex materials with picometer precision,opening the pathway toward exploring ferroelectric,ferroelastic,and chemical phenomena on the atomic scale.Analyses to date extracting a polarization signal from lattice coupled distortions in STEM imaging rely on discovery of atomic positions from intensity maxima/minima and subsequent calculation of polarization and other order parameter fields from the atomic displacements.Here,we explore the feasibility of polarization mapping directly from the analysis of STEM images using deep convolutional neural networks(DCNNs).In this approach,the DCNN is trained on the labeled part of the image(i.e.,for human labelling),and the trained network is subsequently applied to other images.We explore the effects of the choice of the descriptors(centered on atomic columns and grid-based),the effects of observational bias,and whether the network trained on one composition can be applied to a different one.This analysis demonstrates the tremendous potential of the DCNN for the analysis of high-resolution STEM imaging and spectral data and highlights the associated limitations.

Bridging microscopy with molecular dynamics and quantum simulations: an atomAI based pipeline

Recent advances in (scanning) transmission electron microscopy have enabled a routine generation of large volumes of high-veracity structural data on 2D and 3D materials,naturally offering the challenge of using these as starting inputs for atomistic simulations.In this fashion,the theory will address experimentally emerging structures,as opposed to the full range of theoretically possible atomic configurations.However,this challenge is highly nontrivial due to the extreme disparity between intrinsic timescales accessible to modern simulations and microscopy,as well as latencies of microscopy and simulations per se.Addressing this issue requires as a first step bridging the instrumental data flow and physics-based simulation environment,to enable the selection of regions of interest and exploring them using physical simulations.Here we report the development of the machine learning workflow that directly bridges the instrument data stream into Python-based molecular dynamics and density functional theory environments using pre-trained neural networks to convert imaging data to physical descriptors.The pathways to ensure structural stability and compensate for the observational biases universally present in the data are identified in the workflow.This approach is used for a graphene system to reconstruct optimized geometry and simulate temperature-dependent dynamics including adsorption of Cr as an ad-atom and graphene healing effects.However,it is universal and can be used for other material systems.

Anion order in oxysulfide perovskites:origins and implications

Heteroanionic oxysulfide perovskite compounds represent an emerging class of new materials allowing for a wide range of tunability in the electronic structure that could lead to a diverse spectrum of novel and improved functionalities.Unlike cation ordered double perovskites—where the origins and design rules of various experimentally observed cation orderings are well known and understood—anion ordering in heteroanionic perovskites remains a largely uncharted territory.