Pluggable multitask diffractive neural networks based on cascaded metasurfaces

Optical neural networks have significant advantages in terms of power consumption,parallelism,and high computing speed,which has intrigued extensive attention in both academic and engineering communities.It has been considered as one of the powerful tools in promoting the fields of imaging processing and object recognition.However,the existing optical system architecture cannot be reconstructed to the realization of multi-functional artificial intelligence systems simultaneously.To push the development of this issue,we propose the pluggable diffractive neural networks(P-DNN),a general paradigm resorting to the cascaded metasurfaces,which can be applied to recognize various tasks by switching internal plug-ins.As the proof-of-principle,the recognition functions of six types of handwritten digits and six types of fashions are numerical simulated and experimental demonstrated at near-infrared regimes.Encouragingly,the proposed paradigm not only improves the flexibility of the optical neural networks but paves the new route for achieving high-speed,low-power and versatile artificial intelligence systems.

Electrically programmable phase-change photonic memory for optical neural networks with nanoseconds in situ training capability

Optical neural networks (ONNs), enabling low latency and high parallel data processing withoutelectromagnetic interference, have become a viable player for fast and energy-efficient processing andcalculation to meet the increasing demand for hash rate. Photonic memories employing nonvolatile phase-change materials could achieve zero static power consumption, low thermal cross talk, large-scale, andhigh-energy-efficient photonic neural networks. Nevertheless, the switching speed and dynamic energyconsumption of phase-change material-based photonic memories make them inapplicable for in situ training.Here, by integrating a patch of phase change thin film with a PIN-diode-embedded microring resonator,a bifunctional photonic memory enabling both 5-bit storage and nanoseconds volatile modulation wasdemonstrated. For the first time, a concept is presented for electrically programmable phase-changematerial-driven photonic memory integrated with nanosecond modulation to allow fast in situ training and zerostatic power consumption data processing in ONNs. ONNs with an optical convolution kernel constructedby our photonic memory theoretically achieved an accuracy of predictions higher than 95% when testedby the MNIST handwritten digit database. This provides a feasible solution to constructing large-scalenonvolatile ONNs with high-speed in situ training capability.

Class-specific differential detection in diffractive optical neural networks improves inference accuracy

Optical computing provides unique opportunities in terms of parallelization,scalability,power efficiency,and computational speed and has attracted major interest for machine learning.Diffractive deep neural networks have been introduced earlier as an optical machine learning framework that uses task-specific diffractive surfaces designed by deep learning to all-optically perform inference,achieving promising performance for object classification and imaging.We demonstrate systematic improvements in diffractive optical neural networks,based on a differential measurement technique that mitigates the strict nonnegativity constraint of light intensity.In this differential detection scheme,each class is assigned to a separate pair of detectors,behind a diffractive optical network,and the class inference is made by maximizing the normalized signal difference between the photodetector pairs.Using this differential detection scheme,involving 10 photodetector pairs behind 5 diffractive layers with a total of 0.2 million neurons,we numerically achieved blind testing accuracies of 98.54%,90.54%,and 48.51%for MNIST,Fashion-MNIST,and grayscale CIFAR-10 datasets,respectively.Moreover,by utilizing the inherent parallelization capability of optical systems,we reduced the cross-talk and optical signal coupling between the positive and negative detectors of each class by dividing the optical path into two jointly trained diffractive neural networks that work in parallel.We further made use of this parallelization approach and divided individual classes in a target dataset among multiple jointly trained diffractive neural networks.Using this class-specific differential detection in jointly optimized diffractive neural networks that operate in parallel,our simulations achieved blind testing accuracies of 98.52%,91.48%,and 50.82%for MNIST,Fashion-MNIST,and grayscale CIFAR-10 datasets,respectively,coming close to the performance of some of the earlier generations of all-electronic deep neural networks,e.g.,LeNet,which achieves classification accuracies of 98.77%,90.27%,and 55.21%corresponding to the same datasets,respectively.In addition to these jointly optimized diffractive neural networks,we also independently optimized multiple diffractive networks and utilized them in a way that is similar to ensemble methods practiced in machine learning;using 3 independently optimized differential diffractive neural networks that optically project their light onto a common output/detector plane,we numerically achieved blind testing accuracies of 98.59%,91.06%,and 51.44%for MNIST,Fashion-MNIST,and grayscale CIFAR-10 datasets,respectively.Through these systematic advances in designing diffractive neural networks,the reported classification accuracies set the state of the art for all-optical neural network design.The presented framework might be useful to bring optical neural network-based low power solutions for various machine learning applications and help us design new computational cameras that are task-specific.

Low-depth optical neural networks

Optical neural network(ONNs)are emerging as attractive propos-als for machine-learning applications.However,the stability of ONNs decreases with the circuit depth,limiting the scalability of ONNs for practical uses.Here we demonstrate how to compress the circuit depth to scale only logarithmically in terms of the dimension of the data,leading to an exponential gain in terms of noise robustness.Our low-depth(LD)-ONN is based on an architecture,called Optical Com-puTing Of dot-Product UnitS(OCTOPUS),which can also be applied individually as a linear perceptron for solving classification problems.We present both numerical and theoretical evidence showing that LD-ONN can exhibit a significant improvement on robustness,compared with previous ONN proposals based on singular-value decomposition.

Multimode diffractive optical neural network

On-chip diffractive optical neural networks(DONNs)bring the advantages of parallel processing and low energy consumption.However,an accurate representation of the optical field’s evolution in the structure cannot be provided using the previous diffraction-based analysis method.Moreover,the loss caused by the open boundaries poses challenges to applications.A multimode DONN architecture based on a more precise eigenmode analysis method is proposed.We have constructed a universal library of input,output,and metaline structures utilizing this method,and realized a multimode DONN composed of the structures from the library.On the designed multimode DONNs with only one layer of the metaline,the classification task of an Iris plants dataset is verified with an accuracy of 90%on the blind test dataset,and the performance of the one-bit binary adder task is also validated.Compared to the previous architectures,the multimode DONN exhibits a more compact design and higher energy efficiency.

Research progress in optical neural networks:theory,applications and developments

With the advent of the era of big data,artificial intelligence has attracted continuous attention from all walks of life,and has been widely used in medical image analysis,molecular and material science,language recognition and other fields.As the basis of artificial intelligence,the research results of neural network are remarkable.However,due to the inherent defect that electrical signal is easily interfered and the processing speed is proportional to the energy loss,researchers have turned their attention to light,trying to build neural networks in the field of optics,making full use of the parallel processing ability of light to solve the problems of electronic neural networks.After continuous research and development,optical neural network has become the forefront of the world.Here,we mainly introduce the development of this field,summarize and compare some classical researches and algorithm theories,and look forward to the future of optical neural network.

Diffractive Deep Neural Networks at Visible Wavelengths

Optical deep learning based on diffractive optical elements offers unique advantages for parallel processing,computational speed,and power efficiency.One landmark method is the diffractive deep neural network(D^(2) NN)based on three-dimensional printing technology operated in the terahertz spectral range.Since the terahertz bandwidth involves limited interparticle coupling and material losses,this paper extends D^(2) NN to visible wavelengths.A general theory including a revised formula is proposed to solve any contradictions between wavelength,neuron size,and fabrication limitations.A novel visible light D^(2) NN classifier is used to recognize unchanged targets(handwritten digits ranging from 0 to 9)and targets that have been changed(i.e.,targets that have been covered or altered)at a visible wavelength of 632.8 nm.The obtained experimental classification accuracy(84%)and numerical classification accuracy(91.57%)quantify the match between the theoretical design and fabricated system performance.The presented framework can be used to apply a D^(2) NN to various practical applications and design other new applications.

A review:Photonics devices,architectures,and algorithms for optical neural computing

The explosive growth of data and information has motivated various emerging non-von Neumann computational approaches in the More-than-Moore era.Photonics neuromorphic computing has attracted lots of attention due to the fascinating advantages such as high speed,wide bandwidth,and massive parallelism.Here,we offer a review on the optical neural computing in our research groups at the device and system levels.The photonics neuron and photonics synapse plasticity are presented.In addition,we introduce several optical neural computing architectures and algorithms including photonic spiking neural network,photonic convolutional neural network,photonic matrix computation,photonic reservoir computing,and photonic reinforcement learning.Finally,we summarize the major challenges faced by photonic neuromorphic computing,and propose promising solutions and perspectives.

Chip-Based High-Dimensional Optical Neural Network

Parallel multi-thread processing in advanced intelligent processors is the core to realize high-speed and high-capacity signal processing systems.Optical neural network(ONN)has the native advantages of high parallelization,large bandwidth,and low power consumption to meet the demand of big data.Here,we demonstrate the dual-layer ONN with Mach-Zehnder interferometer(MZI)network and nonlinear layer,while the nonlinear activation function is achieved by optical-electronic signal conversion.Two frequency components from the microcomb source carrying digit datasets are simultaneously imposed and intelligently recognized through the ONN.We successfully achieve the digit classification of different frequency components by demultiplexing the output signal and testing power distribution.Efficient parallelization feasibility with wavelength division multiplexing is demonstrated in our high-dimensional ONN.This work provides a high-performance architecture for future parallel high-capacity optical analog computing.

Efficient stochastic parallel gradient descent training for on-chip optical processor

In recent years,space-division multiplexing(SDM)technology,which involves transmitting data information on multiple parallel channels for efficient capacity scaling,has been widely used in fiber and free-space optical communication sys-tems.To enable flexible data management and cope with the mixing between different channels,the integrated reconfig-urable optical processor is used for optical switching and mitigating the channel crosstalk.However,efficient online train-ing becomes intricate and challenging,particularly when dealing with a significant number of channels.Here we use the stochastic parallel gradient descent(SPGD)algorithm to configure the integrated optical processor,which has less com-putation than the traditional gradient descent(GD)algorithm.We design and fabricate a 6×6 on-chip optical processor on silicon platform to implement optical switching and descrambling assisted by the online training with the SPDG algorithm.Moreover,we apply the on-chip processor configured by the SPGD algorithm to optical communications for optical switching and efficiently mitigating the channel crosstalk in SDM systems.In comparison with the traditional GD al-gorithm,it is found that the SPGD algorithm features better performance especially when the scale of matrix is large,which means it has the potential to optimize large-scale optical matrix computation acceleration chips.

Simultaneous sorting of arbitrary vector structured beams with spin-multiplexed diffractive metasurfaces

Vector structured beams(VSBs)offer infinite eigenstates and open up new possibilities for highcapacity optical and quantum communications by the multiplexing of the states.Therefore,the sorting and measuring of VSBs are extremely important.However,the efficient manipulations of a large number of VSBs have simultaneously remained challenging up to now,especially in integrated optical systems.Here,we propose a compact spin-multiplexed diffractive metasurface capable of continuously sorting and detecting arbitrary VSBs through spatial intensity separation.By introducing a diffractive optical neural network with cascaded metasurface systems,we demonstrate arbitrary VSBs sorters that can simultaneously identify Laguerre–Gaussian modes(l=−4 to 4,p=1 to 4),Hermitian–Gaussian modes(m=1 to 4,n=1 to 3),and Bessel–Gaussian modes(l=1 to 12).Such a sorter for arbitrary VSBs could revolutionize applications in integrated and high-dimensional optical communication systems.

Computation at the speed of light:metamaterials for all-optical calculations and neural networks

The explosion in the amount of information that is being processed is prompting the need for new computing systems beyond existing electronic computers.Photonic computing is emerging as an attractive alternative due to performing calculations at the speed of light,the change for massive parallelism,and also extremely low energy consumption.We review the physical implementation of basic optical calculations,such as differentiation and integration,using metamaterials,and introduce the realization of all-optical artificial neural networks.We start with concise introductions of the mathematical principles behind such optical computation methods and present the advantages,current problems that need to be overcome,and the potential future directions in the field.We expect that our review will be useful for both novice and experienced researchers in the field of all-optical computing platforms using metamaterials.

Flexible orbital angular momentum mode switching in multimode fibre using an optical neural network chip

Mode-division multiplexing technology has been proposed as a crucial technique for enhancing communication capacity and alleviating growing communication demands.Optical switching,which is an essential component of optical communication systems,enables information exchange between channels.However,existing optical switching solutions are inadequate for addressing flexible information exchange among the mode channels.In this study,we introduced a flexible mode switching system in a multimode fibre based on an optical neural network chip.This system utilised the flexibility of on-chip optical neural networks along with an all-fibre orbital angular momentum(OAM)mode multiplexer-demultiplexer to achieve mode switching among the three OAM modes within a multimode fibre.The system adopted an improved gradient descent algorithm to achieve training for arbitrary 3×3 exchange matrices and ensured maximum crosstalk of less than-18.7 dB,thus enabling arbitrary inter-mode channel information exchange.The proposed optical-neural-network-based mode-switching system was experimentally validated by successfully transmitting different modulation formats across various modes.This innovative solution holds promise for providing effective optical switching in practical multimode communication networks.

Inverse design of an integrated-nanophotonics optical neural network

Artificial neural networks have dramatically improved the performance of many machine-learning applications such as image recognition and natural language processing. However, the electronic hardware implementations of the above-mentioned tasks are facing performance ceiling because Moore’s Law is slowing down. In this article, we propose an optical neural network architecture based on optical scattering units to implement deep learning tasks with fast speed, low power consumption and small footprint.The optical scattering units allow light to scatter back and forward within a small region and can be optimized through an inverse design method. The optical scattering units can implement high-precision stochastic matrix multiplication with mean squared error < 10-4 and a mere 4*4 um2 footprint.Furthermore, an optical neural network framework based on optical scattering units is constructed by introducing "Kernel Matrix", which can achieve 97.1% accuracy on the classic image classification dataset MNIST.

Integrated diffractive optical neural network with space-time interleaving

Integrated diffractive optical neural networks(DONNs)have significant potential for complex machine learning tasks with high speed and ultralow energy consumption.However,the on-chip implementation of a high-performance optical neural network is limited by input dimensions.In contrast to existing photonic neural networks,a space-time interleaving technology based on arrayed waveguides is designed to realize an on-chip DONN with high-speed,high-dimensional,and all-optical input signal modulation.To demonstrate the performance of the on-chip DONN with high-speed space-time interleaving modulation,an on-chip DONN with a designed footprint of 0.0945 mm~2is proposed to resolve the vowel recognition task,reaching a computation speed of about 1.4×10^(13)operations per second and yielding an accuracy of 98.3%in numerical calculation.In addition,the function of the specially designed arrayed waveguides for realizing parallel signal inputs using space-time conversion has been verified experimentally.This method can realize the on-chip DONN with higher input dimension and lower energy consumption.

Two-photon nanolithography of micrometer scale diffractive neural network with cubical diffraction neurons at the visible wavelength

Free-space diffractive neural networks(DNNs)have been an intense research topic in machine learning for image recognition and encryption due to their high speed,lower power consumption,and high neuron density.Recent advances in DNNs have highlighted the need for smaller device footprints and the shift toward visible wavelengths.However,DNNs fabricated by electron beam lithography,are not suitable for microscopic imaging applications due to their large sizes,and DNNs fabricated by two-photon nanolithography with cylindrical neurons are not optimal for visible wavelengths,as the highorder diffraction could induce low diffraction efficiency.In this paper,we demonstrate that cubical diffraction neurons are more efficient diffraction elements for DNNs compared with cylindrical neurons.Based on the theoretical analysis of the relationship between the detector area sizes and classification accuracy,we reduced the size of DNNs operating at the wavelength of 532 nm for handwritten digit classification to micrometer scale by two-photon nanolithography.The DNNs with cubical neurons demonstrated an experimental classification accuracy(89.3%)for single-layer DNN,and 83.3%for two-layer DNN with device sizes similar to that of biological cells(about 100μm×100μm).Our results paved the pathway to integrate 3D micrometer-scale DNNs with microscopic imaging systems for biological imaging and cell recognition.

Optical neural network quantum state tomography

Quantum state tomography(QST)is a crucial ingredient for almost all aspects of experimental quantum information processing.As an analog of the“imaging”technique in quantum settings,QST is born to be a data science problem,where machine learning techniques,noticeably neural networks,have been applied extensively.We build and demonstrate an optical neural network(ONN)for photonic polarization qubit QST.The ONN is equipped with built-in optical nonlinear activation functions based on electromagnetically induced transparency.The experimental results show that our ONN can determine the phase parameter of the qubit state accurately.As optics are highly desired for quantum interconnections,our ONN-QST may contribute to the realization of optical quantum networks and inspire the ideas combining artificial optical intelligence with quantum information studies.

All optical artificial synapses based on long-afterglow material for optical neural network

Artificial neural network with broad application prospect has attracted particular attention due to the promise of solving the memory wall bottleneck.The neural devices that mix light and electricity provide more degrees of freedom for the design of artificial neural network,but they still do not get rid of the shackles that the response signal needs circuit to transmission.The exploration of all-optical neural devices(optical signal input and output)is expected to solve this problem.Here,an all-optical synaptic device simply based on a long-afterglow material is reported.The optical properties of the all-optical synaptic device are similar to the responses in biological synapses.Unique image displays and memory functions can be achieved by combining alloptical synaptic arrays with synaptic memory behavior.Furthermore,the optical summation of all-optical synaptic array pixels can be completed by combining the focusing characteristics of convex lens,which realizes the photon transmission after preprocessing multiple input signals.Particularly,the simple single-layer structure of all-optical synapses with polydimethylsiloxane(PDMS)as the carrier has high plasticity and is expected to achieve large-scale preparation.This work enriches the diversity of artificial synapses and shows the huge development potential of photoelectric artificial neural networks.

Polarization and wavelength routers based on difractive neural network

In the feld of information processing,all-optical routers are signifcant for achieving high-speed,high-capacity signal processing and transmission.In this study,we developed three types of structurally simple and fexible routers using the deep difractive neural network(D2 NN),capable of routing incident light based on wavelength and polarization.First,we implemented a polarization router for routing two orthogonally polarized light beams.The second type is the wavelength router that can route light with wavelengths of 1550,1300,and 1100 nm,demonstrating outstanding performance with insertion loss as low as 0.013 dB and an extinction ratio of up to 18.96 dB,while also maintaining excellent polarization preservation.The fnal router is the polarization-wavelength composite router,capable of routing six types of input light formed by pairwise combinations of three wavelengths(1550,1300,and 1100 nm)and two orthogonal linearly polarized lights,thereby enhancing the information processing capability of the device.These devices feature compact structures,maintaining high contrast while exhibiting low loss and passive characteristics,making them suitable for integration into future optical components.This study introduces new avenues and methodologies to enhance performance and broaden the applications of future optical information processing systems.

Massively parallel universal linear transformations using a wavelength-multiplexed diffractive optical network

Large-scale linear operations are the cornerstone for performing complex computational tasks.Using optical computing to perform linear transformations offers potential advantages in terms of speed,parallelism,and scalability.Previously,the design of successive spatially engineered diffractive surfaces forming an optical network was demonstrated to perform statistical inference and compute an arbitrary complex-valued linear transformation using narrowband illumination.We report deep-learning-based design of a massively parallel broadband diffractive neural network for all-optically performing a large group of arbitrarily selected,complex-valued linear transformations between an input and output field of view,each with Ni and No pixels,respectively.This broadband diffractive processor is composed of Nw wavelength channels,each of which is uniquely assigned to a distinct target transformation;a large set of arbitrarily selected linear transformations can be individually performed through the same diffractive network at different illumination wavelengths,either simultaneously or sequentially(wavelength scanning).We demonstrate that such a broadband diffractive network,regardless of its material dispersion,can successfully approximate Nw unique complex-valued linear transforms with a negligible error when the number of diffractive neurons(N)in its design is≥2NwNiNo.We further report that the spectral multiplexing capability can be increased by increasing N;our numerical analyses confirm these conclusions for Nw>180 and indicate that it can further increase to Nw∼2000,depending on the upper bound of the approximation error.Massively parallel,wavelength-multiplexed diffractive networks will be useful for designing highthroughput intelligent machine-vision systems and hyperspectral processors that can perform statistical inference and analyze objects/scenes with unique spectral properties.