Teacher-student learning of generative adversarial network-guided diffractive neural networks for visual tracking and imaging

Efficiently tracking and imaging interested moving targets is crucial across various applications,from autonomous systems to surveillance.However,persistent challenges remain in various fields,including environmental intricacies,limitations in perceptual technologies,and privacy considerations.We present a teacher-student learning model,the generative adversarial network(GAN)-guided diffractive neural network(DNN),which performs visual tracking and imaging of the interested moving target.The GAN,as a teacher model,empowers efficient acquisition of the skill to differentiate the specific target of interest in the domains of visual tracking and imaging.The DNN-based student model learns to master the skill to differentiate the interested target from the GAN.The process of obtaining a GAN-guided DNN starts with capturing moving objects effectively using an event camera with high temporal resolution and low latency.Then,the generative power of GAN is utilized to generate data with position-tracking capability for the interested moving target,subsequently serving as labels to the training of the DNN.The DNN learns to image the target during training while retaining the target’s positional information.Our experimental demonstration highlights the efficacy of the GAN-guided DNN in visual tracking and imaging of the interested moving target.We expect the GAN-guided DNN can significantly enhance autonomous systems and surveillance.

Pluggable multitask diffractive neural networks based on cascaded metasurfaces

Optical neural networks have significant advantages in terms of power consumption,parallelism,and high computing speed,which has intrigued extensive attention in both academic and engineering communities.It has been considered as one of the powerful tools in promoting the fields of imaging processing and object recognition.However,the existing optical system architecture cannot be reconstructed to the realization of multi-functional artificial intelligence systems simultaneously.To push the development of this issue,we propose the pluggable diffractive neural networks(P-DNN),a general paradigm resorting to the cascaded metasurfaces,which can be applied to recognize various tasks by switching internal plug-ins.As the proof-of-principle,the recognition functions of six types of handwritten digits and six types of fashions are numerical simulated and experimental demonstrated at near-infrared regimes.Encouragingly,the proposed paradigm not only improves the flexibility of the optical neural networks but paves the new route for achieving high-speed,low-power and versatile artificial intelligence systems.

Two-photon nanolithography of micrometer scale diffractive neural network with cubical diffraction neurons at the visible wavelength

Free-space diffractive neural networks(DNNs)have been an intense research topic in machine learning for image recognition and encryption due to their high speed,lower power consumption,and high neuron density.Recent advances in DNNs have highlighted the need for smaller device footprints and the shift toward visible wavelengths.However,DNNs fabricated by electron beam lithography,are not suitable for microscopic imaging applications due to their large sizes,and DNNs fabricated by two-photon nanolithography with cylindrical neurons are not optimal for visible wavelengths,as the highorder diffraction could induce low diffraction efficiency.In this paper,we demonstrate that cubical diffraction neurons are more efficient diffraction elements for DNNs compared with cylindrical neurons.Based on the theoretical analysis of the relationship between the detector area sizes and classification accuracy,we reduced the size of DNNs operating at the wavelength of 532 nm for handwritten digit classification to micrometer scale by two-photon nanolithography.The DNNs with cubical neurons demonstrated an experimental classification accuracy(89.3%)for single-layer DNN,and 83.3%for two-layer DNN with device sizes similar to that of biological cells(about 100μm×100μm).Our results paved the pathway to integrate 3D micrometer-scale DNNs with microscopic imaging systems for biological imaging and cell recognition.

Multimode diffractive optical neural network

On-chip diffractive optical neural networks(DONNs)bring the advantages of parallel processing and low energy consumption.However,an accurate representation of the optical field’s evolution in the structure cannot be provided using the previous diffraction-based analysis method.Moreover,the loss caused by the open boundaries poses challenges to applications.A multimode DONN architecture based on a more precise eigenmode analysis method is proposed.We have constructed a universal library of input,output,and metaline structures utilizing this method,and realized a multimode DONN composed of the structures from the library.On the designed multimode DONNs with only one layer of the metaline,the classification task of an Iris plants dataset is verified with an accuracy of 90%on the blind test dataset,and the performance of the one-bit binary adder task is also validated.Compared to the previous architectures,the multimode DONN exhibits a more compact design and higher energy efficiency.

Complex-valued universal linear transformations and image encryption using spatially incoherent diffractive networks

As an optical processor,a diffractive deep neural network(D2NN)utilizes engineered diffractive surfaces designed through machine learning to perform all-optical information processing,completing its tasks at the speed of light propagation through thin optical layers.With sufficient degrees of freedom,D2NNs can perform arbitrary complex-valued linear transformations using spatially coherent light.Similarly,D2NNs can also perform arbitrary linear intensity transformations with spatially incoherent illumination;however,under spatially incoherent light,these transformations are nonnegative,acting on diffraction-limited optical intensity patterns at the input field of view.Here,we expand the use of spatially incoherent D2NNs to complex-valued information processing for executing arbitrary complex-valued linear transformations using spatially incoherent light.Through simulations,we show that as the number of optimized diffractive features increases beyond a threshold dictated by the multiplication of the input and output space-bandwidth products,a spatially incoherent diffractive visual processor can approximate any complex-valued linear transformation and be used for all-optical image encryption using incoherent illumination.The findings are important for the all-optical processing of information under natural light using various forms of diffractive surface-based optical processors.

Diffractive Deep Neural Networks at Visible Wavelengths

Optical deep learning based on diffractive optical elements offers unique advantages for parallel processing,computational speed,and power efficiency.One landmark method is the diffractive deep neural network(D^(2) NN)based on three-dimensional printing technology operated in the terahertz spectral range.Since the terahertz bandwidth involves limited interparticle coupling and material losses,this paper extends D^(2) NN to visible wavelengths.A general theory including a revised formula is proposed to solve any contradictions between wavelength,neuron size,and fabrication limitations.A novel visible light D^(2) NN classifier is used to recognize unchanged targets(handwritten digits ranging from 0 to 9)and targets that have been changed(i.e.,targets that have been covered or altered)at a visible wavelength of 632.8 nm.The obtained experimental classification accuracy(84%)and numerical classification accuracy(91.57%)quantify the match between the theoretical design and fabricated system performance.The presented framework can be used to apply a D^(2) NN to various practical applications and design other new applications.

Diffraction deep neural network-based classification for vector vortex beams

The vector vortex beam(VVB)has attracted significant attention due to its intrinsic diversity of information and has found great applications in both classical and quantum communications.However,a VVB is unavoidably affected by atmospheric turbulence(AT)when it propagates through the free-space optical communication environment,which results in detection errors at the receiver.In this paper,we propose a VVB classification scheme to detect VVBs with continuously changing polarization states under AT,where a diffractive deep neural network(DDNN)is designed and trained to classify the intensity distribution of the input distorted VVBs,and the horizontal direction of polarization of the input distorted beam is adopted as the feature for the classification through the DDNN.The numerical simulations and experimental results demonstrate that the proposed scheme has high accuracy in classification tasks.The energy distribution percentage remains above 95%from weak to medium AT,and the classification accuracy can remain above 95%for various strengths of turbulence.It has a faster convergence and better accuracy than that based on a convolutional neural network.

Diffraction deep neural network based orbital angular momentum mode recognition scheme in oceanic turbulence

Orbital angular momentum(OAM)has the characteristics of mutual orthogonality between modes,and has been applied to underwater wireless optical communication(UWOC)systems to increase the channel capacity.In this work,we propose a diffractive deep neural network(DDNN)based OAM mode recognition scheme,where the DDNN is trained to capture the features of the intensity distribution of the OAM modes and output the corresponding azimuthal indices and radial indices.The results show that the proposed scheme can recognize the azimuthal indices and radial indices of the OAM modes accurately and quickly.In addition,the proposed scheme can resist weak oceanic turbulence(OT),and exhibit excellent ability to recognize OAM modes in a strong OT environment.The DDNN-based OAM mode recognition scheme has potential applications in UWOC systems.

Advanced all-optical classification using orbitalangular-momentum-encoded diffractive networks

As a successful case of combining deep learning with photonics,the research on optical machine learning has recently undergone rapid development.Among various optical classification frameworks,diffractive networks have been shown to have unique advantages in all-optical reasoning.As an important property of light,the orbital angular momentum(OAM)of light shows orthogonality and mode-infinity,which can enhance the ability of parallel classification in information processing.However,there have been few all-optical diffractive networks under the OAM mode encoding.Here,we report a strategy of OAM-encoded diffractive deep neural network(OAM-encoded D2NN)that encodes the spatial information of objects into the OAM spectrum of the diffracted light to perform all-optical object classification.We demonstrated three different OAM-encoded D2NNs to realize(1)single detector OAM-encoded D2NN for single task classification,(2)single detector OAM-encoded D2NN for multitask classification,and(3)multidetector OAM-encoded D2NN for repeatable multitask classification.We provide a feasible way to improve the performance of all-optical object classification and open up promising research directions for D2NN by proposing OAMencoded D2NN.

Simultaneous sorting of arbitrary vector structured beams with spin-multiplexed diffractive metasurfaces

Vector structured beams(VSBs)offer infinite eigenstates and open up new possibilities for highcapacity optical and quantum communications by the multiplexing of the states.Therefore,the sorting and measuring of VSBs are extremely important.However,the efficient manipulations of a large number of VSBs have simultaneously remained challenging up to now,especially in integrated optical systems.Here,we propose a compact spin-multiplexed diffractive metasurface capable of continuously sorting and detecting arbitrary VSBs through spatial intensity separation.By introducing a diffractive optical neural network with cascaded metasurface systems,we demonstrate arbitrary VSBs sorters that can simultaneously identify Laguerre–Gaussian modes(l=−4 to 4,p=1 to 4),Hermitian–Gaussian modes(m=1 to 4,n=1 to 3),and Bessel–Gaussian modes(l=1 to 12).Such a sorter for arbitrary VSBs could revolutionize applications in integrated and high-dimensional optical communication systems.

Unidirectional imaging with partially coherent light

.Unidirectional imagers form images of input objects only in one direction,e.g.,from field-of-view(FOV)A to FOV B,while blocking the image formation in the reverse direction,from FOV B to FOV A.Here,we report unidirectional imaging under spatially partially coherent light and demonstrate high-quality imaging only in the forward direction(A→B)with high power efficiency while distorting the image formation in the backward direction(B→A)along with low power efficiency.Our reciprocal design features a set of spatially engineered linear diffractive layers that are statistically optimized for partially coherent illumination with a given phase correlation length.Our analyses reveal that when illuminated by a partially coherent beam with a correlation length of≥∼1.5λ,whereλis the wavelength of light,diffractive unidirectional imagers achieve robust performance,exhibiting asymmetric imaging performance between the forward and backward directions—as desired.A partially coherent unidirectional imager designed with a smaller correlation length of<1.5λstill supports unidirectional image transmission but with a reduced figure of merit.These partially coherent diffractive unidirectional imagers are compact(axially spanning<75λ),polarization-independent,and compatible with various types of illumination sources,making them well-suited for applications in asymmetric visual information processing and communication.

Class-specific differential detection in diffractive optical neural networks improves inference accuracy

Optical computing provides unique opportunities in terms of parallelization,scalability,power efficiency,and computational speed and has attracted major interest for machine learning.Diffractive deep neural networks have been introduced earlier as an optical machine learning framework that uses task-specific diffractive surfaces designed by deep learning to all-optically perform inference,achieving promising performance for object classification and imaging.We demonstrate systematic improvements in diffractive optical neural networks,based on a differential measurement technique that mitigates the strict nonnegativity constraint of light intensity.In this differential detection scheme,each class is assigned to a separate pair of detectors,behind a diffractive optical network,and the class inference is made by maximizing the normalized signal difference between the photodetector pairs.Using this differential detection scheme,involving 10 photodetector pairs behind 5 diffractive layers with a total of 0.2 million neurons,we numerically achieved blind testing accuracies of 98.54%,90.54%,and 48.51%for MNIST,Fashion-MNIST,and grayscale CIFAR-10 datasets,respectively.Moreover,by utilizing the inherent parallelization capability of optical systems,we reduced the cross-talk and optical signal coupling between the positive and negative detectors of each class by dividing the optical path into two jointly trained diffractive neural networks that work in parallel.We further made use of this parallelization approach and divided individual classes in a target dataset among multiple jointly trained diffractive neural networks.Using this class-specific differential detection in jointly optimized diffractive neural networks that operate in parallel,our simulations achieved blind testing accuracies of 98.52%,91.48%,and 50.82%for MNIST,Fashion-MNIST,and grayscale CIFAR-10 datasets,respectively,coming close to the performance of some of the earlier generations of all-electronic deep neural networks,e.g.,LeNet,which achieves classification accuracies of 98.77%,90.27%,and 55.21%corresponding to the same datasets,respectively.In addition to these jointly optimized diffractive neural networks,we also independently optimized multiple diffractive networks and utilized them in a way that is similar to ensemble methods practiced in machine learning;using 3 independently optimized differential diffractive neural networks that optically project their light onto a common output/detector plane,we numerically achieved blind testing accuracies of 98.59%,91.06%,and 51.44%for MNIST,Fashion-MNIST,and grayscale CIFAR-10 datasets,respectively.Through these systematic advances in designing diffractive neural networks,the reported classification accuracies set the state of the art for all-optical neural network design.The presented framework might be useful to bring optical neural network-based low power solutions for various machine learning applications and help us design new computational cameras that are task-specific.

Integrated diffractive optical neural network with space-time interleaving

Integrated diffractive optical neural networks(DONNs)have significant potential for complex machine learning tasks with high speed and ultralow energy consumption.However,the on-chip implementation of a high-performance optical neural network is limited by input dimensions.In contrast to existing photonic neural networks,a space-time interleaving technology based on arrayed waveguides is designed to realize an on-chip DONN with high-speed,high-dimensional,and all-optical input signal modulation.To demonstrate the performance of the on-chip DONN with high-speed space-time interleaving modulation,an on-chip DONN with a designed footprint of 0.0945 mm~2is proposed to resolve the vowel recognition task,reaching a computation speed of about 1.4×10^(13)operations per second and yielding an accuracy of 98.3%in numerical calculation.In addition,the function of the specially designed arrayed waveguides for realizing parallel signal inputs using space-time conversion has been verified experimentally.This method can realize the on-chip DONN with higher input dimension and lower energy consumption.

Quantitative phase imaging(QPI)through random diffusers using a diffractive optical network

Quantitative phase imaging(QPI)is a label-free computational imaging technique used in various fields,including biology and medical research.Modern QPI systems typically rely on digital processing using iterative algorithms for phase retrieval and image reconstruction.Here,we report a diffractive optical network trained to convert the phase information of input objects positioned behind random diffusers into intensity variations at the output plane,all-optically performing phase recovery and quantitative imaging of phase objects completely hidden by unknown,random phase diffusers.This QPI diffractive network is composed of successive diffractive layers,axially spanning in total~70λ,where is the illumination wavelength;unlike existing digital image reconstruction and phase retrieval methods,it forms an all-optical processor that does not require external power beyond the illumination beam to complete its QPI reconstruction at the speed of light propagation.This all-optical diffractive processor can provide a low-power,high frame rate and compact alternative for quantitative imaging of phase objects through random,unknown diffusers and can operate at different parts of the electromagnetic spectrum for various applications in biomedical imaging and sensing.The presented QPI diffractive designs can be integrated onto the active area of standard CCD/CMOS-based image sensors to convert an existing optical microscope into a diffractive QPI microscope,performing phase recovery and image reconstruction on a chip through light diffraction within passive structured layers.

Optical information transfer through random unknown diffusers using electronic encoding and diffractive decoding

Free-space optical information transfer through diffusive media is critical in many applications, such as biomedical devices and optical communication, but remains challenging due to random, unknown perturbations in the optical path. We demonstrate an optical diffractive decoder with electronic encoding to accurately transfer the optical information of interest, corresponding to, e.g., any arbitrary input object or message, through unknown random phase diffusers along the optical path. This hybrid electronic-optical model, trained using supervised learning, comprises a convolutional neural network-based electronic encoder and successive passive diffractive layers that are jointly optimized. After their joint training using deep learning,our hybrid model can transfer optical information through unknown phase diffusers, demonstrating generalization to new random diffusers never seen before. The resulting electronic-encoder and optical-decoder model was experimentally validated using a 3D-printed diffractive network that axially spans <70λ, whereλ = 0.75 mm is the illumination wavelength in the terahertz spectrum, carrying the desired optical information through random unknown diffusers. The presented framework can be physically scaled to operate at different parts of the electromagnetic spectrum, without retraining its components, and would offer low-power and compact solutions for optical information transfer in free space through unknown random diffusive media.

Differential interference contrast phase edging net:an all-optical learning system for edge detection of phase objects

Edge detection for low-contrast phase objects cannot be performed directly by the spatial difference of intensity distribution.In this work,an all-optical diffractive neural network(DPENet)based on the differential interference contrast principle to detect the edges of phase objects in an all-optical manner is proposed.Edge information is encoded into an interference light field by dual Wollaston prisms without lenses and light-speed processed by the diffractive neural network to obtain the scale-adjustable edges.Simulation results show that DPENet achieves F-scores of 0.9308(MNIST)and 0.9352(NIST)and enables real-time edge detection of biological cells,achieving an F-score of 0.7462.

All-optical combinational logical units featuringfifth-order cascade

Modern computational technologies are gradually encounteringsignificant limitations, driving a shift toward alternative paradigms such as optical computing. In this study, novel all-opticalcombinational logic units based on diffractive neural networks(D2NNs) were introduced, which were designed to perform highorder logical operations efficiently and swiftly with the adoptionof only two modulation layers. This innovative design exhibitsincreased processing speed, improved energy efficiency, robustenvironmental stability, and high error tolerance, making itexceptionally well-suited for a broad spectrum of applications inoptical computing and communications. By leveraging thetransfer learning, we successfully developed a fifth-ordercascaded combinational logic circuit for a practical informationtransmission system. Furthermore, we revealed a pioneeringapplication of the device in optical time division multiplexing(OTDM), demonstrating its capability to manage high-speeddata transfer seamlessly without the need for electronic conversion. Extensive simulations and experimental validationsdemonstrate the potential of the model as a foundational technology for future optical computing architectures, which pavesthe way toward more sustainable and efficient optical dataprocessing platforms.

Deep learning‑enabled compact optical trigonometric operator with metasurface

In this paper, a novel strategy based on a metasurface composed of simple and compactunit cells to achieve ultra-high-speed trigonometric operations under specificinput values is theoretically and experimentally demonstrated. An electromagneticwave (EM)-based optical diffractive neural network with only one hidden layer isphysically built to perform four trigonometric operations (sine, cosine, tangent, andcotangent functions). Under the unique composite input mode strategy, the designedoptical trigonometric operator responds to incident light source modes that representdifferent trigonometric operations and input values (within one period), and generatescorrect and clear calculated results in the output layer. Such a wave-based operationis implemented with specific input values, and the proposed concept work may offerbreakthrough inspiration to achieve integrable optical computing devices and photonicsignal processors with ultra-fast running speeds.

Intelligent optoelectronic processor for orbital angular momentum spectrum measurement

Orbital angular momentum(OAM)detection underpins almost all aspects of vortex beams’advances such as communication and quantum analogy.Conventional schemes are frustrated by low speed,complicated system,limited detection range.Here,we devise an intelligent processor composed of photonic and electronic neurons for OAM spectrum measurement in a fast,accurate and direct manner.Specifically,optical layers extract invisible topological charge information from incoming light and a shallow electronic layer predicts the exact spectrum.The integration of optical-computing promises us a compact single-shot system with high speed and energy efficiency(optical operations/electronic operations~10^(3)),neither necessitating reference wave nor repetitive steps.Importantly,our processor is endowed with salient generalization ability and robustness against diverse structured light and adverse effects(mean squared error~10^((−5))).We further raise a universal model interpretation paradigm to reveal the underlying physical mechanisms in the hybrid processor,as distinct from conventional‘black-box’networks.Such interpretation algorithm can improve the detection efficiency up to 25-fold.We also complete the theory of optoelectronic network enabling its efficient training.This work not only contributes to the explorations on OAM physics and applications,and also broadly inspires the advanced links between intelligent computing and physical effects.

Computational imaging without a computer:seeing through random diffusers at the speed of light

Imaging through diffusers presents a challenging problem with various digital image reconstruction solutions demonstrated to date using computers.Here,we present a computer-free,all-optical image reconstruction method to see through random diffusers at the speed of light.Using deep learning,a set of transmissive diffractive surfaces are trained to all-optically reconstruct images of arbitrary objects that are completely covered by unknown,random phase diffusers.After the training stage,which is a one-time effort,the resulting diffractive surfaces are fabricated and form a passive optical network that is physically positioned between the unknown object and the image plane to all-optically reconstruct the object pattern through an unknown,new phase diffuser.We experimentally demonstrated this concept using coherent THz illumination and all-optically reconstructed objects distorted by unknown,random diffusers,never used during training.Unlike digital methods,all-optical diffractive reconstructions do not require power except for the illumination light.This diffractive solution to see through diffusers can be extended to other wavelengths,and might fuel various applications in biomedical imaging,astronomy,atmospheric sciences,oceanography,security,robotics,autonomous vehicles,among many others.