Artificial intelligence assisted light control and computational imaging through scattering media

Coherent optical control within or through scattering media via wavefront shaping has seen broad applications since its invention around 2007.Wavefront shaping is aimed at overcoming the strong scattering,featured by random interference,namely speckle patterns.This randomness occurs due to the refractive index inhomogeneity in complex media like biological tissue or the modal dispersion in multimode fiber,yet this randomness is actually deterministic and potentially can be time reversal or precompensated.Various wavefront shaping approaches,such as optical phase conjugation,iterative optimization,and transmission matrix measurement,have been developed to generate tight and intense optical delivery or high-resolution image of an optical object behind or within a scattering medium.The performance of these modula-tions,however,is far from satisfaction.Most recently,artifcial intelligence has brought new inspirations to this field,providing exciting hopes to tackle the challenges by mapping the input and output optical patterns and building a neuron network that inherently links them.In this paper,we survey the developments to date on this topic and briefly discuss our views on how to harness machine learning(deep learning in particular)for further advancements in the field.

Active wavefront shaping for controlling and improving multimode fi ber sensor

Wavefront shaping(WFS)techniques have been used as a powerful tool to control light propagation in complex media,including multimode fibers.In this paper,we propose a new application of WFS for multimode fber-based sensors.The use of a single multimode fiber alone,without any special fabrication,as a sensor based on the light intensity variations is not an easy task.The twist effect on multimode fiber is used as an example herein.Experimental results show that light intensity through the multimode fiber shows no direct relationship with the twist angle,but the correlation coefficient(CC)of speckle patterns does.Moreover,if WFS is applied to transform the spatially seemingly random light pattern at the exit of the multimode fiber into an optical focus.The focal pattern correlation and intensity both can serve to gauge the twist angle,with doubled measurement range and allowance of using a fast point detector to provide the feedback.With further development,WFS may find potentials to facilitate the development of multimode fber-based sensors in a variety of scenarios.

Nonconvex optimization for optimum retrieval of the transmission matrix of a multimode fiber

Transmission matrix(TM)allows light control through complex media,such as multimode fibers(MMFs),gaining great attention in areas,such as biophotonics,over the past decade.Efforts have been taken to retrieve a complex-valued TM directly from intensity measurements with several representative phase-retrieval algorithms,which still see limitations of slow or suboptimum recovery,especially under noisy environments.Here,we propose a modified nonconvex optimization approach.Through numerical evaluations,it shows that the optimum focusing efficiency is approached with less running time or sampling ratio.The comparative tests under different signal-to-noise levels further indicate its improved robustness.Experimentally,the superior focusing performance of our algorithm is collectively validated by single-and multispot focusing;especially with a sampling ratio of 8,it achieves a 93.6%efficiency of the gold-standard holography method.Based on the recovered TM,image transmission through an MMF is realized with high fidelity.Due to parallel operation and GPU acceleration,our nonconvex approach retrieves a 8685×1024 TM(sampling ratio is 8)with 42.3 s on average on a regular computer.The proposed method provides optimum efficiency and fast execution for TM retrieval that avoids the need for an external reference beam,which will facilitate applications of deep-tissue optical imaging,manipulation,and treatment.

Wavefront shaping: A versatile tool to conquer multiple scattering in multidisciplinary fields

Optical techniques offer a wide variety of applications as light-matter interactions provide extremely sensitive mechanisms to probe or treat target media.Most of these implementations rely on the usage of ballistic or quasi-ballistic photons to achieve high spatial resolution.However,the inherent scattering nature of light in biological tissues or tissue-like scattering media constitutes a critical obstacle that has restricted the penetration depth of non-scattered photons and hence limited the implementation of most optical techniques for wider applications.In addition,the components of an optical system are usually designed and manufactured for a fixed function or performance.Recent advances in wavefront shaping have demonstrated that scattering-or component-induced phase distortions can be compensated by optimizing the wavefront of the input light pattern through iteration or by conjugating the transmission matrix of the scattering medium.

Enhancing spatiotemporal focusing of light deep inside scattering media with Time-Gated Reflection Matrix

Time-gated reflection matrix(RM)has been successfully used for optical imaging deep inside Scattering media.Recently,this method was extended to enhance the spatiotemporal focusing of light ultra-deep inside scattering media.This is achieved by calibrating the decomposition of the RM with the Tikhonov regularization parameter to convert mutiply scattered photons that share the same time of flight with the singly scattered photons into singly scattered photons.Such a capability suggests a reshaping to the interaction mechanism between light and scattering media,which may beneft or inspire wide optical applications that desire enhanced spatiotemporal focusing of light at depths inside scattering media.

Towards ideal focusing of diffused light via optical wavefront shaping

Multiple scattering can significantly scramble the amplitude and phase profile of an optical field.It obscures subtle observations but only speckle patterns can be seen,unlike the ballistic regime where the information or the optical field can be identified with limited distortions.Efficient optical manipulation including information transmission and precise focusing is therefore obstructed as light travels deep into turbidmedia such as fog,turbid fluids,and biological tissues.

Long distance all-optical logic operations through a single multimode fiber empowered by wavefront shaping

Multimode fibers(MMFs)are a promising solution for high-throughput signal transmission in the time domain.However,crosstalk among different optical modes within the MMF scrambles input information and creates seemingly random speckle patterns at the output.To characterize this process,a transmission matrix(TM)can be used to relate input and output fields.Recent innovations use TMs to manipulate the output field by shaping the input wavefront for exciting advances in deep-brain imaging,neuron stimulation,quantum networks,and analog operators.However,these approaches consider input/output segments as independent,limiting their use for separate signal processing,such as logic operations.Our proposed method,which makes input/output segments as interdependent,adjusts the phase of corresponding output fields using phase bias maps superimposed on input segments.Coherent superposition enables signal logic operations through a 15-m-long MMF.In experiments,a single optical logic gate containing three basic logic functions and cascading multiple logic gates to handle binary operands is demonstrated.Bitwise operations are performed for multi-bit logic operations,and multiple optical logic gates are reconstructed simultaneously in a single logic gate with polarization multiplexing.The proposed method may open new avenues for long-range logic signal processing and transmission via MMFs.

Towards smart optical focusing:deep learningempowered dynamic wavefront shaping through nonstationary scattering media

Optical focusing through scattering media is of great significance yet challenging in lots of scenarios,including biomedical imaging,optical communication,cybersecurity,three-dimensional displays,etc.Wavefront shaping is a promising approach to solve this problem,but most implementations thus far have only dealt with static media,which,however,deviates from realistic applications.Herein,we put forward a deep learning-empowered adaptive framework,which is specifically implemented by a proposed Timely-Focusing-Optical-Transformation-Net(TFOTNet),and it effectively tackles the grand challenge of real-time light focusing and refocusing through time-variant media without complicated computation.The introduction of recursive fine-tuning allows timely focusing recovery,and the adaptive adjustment of hyperparameters of TFOTNet on the basis of medium changing speed efficiently handles the spatiotemporal non-stationarity of the medium.Simulation and experimental results demonstrate that the adaptive recursive algorithm with the proposed network significantly improves light focusing and tracking performance over traditional methods,permitting rapid recovery of an optical focus from degradation.It is believed that the proposed deep learning-empowered framework delivers a promising platform towards smart optical focusing implementations requiring dynamic wavefront control.

Adaptive optical focusing through perturbed scattering media with a dynamic mutation algorithm

Optical imaging through or inside scattering media, such as multimode fiber and biological tissues, has a significant impact in biomedicine yet is considered challenging due to the strong scattering nature of light. In the past decade, promising progress has been made in the field, largely benefiting from the invention of iterative optical wavefront shaping, with which deep-tissue high-resolution optical focusing and hence imaging becomes possible. Most of the reported iterative algorithms can overcome small perturbations on the noise level but fail to effectively adapt beyond the noise level, e.g., sudden strong perturbations. Reoptimizations are usually needed for significant decorrelation to the medium since these algorithms heavily rely on the optimization performance in the previous iterations. Such ineffectiveness is probably due to the absence of a metric that can gauge the deviation of the instant wavefront from the optimum compensation based on the concurrently measured optical focusing.In this study, a square rule of binary-amplitude modulation, directly relating the measured focusing performance with the error in the optimized wavefront, is theoretically proved and experimentally validated. With this simple rule, it is feasible to quantify how many pixels on the spatial light modulator incorrectly modulate the wavefront for the instant status of the medium or the whole system. As an example of application, we propose a novel algorithm, the dynamic mutation algorithm, which has high adaptability against perturbations by probing how far the optimization has gone toward the theoretically optimal performance. The diminished focus of scattered light can be effectively recovered when perturbations to the medium cause a significant drop in the focusing performance, which no existing algorithms can achieve due to their inherent strong dependence on previous optimizations. With further improvement, the square rule and the new algorithm may boost or inspire many applications, such as high-resolution optical imaging and stimulation, in instable or dynamic scattering environments.

Edge enhancement through scattering media enabled by optical wavefront shaping

Edge enhancement is a fundamental and important topic in imaging and image processing,as perception of edge is one of the keys to identify and comprehend the contents of an image.Edge enhancement can be performed in many ways,through hardware or computation.Existing methods,however,have been limited in free space or clear media for optical applications;in scattering media such as biological tissue,light is multiple scattered,and information is scrambled to a form of seemingly random speckles.Although desired,it is challenging to accom-plish edge enhancement in the presence of multiple scattering.In this work,we introduce an implementation of optical wavefront shaping to achieve efficient edge enhancement through scattering media by a two-step operation.The first step is to acquire a hologram after the scattering medium,where information of the edge region is accurately encoded,while that of the nonedge region is intentionally encoded with inadequate accuracy.The second step is to decode the edge information by time reversing the scattered light.The capability is demonstrated experimentally,and,further,the performance,as measured by the edge enhancement index(EI)and enhancement-to-noise ratio(ENR),can be controlled easily through tuning the beam ratio.EI and ENR can be reinforced by^8.5 and^263 folds,respectively.To the best of our knowledge,this is the first demonstration that edge information of a spatial pattern can be extracted through strong turbidity,which can potentially enrich the comprehension of actual images obtained from a complex environment.

Learning-based super-resolution interpolation for sub-Nyquist sampled laser speckles

Information retrieval from visually random optical speckle patterns is desired in many scenarios yet considered challenging.It requires accurate understanding or mapping of the multiple scattering process,or reliable capability to reverse or compensate for the scattering-induced phase distortions.In whatever situation,effective resolving and digitization of speckle patterns are necessary.Nevertheless,on some occasions,to increase the acquisition speed and/or signal-to-noise ratio(SNR),speckles captured by cameras are inevitably sampled in the sub-Nyquist domain via pixel binning(one camera pixel contains multiple speckle grains)due to finite size or limited bandwidth of photosensors.Such a down-sampling process is irreversible;it undermines the fine structures of speckle grains and hence the encoded information,preventing successful information extraction.To retrace the lost information,super-resolution interpolation for such sub-Nyquist sampled speckles is needed.In this work,a deep neural network,namely SpkSRNet,is proposed to effectively up sample speckles that are sampled below 1/10 of the Nyquist criterion to well-resolved ones that not only resemble the comprehensive morphology of original speckles(decompose multiple speckle grains from one camera pixel)but also recover the lost complex information(human face in this study)with high fidelity under normal-and low-light conditions,which is impossible with classic interpolation methods.These successful speckle super-resolution interpolation demonstrations are essentially enabled by the strong implicit correlation among speckle grains,which is non-quantifiable but could be discovered by the well-trained network.With further engineering,the proposed learning platform may benefit many scenarios that are physically inaccessible,enabling fast acquisition of speckles with sufficient SNR and opening up new avenues for seeing big and seeing clearly simultaneously in complex scenarios.

Accelerating deep learning with high energy efficiency:From microchip to physical systems

In the era of digits and intemet,massive data have been continuously gener-ated from a variety of sources,including video,photo,audio,text,intemet of things,etc.It is intuitive that more accurate pattems can be obtained by feeding more data for effective analysis;despite the data redundancy,a clearer picture can be delineated for better decision-making.However,traditional methods,even in machine learning,do not benefit from the expanding amount of data,whose perfomance nearty saturates when the data collection is large enough(Figure 1A).Such a dilemma emerges due to their limited capability and insuff cient supply of computation power in the past.