UAV-Assisted Dynamic Avatar Task Migration for Vehicular Metaverse Services: A Multi-Agent Deep Reinforcement Learning Approach

Avatars, as promising digital representations and service assistants of users in Metaverses, can enable drivers and passengers to immerse themselves in 3D virtual services and spaces of UAV-assisted vehicular Metaverses. However, avatar tasks include a multitude of human-to-avatar and avatar-to-avatar interactive applications, e.g., augmented reality navigation,which consumes intensive computing resources. It is inefficient and impractical for vehicles to process avatar tasks locally. Fortunately, migrating avatar tasks to the nearest roadside units(RSU)or unmanned aerial vehicles(UAV) for execution is a promising solution to decrease computation overhead and reduce task processing latency, while the high mobility of vehicles brings challenges for vehicles to independently perform avatar migration decisions depending on current and future vehicle status. To address these challenges, in this paper, we propose a novel avatar task migration system based on multi-agent deep reinforcement learning(MADRL) to execute immersive vehicular avatar tasks dynamically. Specifically, we first formulate the problem of avatar task migration from vehicles to RSUs/UAVs as a partially observable Markov decision process that can be solved by MADRL algorithms. We then design the multi-agent proximal policy optimization(MAPPO) approach as the MADRL algorithm for the avatar task migration problem. To overcome slow convergence resulting from the curse of dimensionality and non-stationary issues caused by shared parameters in MAPPO, we further propose a transformer-based MAPPO approach via sequential decision-making models for the efficient representation of relationships among agents. Finally, to motivate terrestrial or non-terrestrial edge servers(e.g., RSUs or UAVs) to share computation resources and ensure traceability of the sharing records, we apply smart contracts and blockchain technologies to achieve secure sharing management. Numerical results demonstrate that the proposed approach outperforms the MAPPO approach by around 2% and effectively reduces approximately 20% of the latency of avatar task execution in UAV-assisted vehicular Metaverses.

Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Progress on two-dimensional ferrovalley materials

The electron's charge and spin degrees of freedom are at the core of modern electronic devices. With the in-depth investigation of two-dimensional materials, another degree of freedom, valley, has also attracted tremendous research interest. The intrinsic spontaneous valley polarization in two-dimensional magnetic systems, ferrovalley material, provides convenience for detecting and modulating the valley. In this review, we first introduce the development of valleytronics.Then, the valley polarization forms by the p-, d-, and f-orbit that are discussed. Following, we discuss the investigation progress of modulating the valley polarization of two-dimensional ferrovalley materials by multiple physical fields, such as electric, stacking mode, strain, and interface. Finally, we look forward to the future developments of valleytronics.

Distributed Graph Database Load Balancing Method Based on Deep Reinforcement Learning

This paper focuses on the scheduling problem of workflow tasks that exhibit interdependencies.Unlike indepen-dent batch tasks,workflows typically consist of multiple subtasks with intrinsic correlations and dependencies.It necessitates the distribution of various computational tasks to appropriate computing node resources in accor-dance with task dependencies to ensure the smooth completion of the entire workflow.Workflow scheduling must consider an array of factors,including task dependencies,availability of computational resources,and the schedulability of tasks.Therefore,this paper delves into the distributed graph database workflow task scheduling problem and proposes a workflow scheduling methodology based on deep reinforcement learning(DRL).The method optimizes the maximum completion time(makespan)and response time of workflow tasks,aiming to enhance the responsiveness of workflow tasks while ensuring the minimization of the makespan.The experimental results indicate that the Q-learning Deep Reinforcement Learning(Q-DRL)algorithm markedly diminishes the makespan and refines the average response time within distributed graph database environments.In quantifying makespan,Q-DRL achieves mean reductions of 12.4%and 11.9%over established First-fit and Random scheduling strategies,respectively.Additionally,Q-DRL surpasses the performance of both DRL-Cloud and Improved Deep Q-learning Network(IDQN)algorithms,with improvements standing at 4.4%and 2.6%,respectively.With reference to average response time,the Q-DRL approach exhibits a significantly enhanced performance in the scheduling of workflow tasks,decreasing the average by 2.27%and 4.71%when compared to IDQN and DRL-Cloud,respectively.The Q-DRL algorithm also demonstrates a notable increase in the efficiency of system resource utilization,reducing the average idle rate by 5.02%and 9.30%in comparison to IDQN and DRL-Cloud,respectively.These findings support the assertion that Q-DRL not only upholds a lower average idle rate but also effectively curtails the average response time,thereby substantially improving processing efficiency and optimizing resource utilization within distributed graph database systems.

Recent advances in two-dimensional photovoltaic devices

Two-dimensional(2D)materials have attracted tremendous interest in view of the outstanding optoelectronic properties,showing new possibilities for future photovoltaic devices toward high performance,high specific power and flexibility.In recent years,substantial works have focused on 2D photovoltaic devices,and great progress has been achieved.Here,we present the review of recent advances in 2D photovoltaic devices,focusing on 2D-material-based Schottky junctions,homojunctions,2D−2D heterojunctions,2D−3D heterojunctions,and bulk photovoltaic effect devices.Furthermore,advanced strategies for improving the photovoltaic performances are demonstrated in detail.Finally,conclusions and outlooks are delivered,providing a guideline for the further development of 2D photovoltaic devices.

Unlocking the potential of ultra-thin two-dimensional antimony materials:Selective growth and carbon coating for efficient potassium-ion storage

Antimony-based anodes have attracted wide attention in potassium-ion batteries due to their high theoretical specific capacities(∼660 mA h g^(-1))and suitable voltage platforms.However,severe capacity fading caused by huge volume change and limited ion transportation hinders their practical applications.Recently,strategies for controlling the morphologies of Sb-based materials to improve the electrochemical performances have been proposed.Among these,the two-dimensional Sb(2D-Sb)materials present excellent properties due to shorted ion immigration paths and enhanced ion diffusion.Nevertheless,the synthetic methods are usually tedious,and even the mechanism of these strategies remains elusive,especially how to obtain large-scale 2D-Sb materials.Herein,a novel strategy to synthesize 2D-Sb material using a straightforward solvothermal method without the requirement of a complex nanostructure design is provided.This method leverages the selective adsorption of aldehyde groups in furfural to induce crystal growth,while concurrently reducing and coating a nitrogen-doped carbon layer.Compared to the reported methods,it is simpler,more efficient,and conducive to the production of composite nanosheets with uniform thickness(3–4 nm).The 2D-Sb@NC nanosheet anode delivers an extremely high capacity of 504.5 mA h g^(-1) at current densities of 100 mA g^(-1) and remains stable for more than 200 cycles.Through characterizations and molecular dynamic simulations,how potassium storage kinetics between 2D Sb-based materials and bulk Sb-based materials are explored,and detailed explanations are provided.These findings offer novel insights into the development of durable 2D alloy-based anodes for next-generation potassium-ion batteries.

QoS Routing Optimization Based on Deep Reinforcement Learning in SDN

To enhance the efficiency and expediency of issuing e-licenses within the power sector, we must confront thechallenge of managing the surging demand for data traffic. Within this realm, the network imposes stringentQuality of Service (QoS) requirements, revealing the inadequacies of traditional routing allocation mechanismsin accommodating such extensive data flows. In response to the imperative of handling a substantial influx of datarequests promptly and alleviating the constraints of existing technologies and network congestion, we present anarchitecture forQoS routing optimizationwith in SoftwareDefinedNetwork (SDN), leveraging deep reinforcementlearning. This innovative approach entails the separation of SDN control and transmission functionalities, centralizingcontrol over data forwardingwhile integrating deep reinforcement learning for informed routing decisions. Byfactoring in considerations such as delay, bandwidth, jitter rate, and packet loss rate, we design a reward function toguide theDeepDeterministic PolicyGradient (DDPG) algorithmin learning the optimal routing strategy to furnishsuperior QoS provision. In our empirical investigations, we juxtapose the performance of Deep ReinforcementLearning (DRL) against that of Shortest Path (SP) algorithms in terms of data packet transmission delay. Theexperimental simulation results show that our proposed algorithm has significant efficacy in reducing networkdelay and improving the overall transmission efficiency, which is superior to the traditional methods.

Anomalous valley Hall effect in two-dimensional valleytronic materials

The anomalous valley Hall effect(AVHE)can be used to explore and utilize valley degrees of freedom in materials,which has potential applications in fields such as information storage,quantum computing and optoelectronics.AVHE exists in two-dimensional(2D)materials possessing valley polarization(VP),and such 2D materials usually belong to the hexagonal honeycomb lattice.Therefore,it is necessary to achieve valleytronic materials with VP that are more readily to be synthesized and applicated experimentally.In this topical review,we introduce recent developments on realizing VP as well as AVHE through different methods,i.e.,doping transition metal atoms,building ferrovalley heterostructures and searching for ferrovalley materials.Moreover,2D ferrovalley systems under external modulation are also discussed.2D valleytronic materials with AVHE demonstrate excellent performance and potential applications,which offer the possibility of realizing novel low-energy-consuming devices,facilitating further development of device technology,realizing miniaturization and enhancing functionality of them.

Recent progress on valley polarization and valley-polarized topological states in two-dimensional materials

Valleytronics, using valley degree of freedom to encode, process, and store information, may find practical applications in low-power-consumption devices. Recent theoretical and experimental studies have demonstrated that twodimensional(2D) honeycomb lattice systems with inversion symmetry breaking, such as transition-metal dichalcogenides(TMDs), are ideal candidates for realizing valley polarization. In addition to the optical field, lifting the valley degeneracy of TMDs by introducing magnetism is an efficient way to manipulate the valley degree of freedom. In this paper, we first review the recent progress on valley polarization in various TMD-based systems, including magnetically doped TMDs,intrinsic TMDs with both inversion and time-reversal symmetry broken, and magnetic TMD heterostructures. When topologically nontrivial bands are empowered into valley-polarized systems, valley-polarized topological states, namely valleypolarized quantum anomalous Hall effect can be realized. Therefore, we have also reviewed the theoretical proposals for realizing valley-polarized topological states in 2D honeycomb lattices. Our paper can help readers quickly grasp the latest research developments in this field.

Global dust density in two-dimensional complex plasma

The driven-dissipative Langevin dynamics simulation is used to produce a two-dimensional(2D) dense cloud, which is composed of charged dust particles trapped in a quadratic potential. A 2D mesh grid is built to analyze the center-to-wall dust density. It is found that the local dust density in the outer region relative to that of the inner region is more nonuniform,being consistent with the feature of quadratic potential. The dependences of the global dust density on equilibrium temperature, particle size, confinement strength, and confinement shape are investigated. It is found that the particle size, the confinement strength, and the confinement shape strongly affect the global dust density, while the equilibrium temperature plays a minor effect on it. In the direction where there is a stronger confinement, the dust density gradient is bigger.

Reinforcement Learning in Process Industries:Review and Perspective

This survey paper provides a review and perspective on intermediate and advanced reinforcement learning(RL)techniques in process industries. It offers a holistic approach by covering all levels of the process control hierarchy. The survey paper presents a comprehensive overview of RL algorithms,including fundamental concepts like Markov decision processes and different approaches to RL, such as value-based, policy-based, and actor-critic methods, while also discussing the relationship between classical control and RL. It further reviews the wide-ranging applications of RL in process industries, such as soft sensors, low-level control, high-level control, distributed process control, fault detection and fault tolerant control, optimization,planning, scheduling, and supply chain. The survey paper discusses the limitations and advantages, trends and new applications, and opportunities and future prospects for RL in process industries. Moreover, it highlights the need for a holistic approach in complex systems due to the growing importance of digitalization in the process industries.

Role Dynamic Allocation of Human-Robot Cooperation Based on Reinforcement Learning in an Installation of Curtain Wall

A real-time adaptive roles allocation method based on reinforcement learning is proposed to improve humanrobot cooperation performance for a curtain wall installation task.This method breaks the traditional idea that the robot is regarded as the follower or only adjusts the leader and the follower in cooperation.In this paper,a self-learning method is proposed which can dynamically adapt and continuously adjust the initiative weight of the robot according to the change of the task.Firstly,the physical human-robot cooperation model,including the role factor is built.Then,a reinforcement learningmodel that can adjust the role factor in real time is established,and a reward and actionmodel is designed.The role factor can be adjusted continuously according to the comprehensive performance of the human-robot interaction force and the robot’s Jerk during the repeated installation.Finally,the roles adjustment rule established above continuously improves the comprehensive performance.Experiments of the dynamic roles allocation and the effect of the performance weighting coefficient on the result have been verified.The results show that the proposed method can realize the role adaptation and achieve the dual optimization goal of reducing the sum of the cooperator force and the robot’s Jerk.

Emerging two-dimensional Mo-based materials for rechargeable metal-ion batteries:Advances and perspectives

With the rapid development of rechargeable metal-ion batteries(MIBs)with safety,stability and high energy density,significant efforts have been devoted to exploring high-performance electrode materials.In recent years,two-dimensional(2D)molybdenum-based(Mo-based)materials have drawn considerable attention due to their exceptional characteristics,including low cost,unique crystal structure,high theoretical capacity and controllable chemical compositions.However,like other transition metal compounds,Mo-based materials are facing thorny challenges to overcome,such as slow electron/ion transfer kinetics and substantial volume changes during the charge and discharge processes.In this review,we summarize the recent progress in developing emerging 2D Mo-based electrode materials for MIBs,encompassing oxides,sulfides,selenides,carbides.After introducing the crystal structure and common synthesis methods,this review sheds light on the charge storage mechanism of several 2D Mo-based materials by various advanced characterization techniques.The latest achievements in utilizing 2D Mo-based materials as electrode materials for various MIBs(including lithium-ion batteries(LIBs),sodium-ion batteries(SIBs)and zinc-ion batteries(ZIBs))are discussed in detail.Afterwards,the modulation strategies for enhancing the electrochemical performance of 2D Mo-based materials are highlighted,focusing on heteroatom doping,vacancies creation,composite coupling engineering and nanostructure design.Finally,we present the existing challenges and future research directions for 2D Mo-based materials to realize high-performance energy storage systems.

Toward Trustworthy Decision-Making for Autonomous Vehicles:A Robust Reinforcement Learning Approach with Safety Guarantees

While autonomous vehicles are vital components of intelligent transportation systems,ensuring the trustworthiness of decision-making remains a substantial challenge in realizing autonomous driving.Therefore,we present a novel robust reinforcement learning approach with safety guarantees to attain trustworthy decision-making for autonomous vehicles.The proposed technique ensures decision trustworthiness in terms of policy robustness and collision safety.Specifically,an adversary model is learned online to simulate the worst-case uncertainty by approximating the optimal adversarial perturbations on the observed states and environmental dynamics.In addition,an adversarial robust actor-critic algorithm is developed to enable the agent to learn robust policies against perturbations in observations and dynamics.Moreover,we devise a safety mask to guarantee the collision safety of the autonomous driving agent during both the training and testing processes using an interpretable knowledge model known as the Responsibility-Sensitive Safety Model.Finally,the proposed approach is evaluated through both simulations and experiments.These results indicate that the autonomous driving agent can make trustworthy decisions and drastically reduce the number of collisions through robust safety policies.

Boosting MA-based two-dimensional Ruddlesden-Popper perovskite solar cells by incorporating a binary spacer

Two-dimensional Ruddlesden-Popper(2DRP)perovskite exhibits excellent stability in perovskite solar cells(PSCs)due to introducing hydrophobic long-chain organic spacers.However,the poor charge transporting property of bulky organic cation spacers limits the performance of 2DRP PSCs.Inspired by the Asite cation alloying strategy in 3D perovskites,2DRP perovskites with a binary spacer can promote charge transporting compared to the unary spacer counterparts.Herein,the superior MA-based 2DRP perovskite films with a binary spacer,including 3-guanidinopropanoic acid(GPA)and 4-fluorophenethylamine(FPEA)are realized.These films(GPA_(0.85)FPEA_(0.15))_(2)MA_(4)Pb_5I_(16)show good morphology,large grain size,decreased trap state density,and preferential orientation of the as-prepared film.Accordingly,the present 2DRP-based PSC with the binary spacer achieves a remarkable efficiency of 18.37%with a V_(OC)of1.15 V,a J_(SC)of 20.13 mA cm^(-2),and an FF of 79.23%.To our knowledge,the PCE value should be the highest for binary spacer MA-based 2DRP(n≤5)PSCs to date.Importantly,owing to the hydrophobic fluorine group of FPEA and the enhanced interlayer interaction by FPEA,the unencapsulated 2DRP PSCs based on binary spacers exhibit much excellent humidity stability and thermal stability than the unary spacer counterparts.

Low-frequency hybridized excess vibrations of two-dimensional glasses

One hallmark of glasses is the existence of excess vibrational modes at low frequenciesωbeyond Debye’s prediction.Numerous studies suggest that understanding low-frequency excess vibrations could help gain insight into the anomalous mechanical and thermodynamic properties of glasses.However,there is still intensive debate as to the frequency dependence of the population of low-frequency excess vibrations.In particular,excess modes could hybridize with phonon-like modes and the density of hybridized excess modes has been reported to follow D_(exc)(ω)~ω^(2)in 2D glasses with an inverse power law potential.Yet,the universality of the quadratic scaling remains unknown,since recent work suggested that interaction potentials could influence the scaling of the vibrational spectrum.Here,we extend the universality of the quadratic scaling for hybridized excess modes in 2D to glasses with potentials ranging from the purely repulsive soft-core interaction to the hard-core one with both repulsion and attraction as well as to glasses with significant differences in density or interparticle repulsion.Moreover,we observe that the number of hybridized excess modes exhibits a decrease in glasses with higher density or steeper interparticle repulsion,which is accompanied by a suppression of the strength of the sound attenuation.Our results indicate that the density bears some resemblance to the repulsive steepness of the interaction in influencing low-frequency properties.

Image segmentation of exfoliated two-dimensional materials by generative adversarial network-based data augmentation

Mechanically cleaved two-dimensional materials are random in size and thickness.Recognizing atomically thin flakes by human experts is inefficient and unsuitable for scalable production.Deep learning algorithms have been adopted as an alternative,nevertheless a major challenge is a lack of sufficient actual training images.Here we report the generation of synthetic two-dimensional materials images using StyleGAN3 to complement the dataset.DeepLabv3Plus network is trained with the synthetic images which reduces overfitting and improves recognition accuracy to over 90%.A semi-supervisory technique for labeling images is introduced to reduce manual efforts.The sharper edges recognized by this method facilitate material stacking with precise edge alignment,which benefits exploring novel properties of layered-material devices that crucially depend on the interlayer twist-angle.This feasible and efficient method allows for the rapid and high-quality manufacturing of atomically thin materials and devices.

Self organizing optimization and phase transition in reinforcement learning minority game system

Whether the complex game system composed of a large number of artificial intelligence(AI)agents empowered with reinforcement learning can produce extremely favorable collective behaviors just through the way of agent self-exploration is a matter of practical importance.In this paper,we address this question by combining the typical theoretical model of resource allocation system,the minority game model,with reinforcement learning.Each individual participating in the game is set to have a certain degree of intelligence based on reinforcement learning algorithm.In particular,we demonstrate that as AI agents gradually becomes familiar with the unknown environment and tries to provide optimal actions to maximize payoff,the whole system continues to approach the optimal state under certain parameter combinations,herding is effectively suppressed by an oscillating collective behavior which is a self-organizing pattern without any external interference.An interesting phenomenon is that a first-order phase transition is revealed based on some numerical results in our multi-agents system with reinforcement learning.In order to further understand the dynamic behavior of agent learning,we define and analyze the conversion path of belief mode,and find that the self-organizing condensation of belief modes appeared for the given trial and error rates in the AI system.Finally,we provide a detection method for period-two oscillation collective pattern emergence based on the Kullback–Leibler divergence and give the parameter position where the period-two appears.

Single-cell manipulation by two-dimensional micropatterning

Cells are highly sensitive to their geometrical and mechanical microenvironment that directly regulate cell shape,cytoskeleton and organelle,as well as the nucleus morphology and genetic expression.The emerging two-dimensional micropatterning techniques offer powerful tools to construct controllable and well-organized microenvironment for single-cell level investigations with qualitative analysis,cellular standardization,and in vivo environment mimicking.Here,we provide an overview of the basic principle and characteristics of the two most widely-used micropatterning techniques,including photolithographic micropatterning and soft lithography micropatterning.Moreover,we summarize the application of micropatterning technique in controlling cytoskeleton,cell migration,nucleus and gene expression,as well as intercellular communication.

Knowledge Reasoning Method Based on Deep Transfer Reinforcement Learning:DTRLpath

In recent years,with the continuous development of deep learning and knowledge graph reasoning methods,more and more researchers have shown great interest in improving knowledge graph reasoning methods by inferring missing facts through reasoning.By searching paths on the knowledge graph and making fact and link predictions based on these paths,deep learning-based Reinforcement Learning(RL)agents can demonstrate good performance and interpretability.Therefore,deep reinforcement learning-based knowledge reasoning methods have rapidly emerged in recent years and have become a hot research topic.However,even in a small and fixed knowledge graph reasoning action space,there are still a large number of invalid actions.It often leads to the interruption of RL agents’wandering due to the selection of invalid actions,resulting in a significant decrease in the success rate of path mining.In order to improve the success rate of RL agents in the early stages of path search,this article proposes a knowledge reasoning method based on Deep Transfer Reinforcement Learning path(DTRLpath).Before supervised pre-training and retraining,a pre-task of searching for effective actions in a single step is added.The RL agent is first trained in the pre-task to improve its ability to search for effective actions.Then,the trained agent is transferred to the target reasoning task for path search training,which improves its success rate in searching for target task paths.Finally,based on the comparative experimental results on the FB15K-237 and NELL-995 datasets,it can be concluded that the proposed method significantly improves the success rate of path search and outperforms similar methods in most reasoning tasks.