Multi-agent reinforcement learning for cooperative lane changing of connected and autonomous vehicles in mixed traffic

Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits,including releasing drivers from exhausting driving and mitigating traffic congestion,among others.Despite promising progress,lane-changing remains a great challenge for autonomous vehicles(AV),especially in mixed and dynamic traffic scenarios.Recently,reinforcement learning(RL)has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated.However,the majority of those studies are focused on a single-vehicle setting,and lane-changing in the context of multiple AVs coexisting with human-driven vehicles(HDVs)have received scarce attention.In this paper,we formulate the lane-changing decision-making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning(MARL)problem,where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs.Specifically,a multi-agent advantage actor-critic(MA2C)method is proposed with a novel local reward design and a parameter sharing scheme.In particular,a multi-objective reward function is designed to incorporate fuel efficiency,driving comfort,and the safety of autonomous driving.A comprehensive experimental study is made that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency,safety,and driver comfort.

Autonomous maneuver strategy of swarm air combat based on DDPG

Unmanned aerial vehicles(UAVs)have been found significantly important in the air combats,where intelligent and swarms of UAVs will be able to tackle with the tasks of high complexity and dynamics.The key to empower the UAVs with such capability is the autonomous maneuver decision making.In this paper,an autonomous maneuver strategy of UAV swarms in beyond visual range air combat based on reinforcement learning is proposed.First,based on the process of air combat and the constraints of the swarm,the motion model of UAV and the multi-to-one air combat model are established.Second,a two-stage maneuver strategy based on air combat principles is designed which include inter-vehicle collaboration and target-vehicle confrontation.Then,a swarm air combat algorithm based on deep deterministic policy gradient strategy(DDPG)is proposed for online strategy training.Finally,the effectiveness of the proposed algorithm is validated by multi-scene simulations.The results show that the algorithm is suitable for UAV swarms of different scales.

Decision making framework for autonomous vehicles driving behavior in complex scenarios via hierarchical state machine

With the development of autonomous car,a vehicle is capable to sense its environment more precisely.That allows improved drving behavior decision strategy to be used for more safety and effectiveness in complex scenarios.In this paper,a decision making framework based on hierarchical state machine is proposed with a top-down structure of three-layer finite state machine decision system.The upper layer classifies the driving scenario based on relative position of the vehicle and its surrounding vehicles.The middle layer judges the optimal driving behavior according to the improved energy efficiency function targeted at multiple criteria including driving efficiency,safety and the grid-based lane vacancy rate.The lower layer constructs the state transition matrix combined with the calculation results of the previous layer to predict the optimal pass way in the region.The simulation results show that the proposed driving strategy can integrate multiple criteria to evaluate the energy efficiency value of vehicle behavior in real time,and realize the selection of optimal vehicle driving strategy.With popularity of automatic vehicles in future,the driving strategy can be used as a reference to provide assistance for human drive or even the real-time decision-making of autonomous driving.

Nash equilibrium seeking over directed graphs

In this paper,we aim to develop distributed continuous-time algorithms over directed graphs to seek the Nash equilibrium in a noncooperative game.Motivated by the recent consensus-based designs,we present a distributed algorithm with a proportional gain for weight-balanced directed graphs.By further embedding a distributed estimator of the left eigenvector associated with zero eigenvalue of the graph Laplacian,we extend it to the case with arbitrary strongly connected directed graphs having possible unbalanced weights.In both cases,the Nash equilibrium is proven to be exactly reached with an exponential convergence rate.An example is given to illustrate the validity of the theoretical results.

End-of-Life Decision making in circular economy using generalized colored stochastic Petri nets

Circular economy enables to restore product value at the end of life i.e.when no longer used or damaged.Thus,the product life cycle is extended and this economy permits to reduce waste increase and resources rarefaction.There are several revaluation options(reuse,remanufacturing,recycling,...).So,decisionmakers need to assess these options to determine which is the best decision.Thus,we will present a study about an End-Of-Life(EoL)decision making which aims to facilitate the industrialization of circular economy.For this,it is essential to consider all variables and parameters impacting the decision of the product trajectory.A first part of the work proposes to identify the variables and parameters impacting the decision making.A second part proposes an assessment approach based on a modeling by Generalized Colored Stochastic Petri Net(GCSPN)and on a Monte-Carlo simulation.The approach developed is tested on an industrial example from the literature to analyze the efficiency and effectiveness of the model.This first application showed the feasibility of the approach,and also the limits of the GCSPN modelling.

Exponentially convergent distributed Nash equilibrium seeking for constrained aggregative games

Distributed Nash equilibrium seeking of aggregative games is investigated and a continuous-time algorithm is proposed.The algorithm is designed by virtue of projected gradient play dynamics and aggregation tracking dynamics,and is applicable to games with constrained strategy sets and weight-balanced communication graphs.The key feature of our method is that the proposed projected dynamics achieves exponential convergence,whereas such convergence results are only obtained for non-projected dynamics in existing works on distributed optimization and equilibrium seeking.Numerical examples illustrate the effectiveness of our methods.

Neural network-based adaptive sliding mode control for underactuated dual overhead cranes suffering from matched and unmatched disturbances

To improve transportation capacity,dual overhead crane systems(DOCSs)are playing an increasingly important role in the transportation of large/heavy cargos and containers.Unfortunately,when trying to deal with the control problem,current methods fail to fully consider such factors as external disturbances,input dead zones,parameter uncertainties,and other unmodeled dynamics that DOCSs usually suffer from.As a result,dramatic degradation is caused in the control performance,which badly hinders the practical applications of DOCSs.Motivated by this fact,this paper designs a neural network-based adaptive sliding mode control(SMC)method for DOCS to solve the aforementioned issues,which achieves satisfactory control performance for both actuated and underactuated state variables,even in the presence of matched and mismatched disturbances.The asymptotic stability of the desired equilibrium point is proved with rigorous Lyapunov-based analysis.Finally,extensive hardware experimental results are collected to verify the efficiency and robustness of the proposed method.

Hastily formed knowledge networks and distributed situation awareness for collaborative robotics

In the context of collaborative robotics,distributed situation awareness is essential for supporting collective intelligence in teams of robots and human agents where it can be used for both individual and collective decision support.This is particularly important in applications pertaining to emergency rescue and crisis management.During operational missions,data and knowledge are gathered incrementally and in different ways by heterogeneous robots and humans.We describe this as the creation of Hastily Formed Knowledge Networks(HFKNs).The focus of this paper is the specification and prototyping of a general distributed system architecture that supports the creation of HFKNs by teams of robots and humans.The information collected ranges from low-level sensor data to high-level semantic knowledge,the latter represented in part as RDF Graphs.The framework includes a synchronization protocol and associated algorithms that allow for the automatic distribution and sharing of data and knowledge between agents.This is done through the distributed synchronization of RDF Graphs shared between agents.High-level semantic queries specified in SPARQL can be used by robots and humans alike to acquire both knowledge and data content from team members.The system is empirically validated and complexity results of the proposed algorithms are provided.Additionally,a field robotics case study is described,where a 3D mapping mission has been executed using several UAVs in a collaborative emergency rescue scenario while using the full HFKN Framework.

Online distributed tracking of generalized Nash equilibrium on physical networks——Closing the Loop via Measurement Feedback

In generalized Nash equilibrium(GNE)seeking problems over physical networks such as power grids,the enforcement of network constraints and time-varying environment may bring high computational costs.Developing online algorithms is recognized as a promising method to cope with this challenge,where the task of computing system states is replaced by directly using measured values from the physical network.In this paper,we propose an online distributed algorithm via measurement feedback to track the GNE in a time-varying networked resource sharing market.Regarding that some system states are not measurable and measurement noise always exists,a dynamic state estimator is incorporated based on a Kalman filter,rendering a closed-loop dynamics of measurement-feedback driven online algorithm.We prove that,with a fixed step size,this online algorithm converges to a neighborhood of the GNE in expectation.Numerical simulations validate the theoretical results.

Lessons from human vision for robotic design

The visual guidance of goal-directed movements requires transformations of incoming visual information that are different from those required for visual perception.For us to grasp an object successfully,our brain must use justin-time computations of the object’s real-world size and shape,and its orientation and disposition with respect to our hand.These requirements have led to the emergence of dedicated visuomotor modules in the posterior parietal cortex of the human brain(the dorsal visual stream)that are functionally distinct from networks in the occipito-temporal cortex(the ventral visual stream)that mediate our conscious perception of the world.Although the identification and selection of goal objects and an appropriate course of action depends on the perceptual machinery of the ventral stream and associated cognitive modules,the execution of the subsequent goal-directed action is mediated by dedicated online control systems in the dorsal stream and associated motor areas.The dorsal stream allows an observer to reach out and grasp objects with exquisite ease,but by itself,deals only with objects that are visible at the moment the action is being programmed.The ventral stream,however,allows an observer to escape the present and bring to bear information from the past-including information about the function of objects,their intrinsic properties,and their location with reference to other objects in the world.Ultimately then,both streams contribute to the production of goal-directed actions.The principles underlying this division of labour between the dorsal and ventral streams are relevant to the design and implementation of autonomous robotic systems.

Robust flocking for non-identical second-order nonlinear multi-agent systems

This paper investigates the robust flocking problem for second-order nonlinear systems with a leader and external disturbances.In contrast with most of second-order systems in the literature,the intrinsic dynamics here are nonlinear and non-identical that depend not only on the velocity but also on the position,which is more realistic.Moreover,the interaction topology is undirected and switching.Provided that the leader’s velocity may be constant or time-varying,two distributed flocking control laws have been proposed for two cases to make the differences of the velocities between all followers and the leader approach to zero asymptotically.The proposed distributed flocking control laws are both model-independent which results in the effectiveness of the controllers to cope with the different intrinsic dynamics of the followers and the leader under some assumptions on boundedness of several states.An example is given to illustrate the validity of the theoretical results.

Task scheduling for transport and pick robots in logistics:a comparative study on constructive heuristics

We study the Transport and Pick Robots Task Scheduling(TPS)problem,in which two teams of specialized robots,transport robots and pick robots,collaborate to execute multi-station order fulfillment tasks in logistic environments.The objective is to plan a collective time-extended task schedule with the minimization of makespan.However,for this recently formulated problem,it is still unclear how to obtain satisfying results efficiently.In this research,we design several constructive heuristics to solve this problem based on the introduced sequence models.Theoretically,we give time complexity analysis or feasibility guarantees of these heuristics;empirically,we evaluate the makespan performance criteria and computation time on designed dataset.Computational results demonstrate that coupled append heuristic works better for the most cases within reasonable computation time.Coupled heuristics work better than decoupled heuristics prominently on instances with relative few pick robot numbers and large work zones.The law of diminishing marginal utility is also observed concerning the overall system performance and different transport-pick robot numbers.

Semi-global leader-following output consensus of heterogeneous systems subject to actuator position and rate saturation

We study in this paper a semi-global leader-following output consensus problem for multiple heterogeneous linear systems in the presence of actuator position and rate saturation over a directed topology.For each follower,via the low gain feedback design technique and output regulation theory,both a state feedback consensus protocol and an output feedback consensus protocol are constructed.In the output feedback case,different distributed observers are designed for the informed followers and uninformed followers to estimate the state of the leader and the follower itself.We show that the semi-global leader-following output consensus of heterogeneous linear systems can be achieved by the two consensus protocols if each follower is reachable from the leader in the directed communication topology.

Distributed multi-robot sweep coverage for a region with unknown workload distribution

This paper considers the scenario where multiple robots collaboratively cover a region in which the exact distribution of workload is unknown prior to the operation.The workload distribution is not uniform in the region,meaning that the time required to cover a unit area varies at different locations of the region.In our approach,we divide the target region into multiple horizontal stripes,and the robots sweep the current stripe while partitioning the next stripe concurrently.We propose a distributed workload partition algorithm and prove that the operation time on each stripe converges to the minimum under the discrete-time update law.We conduct comprehensive simulation studies and compare our method with the existing methods to verify the theoretical results and the advantage of the proposed method.Flight experiments on mini drones are also conducted to demonstrate the practicality of the proposed algorithm.

Formation control of unmanned rotorcraft systems with state constraints and inter-agent collision avoidance

We present in this paper a novel framework and distributed control laws for the formation of multiple unmanned rotorcraft systems,be it single-rotor helicopters or multi-copters,with physical constraints and with inter-agent collision avoidance,in cluttered environments.The proposed technique is composed of an analytical distributed consensus control solution in the free space and an optimization based motion planning algorithm for inter-agent and obstacle collision avoidance.More specifically,we design a distributed consensus control law to tackle a series of state constraints that include but not limited to the physical limitations of velocity,acceleration and jerk,and an optimization-based motion planning technique is utilized to generate numerical solutions when the consensus control fails to provide a collision-free trajectory.Besides,a sufficiency condition is given to guarantee the stability of the switching process between the consensus control and motion planning.Finally,both simulation and real flight experiments successfully demonstrate the effectiveness of the proposed technique.

A tripartite evolutionary game analysis of providing subsidies for pick-up/drop-off strategy in carpooling problem

Although the pick-up/drop-off(PUDO)strategy in carpooling offers the convenience of short-distance walking for passengers during boarding and disembarking,there is a noticeable hesitancy among commuters to adopt this travel method,despite its numerous benefits.Here,this paper establishes a tripartite evolutionary game theory(EGT)model to verify the evolutionary stability of choosing the PUDO strategy of drivers and passengers and offering subsidies strategy of carpooling platforms in carpooling system.The model presented in this paper serves as a valuable tool for assessing the dissemination and implementation of PUDO strategy and offering subsidies strategy in carpooling applications.Subsequently,an empirical analysis is conducted to examine and compare the sensitivity of the parameters across various scenarios.The findings suggest that:firstly,providing subsidies to passengers and drivers,along with deductions for drivers through carpooling platforms,is an effective way to promote wider adoption of the PUDO strategy.Then,the decision-making process is divided into three stages:initial stage,middle stage,and mature stage.PUDO strategy progresses from initial rejection to widespread acceptance among drivers in the middle stage and,in the mature stage,both passengers and drivers tend to adopt it under carpooling platform subsidies;the factors influencing the costs of waiting and walking times,as well as the subsidies granted to passengers,are essential determinants that require careful consideration by passengers,drivers,and carpooling platforms when choosing the PUDO strategy.Our work provides valuable insight into the PUDO strategy’s applicability and the declared results provide implications for traffic managers and carpooling platforms to offer a suitable incentive.

A novel collaborative decision-making method based on generalized abductive learning for resolving design conflicts

In complex product design,lots of time and resources are consumed to choose a preference-based compromise decision from non-inferior preliminary design models with multi-objective conflicts.However,since complex products involve intensive multi-domain knowledge,preference is not only a comprehensive representation of objective data and subjective knowledge but also characterized by fuzzy and uncertain.In recent years,enormous challenges are involved in the design process,within the increasing complexity of preference.This article mainly proposes a novel decision-making method based on generalized abductive learning(G-ABL)to achieve autonomous and efficient decision-making driven by data and knowledge collaboratively.The proposed G-ABL framework,containing three cores:classifier,abductive kernel,and abductive machine,supports preference integration from data and fuzzy knowledge.In particular,a subtle improvement is presented for WK-means based on the entropy weight method(EWM)to address the local static weight problem caused by the fixed data preferences as the decision set is locally invariant.Furthermore,fuzzy comprehensive evaluation(FCE)and Pearson correlation are adopted to quantify domain knowledge and obtain abducted labels.Multi-objective weighted calculations are utilized only to label and compare solutions in the final decision set.Finally,an engineering application is provided to verify the effectiveness of the proposed method,and the superiority of which is illustrated by comparative analysis.

Distilling base-and-meta network with contrastive learning for few-shot semantic segmentation

Current studies in few-shot semantic segmentation mostly utilize meta-learning frameworks to obtain models that can be generalized to new categories.However,these models trained on base classes with sufficient annotated samples are biased towards these base classes,which results in semantic confusion and ambiguity between base classes and new classes.A strategy is to use an additional base learner to recognize the objects of base classes and then refine the prediction results output by the meta learner.In this way,the interaction between these two learners and the way of combining results from the two learners are important.This paper proposes a new model,namely Distilling Base and Meta(DBAM)network by using self-attention mechanism and contrastive learning to enhance the few-shot segmentation performance.First,the self-attention-based ensemble module(SEM)is proposed to produce a more accurate adjustment factor for improving the fusion of two predictions of the two learners.Second,the prototype feature optimization module(PFOM)is proposed to provide an interaction between the two learners,which enhances the ability to distinguish the base classes from the target class by introducing contrastive learning loss.Extensive experiments have demonstrated that our method improves on the PASCAL-5i under 1-shot and 5-shot settings,respectively.

Remote collaborative process optimization in research and design of industrial manufacturing

In response to the impact of COVID-19,the manufacturing industry and academic industrial research have largely shifted to online or hybrid conference formats.The sudden change has posed challenges for researchers and teams to adapt.Based on the current state of online conferences,inadequate communication,disruptions during meetings,confusion and loss of meeting information,and difficulties in conducting online collaborations are observed.This paper presents a design of a real-time discussion board that combines online conferences and synchronous discussions to address the issues arising from remote collaborations in industrial research.The research demonstrates that synchronous discussions conducted within multi-team industrial collaboration teams with specific and diverse issues can better control the flow of meetings,enhance meeting efficiency,promote participant interaction and engagement,reduce information loss,and weaken the boundaries between online and offline collaboration.

Output-based adaptive distributed observer for general linear leader systems over periodic switching digraphs

In this paper,we present a sufficient condition for the exponential stability of a class of linear switched systems.As an application of this stability result,we establish an output-based adaptive distributed observer for a general linear leader system over a periodic jointly connected switching communication network,which extends the applicability of the output-based adaptive distributed observer from a marginally stable linear leader system to any linear leader system and from an undirected switching graph to a directed switching graph.This output-based adaptive distributed observer will be applied to solve the leader-following consensus problem for multiple double-integrator systems.