Evolutionary Decision-Making and Planning for Autonomous Driving Based on Safe and Rational Exploration and Exploitation

Decision-making and motion planning are extremely important in autonomous driving to ensure safe driving in a real-world environment.This study proposes an online evolutionary decision-making and motion planning framework for autonomous driving based on a hybrid data-and model-driven method.First,a data-driven decision-making module based on deep reinforcement learning(DRL)is developed to pursue a rational driving performance as much as possible.Then,model predictive control(MPC)is employed to execute both longitudinal and lateral motion planning tasks.Multiple constraints are defined according to the vehicle’s physical limit to meet the driving task requirements.Finally,two principles of safety and rationality for the self-evolution of autonomous driving are proposed.A motion envelope is established and embedded into a rational exploration and exploitation scheme,which filters out unreasonable experiences by masking unsafe actions so as to collect high-quality training data for the DRL agent.Experiments with a high-fidelity vehicle model and MATLAB/Simulink co-simulation environment are conducted,and the results show that the proposed online-evolution framework is able to generate safer,more rational,and more efficient driving action in a real-world environment.

Toward Trustworthy Decision-Making for Autonomous Vehicles:A Robust Reinforcement Learning Approach with Safety Guarantees

While autonomous vehicles are vital components of intelligent transportation systems,ensuring the trustworthiness of decision-making remains a substantial challenge in realizing autonomous driving.Therefore,we present a novel robust reinforcement learning approach with safety guarantees to attain trustworthy decision-making for autonomous vehicles.The proposed technique ensures decision trustworthiness in terms of policy robustness and collision safety.Specifically,an adversary model is learned online to simulate the worst-case uncertainty by approximating the optimal adversarial perturbations on the observed states and environmental dynamics.In addition,an adversarial robust actor-critic algorithm is developed to enable the agent to learn robust policies against perturbations in observations and dynamics.Moreover,we devise a safety mask to guarantee the collision safety of the autonomous driving agent during both the training and testing processes using an interpretable knowledge model known as the Responsibility-Sensitive Safety Model.Finally,the proposed approach is evaluated through both simulations and experiments.These results indicate that the autonomous driving agent can make trustworthy decisions and drastically reduce the number of collisions through robust safety policies.

AUTONOMOUS AGENT FRAMEWORK AND ITS DECISION-MAKING

Autonomy, a key property associated with the agent, is an important topic in the current research of the agent theory. Although no definition of the agent autonomy is universally accepted, an important aspect of the agent autonomy is the decision-making capability of the agents. This paper investigates the autonomy of the agent, presents a framework for autonomous agent and discusses its decision-making process. Started with introducing a language for representing autonomous agent, a framework is proposed for modeling autonomous agent based on a BDI model and the situation calculus. Finally, a kind of decision-making process of the autonomous agent is presented.

Planning and Decision-making for Connected Autonomous Vehicles at Road Intersections:A Review

Planning and decision-making technology at intersections is a comprehensive research problem in intelligent transportation systems due to the uncertainties caused by a variety of traffic participants.As wireless communication advances,vehicle infrastructure integrated algorithms designed for intersection planning and decision-making have received increasing attention.In this paper,the recent studies on the planning and decision-making technologies at intersections are primarily overviewed.The general planning and decision-making approaches are presented,which include graph-based approach,prediction base approach,optimization-based approach and machine learning based approach.Since connected autonomous vehicles(CAVs)is the future direction for the automated driving area,we summarized the evolving planning and decision-making methods based on vehicle infrastructure cooperative technologies.Both four-way signalized and unsignalized intersection(s)are investigated under purely automated driving traffic and mixed traffic.The study benefit from current strategies,protocols,and simulation tools to help researchers identify the presented approaches’challenges and determine the research gaps,and several remaining possible research problems that need to be solved in the future.

Probabilistic Lane-Change Decision-Making and Planning for Autonomous Heavy Vehicles

To improve the safety and driving stability of the autonomous heavy truck, it is necessary to consider the differences of driving behavior and drivable trajectories between the heavy trucks and passenger cars. This study proposes a probabilistic decision-making and trajectory planning framework for the autonomous heavy trucks. Firstly, the driving decision process is divided into intention generation and feasibility evaluations, which are realized using the utility theory and risk assessment, respectively. Subsequently the driving decision is made and sent to the trajectory planning module. In order to reflect the greater risks of the truck to other surrounding vehicles, the aggressiveness index(AI) is proposed and quantified to infer the asymmetrical risk level of lane-change maneuver. In the planning stage, the lateral and roll dynamics stability domains are developed as the constraints to exclude the candidate trajectories that would cause vehicle instability. Finally, the simulation results are compared between the proposed model and the artificial potential filed model in the scenarios extracted from the naturalistic driving data. It is shown that the proposed framework can provide the human-like lane-change decisions and truck-friendly trajectories, and performs well in dynamic driving environments.

Driving decision-making analysis of car-following for autonomous vehicle under complex urban environment

The decision-making under complex urban environment become one of the key issues that restricts the rapid development of the autonomous vehicles. The difficulty in making timely and accurate decisions like human beings under highly dynamic traffic environment is a major challenge for autonomous driving. Car-following has been regarded as the simplest but essential driving behavior among driving tasks and has received extensive attention from researchers around the world. This work addresses this problem and proposes a novel method RSAN(rough-set artificial neural network) to learn the decisions from excellent human drivers. A virtual urban traffic environment was built by Pre Scan and driving simulation was conducted to obtain a broad set of relevant data such as experienced drivers' behavior data and surrounding vehicles' motion data. Then, rough set was used to preprocess these data to extract the key influential factors on decision and reduce the impact of uncertain data and noise data. And the car-following decision was learned by neural network in which key factor was the input and acceleration was the output. The result shows the better convergence speed and the better decision accuracy of RSAN than ANN. Findings of this work contributes to the empirical understanding of driver's decision-making process and it provides a theoretical basis for the study of car-following decision-making under complex and dynamic environment.

Human-Like Decision-Making of Autonomous Vehicles in Dynamic Traffic Scenarios

With the maturation of autonomous driving technology, the use of autonomous vehicles in a socially acceptable manner has become a growing demand of the public. Human-like autonomous driving is expected due to the impact of the differences between autonomous vehicles and human drivers on safety.Although human-like decision-making has become a research hotspot, a unified theory has not yet been formed, and there are significant differences in the implementation and performance of existing methods. This paper provides a comprehensive overview of human-like decision-making for autonomous vehicles. The following issues are discussed: 1) The intelligence level of most autonomous driving decision-making algorithms;2) The driving datasets and simulation platforms for testing and verifying human-like decision-making;3) The evaluation metrics of human-likeness;personalized driving;the application of decisionmaking in real traffic scenarios;and 4) The potential research direction of human-like driving. These research results are significant for creating interpretable human-like driving models and applying them in dynamic traffic scenarios. In the future, the combination of intuitive logical reasoning and hierarchical structure will be an important topic for further research. It is expected to meet the needs of human-like driving.

Modeling and TOPSIS-GRA Algorithm for Autonomous Driving Decision-Making Under 5G-V2X Infrastructure

This paper is to explore the problems of intelligent connected vehicles(ICVs)autonomous driving decision-making under a 5G-V2X structured road environment.Through literature review and interviews with autonomous driving practitioners,this paper firstly puts forward a logical framework for designing a cerebrum-like autonomous driving system.Secondly,situated on this framework,it builds a hierarchical finite state machine(HFSM)model as well as a TOPSIS-GRA algorithm for making ICV autonomous driving decisions by employing a data fusion approach between the entropy weight method(EWM)and analytic hierarchy process method(AHP)and by employing a model fusion approach between the technique for order preference by similarity to an ideal solution(TOPSIS)and grey relational analysis(GRA).The HFSM model is composed of two layers:the global FSM model and the local FSM model.The decision of the former acts as partial input information of the latter and the result of the latter is sent forward to the local pathplanning module,meanwhile pulsating feedback to the former as real-time refresh data.To identify different traffic scenarios in a cerebrum-like way,the global FSM model is designed as 7 driving behavior states and 17 driving characteristic events,and the local FSM model is designed as 16 states and 8 characteristic events.In respect to designing a cerebrum-like algorithm for state transition,this paper firstly fuses AHP weight and EWM weight at their output layer to generate a synthetic weight coefficient for each characteristic event;then,it further fuses TOPSIS method and GRA method at the model building layer to obtain the implementable order of state transition.To verify the feasibility,reliability,and safety of theHFSMmodel aswell as its TOPSISGRA state transition algorithm,this paper elaborates on a series of simulative experiments conducted on the PreScan8.50 platform.The results display that the accuracy of obstacle detection gets 98%,lane line prediction is beyond 70 m,the speed of collision avoidance is higher than 45 km/h,the distance of collision avoidance is less than 5 m,path planning time for obstacle avoidance is averagely less than 50 ms,and brake deceleration is controlled under 6 m/s2.These technical indexes support that the driving states set and characteristic events set for the HFSM model as well as its TOPSIS-GRA algorithm may bring about cerebrum-like decision-making effectiveness for ICV autonomous driving under 5G-V2X intelligent road infrastructure.

UAV maneuvering decision-making algorithm based on deep reinforcement learning under the guidance of expert experience

Autonomous umanned aerial vehicle(UAV) manipulation is necessary for the defense department to execute tactical missions given by commanders in the future unmanned battlefield. A large amount of research has been devoted to improving the autonomous decision-making ability of UAV in an interactive environment, where finding the optimal maneuvering decisionmaking policy became one of the key issues for enabling the intelligence of UAV. In this paper, we propose a maneuvering decision-making algorithm for autonomous air-delivery based on deep reinforcement learning under the guidance of expert experience. Specifically, we refine the guidance towards area and guidance towards specific point tasks for the air-delivery process based on the traditional air-to-surface fire control methods.Moreover, we construct the UAV maneuvering decision-making model based on Markov decision processes(MDPs). Specifically, we present a reward shaping method for the guidance towards area and guidance towards specific point tasks using potential-based function and expert-guided advice. The proposed algorithm could accelerate the convergence of the maneuvering decision-making policy and increase the stability of the policy in terms of the output during the later stage of training process. The effectiveness of the proposed maneuvering decision-making policy is illustrated by the curves of training parameters and extensive experimental results for testing the trained policy.

HEURISTIC PARTICLE SWARM OPTIMIZATION ALGORITHM FOR AIR COMBAT DECISION-MAKING ON CMTA

Combining the heuristic algorithm （HA） developed based on the specific knowledge of the cooperative multiple target attack （CMTA） tactics and the particle swarm optimization （PSO）, a heuristic particle swarm optimization （HPSO） algorithm is proposed to solve the decision-making （DM） problem. HA facilitates to search the local optimum in the neighborhood of a solution, while the PSO algorithm tends to explore the search space for possible solutions. Combining the advantages of HA and PSO, HPSO algorithms can find out the global optimum quickly and efficiently. It obtains the DM solution by seeking for the optimal assignment of missiles of friendly fighter aircrafts （FAs） to hostile FAs. Simulation results show that the proposed algorithm is superior to the general PSO algorithm and two GA based algorithms in searching for the best solution to the DM problem.

Efficient Autonomous Defense System Using Machine Learning on Edge Device

As a large amount of data needs to be processed and speed needs to be improved,edge computing with ultra-low latency and ultra-connectivity is emerging as a new paradigm.These changes can lead to new cyber risks,and should therefore be considered for a security threat model.To this end,we constructed an edge system to study security in two directions,hardware and software.First,on the hardware side,we want to autonomically defend against hardware attacks such as side channel attacks by configuring field programmable gate array(FPGA)which is suitable for edge computing and identifying communication status to control the communication method according to priority.In addition,on the software side,data collected on the server performs end-to-end encryption via symmetric encryption keys.Also,we modeled autonomous defense systems on the server by using machine learning which targets to incoming and outgoing logs.Server log utilizes existing intrusion detection datasets that should be used in real-world environments.Server log was used to detect intrusion early by modeling an intrusion prevention system to identify behaviors that violate security policy,and to utilize the existing intrusion detection data set that should be used in a real environment.Through this,we designed an efficient autonomous defense system that can provide a stable system by detecting abnormal signals from the device and converting them to an effective method to control edge computing,and to detect and control abnormal intrusions on the server side.

Cooperative Decision-Making for Multiple UAVs Autonomous Confrontation

This paper presents a rule-based framework for addressing decision-making problems within the context of the "UI-STRIVE"Competition.First,two distinct autonomous confrontation scenarios are described:autonomous air combat and cooperative interception.Second,a State-Event-Condition-Action(SECA)decision-making framework is developed,which integrates thefinite state machine and event-condition-action frameworks.This framework provides three products to describe rules,i.e.the SECA model,the SECA state chart,and the SECA rule description.Third,the situation assessment and target assignment during autonomous air combat are investigated,and the mathematical models are established.Finally,the decisionmaking model's rationality and feasibility are verified through data simulation and analysis.

An actor-critic based learning method for decision-making and planning of autonomous vehicles

In order to improve the agility and applicability of trajectory planning algorithm for autonomous vehicles, this paper proposes a novel actor-critic based learning method for decision-making and planning in multi-vehicle complex traffic. It is the coupling planning of vehicle’s path and speed thus to make the trajectory more flexible. First, generations from the decided action to the planned trajectory are described by the end-point of the trajectory. Then, the actor-critic based learning method is built to learn an optimal policy for the decision process. It can update the policy by the gradient of the current policy’s advantage. In this process,features of the real traffic are carefully extracted by time headway(TH) and speed distribution. Reward function is built by the safety, efficiency and driving comfort. Furthermore, to make the policy network have better convergency, the policy network is modularized in two parts: the lane-changing network and the lane-keeping network, which decide the optimal end-point of the path and speed candidates respectively. Finally, the curved overtaking scenario and the interaction process with human driver are conducted to illustrate the feasibility and superiority. The results show that the proposed method has better real-time performance and can make the planned coupling trajectory more continuous and smoother than the existing rule-based method.

Deep Reinforcement Learning Enabled Decision-Making for Autonomous Driving at Intersections

Road intersection is one of the most complex and accident-prone traffic scenarios,so it’s challenging for autonomous vehicles(AVs)to make safe and efficient decisions at the intersections.Most of the related studies focus on the solution to a single scenario or only guarantee safety without considering driving efficiency.To address these problems,this study proposed a deep reinforcement learning enabled decision-making framework for AVs to drive through intersections automatically,safely and efficiently.The mapping relationship between traffic images and vehicle operations was obtained by an end-to-end decision-making framework established by convolutional neural networks.Traffic images collected at two timesteps were used to calculate the relative velocity between vehicles.Markov decision process was employed to model the interaction between AVs and other vehicles,and the deep Q-network algorithm was utilized to obtain the optimal driving policy regarding safety and efficiency.To verify the effectiveness of the proposed decision-making framework,the top three accident-prone crossing path crash scenarios at intersections were simulated,when different initial vehicle states were adopted for better generalization capability.The results showed that the developed method could make AVs drive safely and efficiently through intersections in all of the tested scenarios.

针对自动驾驶智能模型的攻击与防御

近年来,以深度学习算法为代表的人工智能技术为人类生产生活的方方面面带来了巨大的革新,尤其是在自动驾驶领域,部署着自动驾驶系统的智能汽车已经走进入们的生活,成为了重要的生产力工具.然而,自动驾驶系统中的人工智能模型面临着潜在的安全隐患和风险,这给人民群众生命财产安全带来了严重威胁.本文通过回顾自动驾驶智能模型攻击和防御的相关研究工作,揭示自动驾驶系统在物理世界下面临的安全风险并归纳总结了相应的防御对策.具体来说,本文首先介绍了包含攻击面、攻击能力和攻击目标的自动驾驶系统安全风险模型.其次,面向自动驾驶系统的三个关键功能层——传感器层、感知层和决策层,本文依据受攻击的智能模型和攻击手段归纳、分析了对应的攻击方法以及防御对策,并探讨了现有方法的局限性.最后,本文讨论和展望了自动驾驶智能模型攻击与防御技术面临的难题与挑战,并指出了未来潜在的研究方向和发展趋势.

面向智算融合网络的自主防御范式研究

随着数字经济时代算力供给模式的变革,以算力为核心的新型网络基础设施已成为实现算力资源共享、支撑数字经济转型的重要动力.在算力网络中,多元异构用户终端通过多种方式高频接入网络以随时随地获取算力服务,网络的开放性和动态性增大,算力网络将面临更严峻的安全挑战.然而,基于传统网络的安全防御模式通常针对具体安全问题静态式增补安全防护组件,无法主动适配用户需求灵活调整防御策略,难以应对算力网络中的安全风险.因此,本文面向新型算力网络安全需求,将安全功能作为网络内部属性,基于智算融合网络提出一种多维协同自主防御范式.结合智算融合网络“三层”“三域”的设计思想,在“三层”中,以广义服务层定义安全固有服务,以映射适配层智慧适配安全功能,以融合组件层执行安全策略;在“三域”中,以实体域先导资源适配,以知识域驱动安全服务流程,以感控域实施具体安全技术,构建“检测”“溯源”“防御”三维一体的完整基础管控流程,其中安全策略与技术可根据场景扩展性与业务安全性进行灵活调整.最终,通过仿真实验对所提范式有效性进行了验证,为未来智算融合安全的进一步研究和应用提供参考.

基于节点路径重构和ELM的无线通信网络DDoS攻击源追踪

在无线通信网络中,DDoS攻击通常涉及大量的攻击者和恶意节点,并以多种形式发起攻击。攻击流量经过中间节点和反射/放大攻击等技术手段后变得更加复杂,追踪其溯源路径和确定唯一的攻击源变得复杂。为此,文中研究基于节点路径重构和ELM的无线通信网络DDoS攻击源追踪方法。通过正则化方式优化ELM的参数,检测获取DDoS攻击数据包;采用路由器标记算法标记DDoS攻击数据包,在无线通信网络域间重构攻击节点路径,获取DDoS攻击源位置,完成无线通信网络DDoS攻击源追踪。实验结果证明:文中方法可精准检测获取DDoS攻击数据包,并完成攻击数据包的标记,且可有效重构攻击节点路径,追踪到DDoS攻击源。

面向自动驾驶感知系统的对抗样本攻击研究综述

自动驾驶感知系统通过多种传感器采集周围环境信息并进行数据处理,用于检测车辆、行人和障碍物等,为后续的控制决策功能提供实时的基础数据.由于传感器直接与外部环境相连,且其自身往往缺乏辨别输入可信度的能力,因此感知系统成为众多攻击的潜在目标.对抗样本攻击是一种具有高隐蔽性和危害性的主流攻击方式,攻击者通过篡改或伪造感知系统的输入数据,欺骗感知算法,导致系统产生错误的输出结果,从而严重威胁自动驾驶安全.系统总结分析了自动驾驶感知系统的工作方式和面向感知系统的对抗样本攻击进展.从基于信号的对抗样本攻击和基于实物的对抗样本攻击2方面对比分析了面向自动驾驶感知系统的对抗样本攻击方案.同时,从异常检测、模型防御和物理防御3个方面全面分析了面向感知系统的对抗样本攻击的防御策略.最后,给出了面向自动驾驶感知系统的对抗样本攻击未来研究方向.

Safe Efficient Policy Optimization Algorithm for Unsignalized Intersection Navigation

Unsignalized intersections pose a challenge for autonomous vehicles that must decide how to navigate them safely and efficiently.This paper proposes a reinforcement learning(RL)method for autonomous vehicles to navigate unsignalized intersections safely and efficiently.The method uses a semantic scene representation to handle variable numbers of vehicles and a universal reward function to facilitate stable learning.A collision risk function is designed to penalize unsafe actions and guide the agent to avoid them.A scalable policy optimization algorithm is introduced to improve data efficiency and safety for vehicle learning at intersections.The algorithm employs experience replay to overcome the on-policy limitation of proximal policy optimization and incorporates the collision risk constraint into the policy optimization problem.The proposed safe RL algorithm can balance the trade-off between vehicle traffic safety and policy learning efficiency.Simulated intersection scenarios with different traffic situations are used to test the algorithm and demonstrate its high success rates and low collision rates under different traffic conditions.The algorithm shows the potential of RL for enhancing the safety and reliability of autonomous driving systems at unsignalized intersections.

A New Generation of Rules-based Approach:Mivar-based Intelligent Planning of Robot Actions(MIPRA)and Brains for Autonomous Robots

To create autonomous robots,both hardware and software are needed.If enormous progress has already been made in the field of equipment,then robot software depends on the development of artificial intelligence.This article proposes a solution for creating“logical”brains for autonomous robots,namely,an approach for creating an intelligent robot action planner based on Mivar expert systems.The application of this approach provides opportunities to reduce the computational complexity of solving planning problems and the requirements for the computational characteristics of hardware platforms on which intelligent planning systems are deployed.To theoretically and practically justify the expediency of using logically solving systems,in particular Mivar expert systems,to create intelligent planners,the MIPRA(Mivar-based Intelligent Planning of Robot Actions)planner was created to solve problems such as STRIPS for permutation cubes in the Blocks World domain.The planner is based on the platform for creating expert systems of the Razumator.As a result,the Mivar planner can process information about the state of the subject area based on the analysis of cause-effect relationships and an algorithm for automatically constructing logical inference(finding a solution from“Given”to“Find”).Moreover,an important feature of the MIPRA is that the system is built on the principles of a“white box”,due to which the system can explain any of its decisions and provide justification for the actions performed in the form of a retrospective of the stages of the decision-making process.When preparing a set of robot actions aimed at changing control objects,expert knowledge is used,which is the basis for the functioning algorithms of the planner.This approach makes it possible to include an expert in the process of organizing the work of the intelligent planner and use existing knowledge about the subject area.Practical experiments of this study have shown that instead of many hours and powerful multiprocessor servers,the MIPRA on a personal computer solves the planning problems with the following number of cubes:10 cubes can be rearranged in 0.028 seconds,100 cubes in 0.938 seconds,and 1000 cubes in 84.188 seconds.The results of this study can be used to reduce the computational complexity of solving tasks of planning the actions of robots,as well as their groups,multilevel heterogeneous robotic systems,and cyber-physical systems of various bases and purposes.