端到端自动驾驶系统研究综述

Survey of end-to-end autonomous driving systems

近年深度学习技术助力端到端自动驾驶框架的发展和进步,涌现出一系列创新研究议题与应用部署方案。本文首先以经典的模块化系统切入,对自动驾驶感知—预测—规划—决策4大功能模块进行简要概述,分析传统的模块化和多任务方法的局限性;其次从输入—输出模态到系统架构角度对当前新兴的端到端自动驾驶框架进行广泛地调研,详细描述弱解释性端到端与模块化联合端到端两大主流范式,深入探究现有研究工作存在的不足和弊端;之后简单介绍了端到端自动驾驶系统的开环—闭环评估方法及适用场景;最后总结了端到端自动驾驶系统的研究工作,并从数据挖掘和架构设计角度展望领域潜在挑战和亟待解决的关键问题。

Deep learning technologies have accelerated the development and advancement of end-to-end autonomous driving frameworks in recent years,sparking the emergence of numerous cutting-edge research topics and application deployment solutions.The“divide and conquer”architecture design concept,which aims to construct multiple independent but related module components,integrate them into the developed software system in a specific semantic or geometric order,and ultimately deploy these components to the actual vehicle,is the foundation for the majority of the autonomous driving systems currently in use,also known as modular systems.However,a well-developed modular design typically comprises thousands of components,placing a considerable burden on the graphics memory and processing capacity of automotive CPUs.Furthermore,the intrinsic mistakes of each stacked module during prediction will rise with the number of stacked modules,and upstream flaws cannot be fixed in downstream modules,presenting a major risk to vehicle safety.A multitask architecture based on the“task parallelism”principle aims to efficiently infer multiple tasks in parallel by designing various decoded heads with a shared backbone network to reduce computational consumption.However,the optimization goals for various tasks may not be consistent,and sharing features mindlessly can even degrade the overall performance of the system.In contrast to the previous two system architectures,the end-to-end technology paradigm eliminates information bottlenecks and cumulative errors due to the integration of numerous intermediate components based on rule interfaces,allowing the network to continually optimize toward a unified objective.A large model can be used to generate low-level control signals or vehicle motion planning based on inputs such as sensor data and vehicle status.With sensors serving as inputs,the early end-to-end design based on imitation and reinforcement learning directly outputs the final control commands for steering,braking,and acceleration.However,no explicit representation of driving scenarios in this completely“black box”network,which is also referred to as weakly interpretable end-to-end methods,is available.Thus,understanding the reasoning behind the decision or prediction of a vehicle is difficult for humans,making debugging,validation,and optimization challenging.Even worse,once the model malfunctions or unexpected situations occur,accurately detecting,avoiding,and repairing problems in a timely manner becomes difficult,all of which are crucial for maintaining the safe operation of intelligent vehicles.The component decoupling approach facilitates the development and optimization of individual modules in the conventional modular system,thereby guaranteeing steady representation performance and strong interpretability for each submodule.Unfortunately,this method falls short of achieving unified goals at the optimization level,that is,integrating optimization and learning toward the ultimate planning goal.A modular joint end-to-end autonomous driving architecture,which preserves the modular driving system while allowing the differentiability of each module,is a workable solution to ensure that every module has sufficient interpretability and overall automatic optimization capabilities.The basic idea behind this technology lies in the creation of a unique neural network that connects all independent modules and enables the gradients from the planning modules to be fed back down to the initial sensor input for end-to-end execution.In other words,this kind of approach merely modifies the submodule connection mechanism while maintaining the classic modular technology stack;that is,this approach substitutes a new implicit interface for the previous explicit interfaces,which were rule-based and required manual creation.Modular joint end-to-end procedures offer a certain interpretability because of the distinct separation between modules.The explicit end-to-end system is a relative decoupling based on overall design and exhibits some degree of logic in its sequential functioning from perception to prediction,and then to planning modules during decision inference.The model can be intentionally adjusted when it encounters unknown and uncontrollable results by understanding the operational logic underlying the explicit solution.Furthermore,visualization methods,such as internal features or intermediate results of specific tasks or modules,can be utilized to analyze the decision-making operation mechanism,which can prevent potential risks caused by black box models and ensure the safe and efficient driving of intelligent vehicles.Therefore,this article conducts comprehensive analysis and research on the emerging field of end-to-end autonomous driving with promising development prospects,which summarizes the main technical routes and representative research methods around the development path of end-to-end driving systems.More specifically,this article,which begins with the classic modular system,analyzes the shortcomings of conventional modular and multitasking approaches while providing a brief introduction to the four functional modules of the autonomous driving system.These modules primarily include perception,prediction,planning,and decision making.Subsequently,extensive research on the emerging end-to-end autonomous driving frameworks is conducted from the perspective of input-output modality to system architecture,describing in detail the two dominant paradigms and delving into the shortcomings and drawbacks of existing research work.The existing end-to-end architecture can be categorized into two categories based on interpretable performance:weakly interpretable end-to-end,which is explored from the aspects of imitation learning,reinforcement learning,and interpretability;or modular joint end-to-end,which is progressively investigated from bird’s-eye view representation,to joint perception prediction,and ultimately,planning-oriented end-to-end methods.Afterward,a thorough discussion of the end-to-end driving system assessment is provided for closed-and open-loop evaluations,along with the corresponding situations.Finally,the research works on end-to-end autonomous driving systems are summarized,and the potential challenges and key problems that still need to be addressed are discussed from the perspectives of data mining and architecture design.