面向道路目标检测的多模态融合语义传输

Multimodal Fusion-Based Semantic Transmission for Road Object Detection

在长尾效应的极端场景下,多车多传感器协作感知可为车辆提供有效的感知信息,但异构数据的差异化带宽限制和不同的数据格式使车辆在处理信息时难以进行统一高效的调度.为了在有限通信带宽下实现不同车辆间多传感器信息的有机融合,文中从语义通信的角度出发,提出基于Transformer的多模态融合目标检测语义通信模型.不同于传统的数据传输方案,文中模型利用自注意力机制融合不同模态的数据,着重探究各模态数据之间的语义相关性与依赖性.在有限的通信资源下帮助车辆进行信息传递和相互协作,提高车辆对复杂路况的理解能力.在Teledyne FLIR Free ADAS Thermal数据集上的实验表明,文中模型在多模态目标检测语义通信任务中表现出色,不仅大幅提升目标检测的准确性,同时也减少一半传输代价.

In extreme scenarios with long-tail effects,collaborative perception involving multiple vehicles and sensors can provide effective sensory information for vehicles.However,the differentiation in heterogeneous data,coupled with bandwidth constraints and diverse data formats,makes it challenging for vehicles to achieve unified and efficient scheduling in processing.To organically integrate multi-sensor information among different vehicles under limited communication bandwidth,a semantic communication framework for multimodal fusion object detection based on Transformer is proposed in this paper.Unlike traditional data transmission solutions,self-attention mechanisms are utilized in the proposed framework to fuse data from different modalities,focusing on exploring the semantic correlation and dependencies among modal data.It helps vehicles transmit information and collaborate under limited communication resources,thereby enhancing their understanding of complex road conditions.The experimental results on Teledyne FLIR Free ADAS Thermal dataset show that the proposed model performs well in multimodal object detection semantic communication tasks with accuracy of object detection significantly improved and transmission costs reduced by half.