跨模态编码

Cross-modal coding

随着多媒体通信技术的发展,包含音频、视频和触觉信号的跨模态通信有望成为未来多媒体通信的主流。为满足跨模态异构码流的通信需求,文中提出了利用不同模态信号间语义相关性的跨模态编码方式。首先阐述了目前触觉编码的研究进展和相关的理论方法,然后详细分析了跨模态编码的信息论原理,并从稳定性控制、分级编码、跨模态嵌入同步和跨模态语义重构4个方面论述了跨模态编码架构的设计方案,最后对跨模态编码的研究做出了展望。

With the development of a multimedia communication technology,the cross-modal communica­tion,including audio,video and haptic signals,is expected to become the mainstream of multimedia com­munications in the future.To meet the communication requirements of cross-modal heterogeneous streams,a cross-modal coding method is proposed by using the semantic correlation between different modal sig­nals.Firstly,the current research progress of tactile coding and related theoretical methods are discussed.Then,the information theory principles of cross-modal coding are analyzed in detail,and a design scheme for cross-modal coding architecture is described from the four aspects of stability control,hierarchical cod­ing,cross-modal embedded synchronization,and cross-modal semantic reconstruction.Finally,a prospect for the future research of cross-modal coding is pointed out.