Recently, the development of the Metaverse has become a frontier spotlight, which is an important demonstration of the integration innovation of advanced technologies in the Internet. Moreover, artificial intelligence...Recently, the development of the Metaverse has become a frontier spotlight, which is an important demonstration of the integration innovation of advanced technologies in the Internet. Moreover, artificial intelligence (AI) and 6G communications will be widely used in our daily lives. However, the effective interactions with the representations of multimodal data among users via 6G communications is the main challenge in the Metaverse. In this work, we introduce an intelligent cross-modal graph semantic communication approach based on generative AI and 3-dimensional (3D) point clouds to improve the diversity of multimodal representations in the Metaverse. Using a graph neural network, multimodal data can be recorded by key semantic features related to the real scenarios. Then, we compress the semantic features using a graph transformer encoder at the transmitter, which can extract the semantic representations through the cross-modal attention mechanisms. Next, we leverage a graph semantic validation mechanism to guarantee the exactness of the overall data at the receiver. Furthermore, we adopt generative AI to regenerate multimodal data in virtual scenarios. Simultaneously, a novel 3D generative reconstruction network is constructed from the 3D point clouds, which can transfer the data from images to 3D models, and we infer the multimodal data into the 3D models to increase realism in virtual scenarios. Finally, the experiment results demonstrate that cross-modal graph semantic communication, assisted by generative AI, has substantial potential for enhancing user interactions in the 6G communications and Metaverse.展开更多
An enormous number and tremendous diversity of microorganisms(such as bacteria and viruses)are surrounding us in our daily life,most of which we can live together in harmony under common circumstance.
基金supported in part by the National Natural Science Foundation of China (62001246, 62231017, 62201277, and 62071255)Key R and D Program of Jiangsu Province Key project and topics under Grants BE2021095 and BE2023035+3 种基金the Natural Science Foundation of Jiangsu Province under Grant BK20220390the Natural Science Research Startup Foundation of Recruiting Talents of Nanjing University of Posts and Telecommunications (Grant No. NY221011)The Key Project of Natural Science Foundation of Jiangsu Province (BE2023087)the major projects of the Natural Science Foundation of the Jiangsu Higher Education institutions (20KJA510009).
文摘Recently, the development of the Metaverse has become a frontier spotlight, which is an important demonstration of the integration innovation of advanced technologies in the Internet. Moreover, artificial intelligence (AI) and 6G communications will be widely used in our daily lives. However, the effective interactions with the representations of multimodal data among users via 6G communications is the main challenge in the Metaverse. In this work, we introduce an intelligent cross-modal graph semantic communication approach based on generative AI and 3-dimensional (3D) point clouds to improve the diversity of multimodal representations in the Metaverse. Using a graph neural network, multimodal data can be recorded by key semantic features related to the real scenarios. Then, we compress the semantic features using a graph transformer encoder at the transmitter, which can extract the semantic representations through the cross-modal attention mechanisms. Next, we leverage a graph semantic validation mechanism to guarantee the exactness of the overall data at the receiver. Furthermore, we adopt generative AI to regenerate multimodal data in virtual scenarios. Simultaneously, a novel 3D generative reconstruction network is constructed from the 3D point clouds, which can transfer the data from images to 3D models, and we infer the multimodal data into the 3D models to increase realism in virtual scenarios. Finally, the experiment results demonstrate that cross-modal graph semantic communication, assisted by generative AI, has substantial potential for enhancing user interactions in the 6G communications and Metaverse.
基金supported in part by the National Natural Science Foundation of China(22075149,22105104,62075102,and 22275097)Jiangsu Specially-Appointed Professor Plan,the Six Talent Plan of Jiangsu Province(XCL-049),HuaLi Talents Program of Nanjing University of Posts and Telecommunications,the Open Fund of the State Key Laboratory of Luminescent Materials and Devices(South China University of Technology)(2022-skllmd-01)+3 种基金the Open Research Fund of Songshan Lake Materials Laboratory(2022SLABFN16)the Innovation and Entrepreneurship Program of Jiangsu Province,China(JSSCBS20210536)the Fifth 333-Project of Jiangsu Province of China(BRA2019080)Nanjing University of Posts and Telecommunications Start-up Fund(NY220151 and NY219007).
基金supported by the National Natural Science Foundation of China (61974071, 61601394, T2125003, and 61875015)the National Key Research and Development Program of China (2017YFA0205302)+3 种基金Priority Academic Program Development of Jiangsu Higher Education Institutions (YX030003)Jiangsu Provincial Key Research and Development Program (BE2018732)Jiangsu Shuangchuang Talent Program, the Science and Technology Innovation Project for Overseas Students in NanjingStart-Up Fund from Nanjing University of Posts and Telecommunications (NY221129, NY218151, and NY218157)
文摘An enormous number and tremendous diversity of microorganisms(such as bacteria and viruses)are surrounding us in our daily life,most of which we can live together in harmony under common circumstance.
