M3SC:A Generic Dataset for Mixed Multi-Modal(MMM)Sensing and Communication Integration

The sixth generation(6G)of mobile communication system is witnessing a new paradigm shift,i.e.,integrated sensing-communication system.A comprehensive dataset is a prerequisite for 6G integrated sensing-communication research.This paper develops a novel simulation dataset,named M3SC,for mixed multi-modal(MMM)sensing-communication integration,and the generation framework of the M3SC dataset is further given.To obtain multimodal sensory data in physical space and communication data in electromagnetic space,we utilize Air-Sim and WaveFarer to collect multi-modal sensory data and exploit Wireless InSite to collect communication data.Furthermore,the in-depth integration and precise alignment of AirSim,WaveFarer,andWireless InSite are achieved.The M3SC dataset covers various weather conditions,multiplex frequency bands,and different times of the day.Currently,the M3SC dataset contains 1500 snapshots,including 80 RGB images,160 depth maps,80 LiDAR point clouds,256 sets of mmWave waveforms with 8 radar point clouds,and 72 channel impulse response(CIR)matrices per snapshot,thus totaling 120,000 RGB images,240,000 depth maps,120,000 LiDAR point clouds,384,000 sets of mmWave waveforms with 12,000 radar point clouds,and 108,000 CIR matrices.The data processing result presents the multi-modal sensory information and communication channel statistical properties.Finally,the MMM sensing-communication application,which can be supported by the M3SC dataset,is discussed.

Multi-task Learning of Semantic Segmentation and Height Estimation for Multi-modal Remote Sensing Images

Deep learning based methods have been successfully applied to semantic segmentation of optical remote sensing images.However,as more and more remote sensing data is available,it is a new challenge to comprehensively utilize multi-modal remote sensing data to break through the performance bottleneck of single-modal interpretation.In addition,semantic segmentation and height estimation in remote sensing data are two tasks with strong correlation,but existing methods usually study individual tasks separately,which leads to high computational resource overhead.To this end,we propose a Multi-Task learning framework for Multi-Modal remote sensing images(MM_MT).Specifically,we design a Cross-Modal Feature Fusion(CMFF)method,which aggregates complementary information of different modalities to improve the accuracy of semantic segmentation and height estimation.Besides,a dual-stream multi-task learning method is introduced for Joint Semantic Segmentation and Height Estimation(JSSHE),extracting common features in a shared network to save time and resources,and then learning task-specific features in two task branches.Experimental results on the public multi-modal remote sensing image dataset Potsdam show that compared to training two tasks independently,multi-task learning saves 20%of training time and achieves competitive performance with mIoU of 83.02%for semantic segmentation and accuracy of 95.26%for height estimation.

Remote sensing of air pollution incorporating integrated-path differential-absorption and coherent-Doppler lidar

An innovative complex lidar system deployed on an airborne rotorcraft platform for remote sensing of atmospheric pollution is proposed and demonstrated.The system incorporates integrated-path differential absorption lidar(DIAL) and coherent-doppler lidar(CDL) techniques using a dual tunable TEA CO_(2)laser in the 9—11 μm band and a 1.55 μm fiber laser.By combining the principles of differential absorption detection and pulsed coherent detection,the system enables agile and remote sensing of atmospheric pollution.Extensive static tests validate the system’s real-time detection capabilities,including the measurement of concentration-path-length product(CL),front distance,and path wind speed of air pollution plumes over long distances exceeding 4 km.Flight experiments is conducted with the helicopter.Scanning of the pollutant concentration and the wind field is carried out in an approximately 1 km slant range over scanning angle ranges from 45°to 65°,with a radial resolution of 30 m and10 s.The test results demonstrate the system’s ability to spatially map atmospheric pollution plumes and predict their motion and dispersion patterns,thereby ensuring the protection of public safety.

Building Feedback-Regulation System Through Atomic Design for Highly Active SO_(2)Sensing

Reasonably constructing an atomic interface is pronouncedly essential for surface-related gas-sensing reaction.Herein,we present an ingen-ious feedback-regulation system by changing the interactional mode between single Pt atoms and adjacent S species for high-efficiency SO_(2)sensing.We found that the single Pt sites on the MoS_(2)surface can induce easier volatiliza-tion of adjacent S species to activate the whole inert S plane.Reversely,the activated S species can provide a feedback role in tailoring the antibonding-orbital electronic occupancy state of Pt atoms,thus creating a combined system involving S vacancy-assisted single Pt sites(Pt-Vs)to synergistically improve the adsorption ability of SO_(2)gas molecules.Further-more,in situ Raman,ex situ X-ray photoelectron spectroscopy testing and density functional theory analysis demonstrate the intact feedback-regulation system can expand the electron transfer path from single Pt sites to whole Pt-MoS_(2)supports in SO_(2)gas atmosphere.Equipped with wireless-sensing modules,the final Pt1-MoS_(2)-def sensors array can further realize real-time monitoring of SO_(2)levels and cloud-data storage for plant growth.Such a fundamental understanding of the intrinsic link between atomic interface and sensing mechanism is thus expected to broaden the rational design of highly effective gas sensors.

Tailoring MXene Thickness and Functionalization for Enhanced Room‑Temperature Trace NO_(2) Sensing

In this study,precise control over the thickness and termination of Ti3C2TX MXene flakes is achieved to enhance their electrical properties,environmental stability,and gas-sensing performance.Utilizing a hybrid method involving high-pressure processing,stirring,and immiscible solutions,sub-100 nm MXene flake thickness is achieved within the MXene film on the Si-wafer.Functionalization control is achieved by defunctionalizing MXene at 650℃ under vacuum and H2 gas in a CVD furnace,followed by refunctionalization with iodine and bromine vaporization from a bubbler attached to the CVD.Notably,the introduction of iodine,which has a larger atomic size,lower electronegativity,reduce shielding effect,and lower hydrophilicity(contact angle:99°),profoundly affecting MXene.It improves the surface area(36.2 cm^(2) g^(-1)),oxidation stability in aqueous/ambient environments(21 days/80 days),and film conductivity(749 S m^(-1)).Additionally,it significantly enhances the gas-sensing performance,including the sensitivity(0.1119Ωppm^(-1)),response(0.2% and 23%to 50 ppb and 200 ppm NO_(2)),and response/recovery times(90/100 s).The reduced shielding effect of the–I-terminals and the metallic characteristics of MXene enhance the selectivity of I-MXene toward NO2.This approach paves the way for the development of stable and high-performance gas-sensing two-dimensional materials with promising prospects for future studies.

A Hand Features Based Fusion Recognition Network with Enhancing Multi-Modal Correlation

Fusing hand-based features in multi-modal biometric recognition enhances anti-spoofing capabilities.Additionally,it leverages inter-modal correlation to enhance recognition performance.Concurrently,the robustness and recognition performance of the system can be enhanced through judiciously leveraging the correlation among multimodal features.Nevertheless,two issues persist in multi-modal feature fusion recognition:Firstly,the enhancement of recognition performance in fusion recognition has not comprehensively considered the inter-modality correlations among distinct modalities.Secondly,during modal fusion,improper weight selection diminishes the salience of crucial modal features,thereby diminishing the overall recognition performance.To address these two issues,we introduce an enhanced DenseNet multimodal recognition network founded on feature-level fusion.The information from the three modalities is fused akin to RGB,and the input network augments the correlation between modes through channel correlation.Within the enhanced DenseNet network,the Efficient Channel Attention Network(ECA-Net)dynamically adjusts the weight of each channel to amplify the salience of crucial information in each modal feature.Depthwise separable convolution markedly reduces the training parameters and further enhances the feature correlation.Experimental evaluations were conducted on four multimodal databases,comprising six unimodal databases,including multispectral palmprint and palm vein databases from the Chinese Academy of Sciences.The Equal Error Rates(EER)values were 0.0149%,0.0150%,0.0099%,and 0.0050%,correspondingly.In comparison to other network methods for palmprint,palm vein,and finger vein fusion recognition,this approach substantially enhances recognition performance,rendering it suitable for high-security environments with practical applicability.The experiments in this article utilized amodest sample database comprising 200 individuals.The subsequent phase involves preparing for the extension of the method to larger databases.

Preparation of single atom catalysts for high sensitive gas sensing

Single atom catalysts(SACs)have garnered significant attention in the field of catalysis over the past decade due to their exceptional atom utilization efficiency and distinct physical and chemical properties.For the semiconductor-based electrical gas sensor,the core is the catalysis process of target gas molecules on the sensitive materials.In this context,the SACs offer great potential for highly sensitive and selective gas sensing,however,only some of the bubbles come to the surface.To facilitate practical applications,we present a comprehensive review of the preparation strategies for SACs,with a focus on overcoming the challenges of aggregation and low loading.Extensive research efforts have been devoted to investigating the gas sensing mechanism,exploring sensitive materials,optimizing device structures,and refining signal post-processing techniques.Finally,the challenges and future perspectives on the SACs based gas sensing are presented.

A Comprehensive Survey on Deep Learning Multi-Modal Fusion:Methods,Technologies and Applications

Multi-modal fusion technology gradually become a fundamental task in many fields,such as autonomous driving,smart healthcare,sentiment analysis,and human-computer interaction.It is rapidly becoming the dominant research due to its powerful perception and judgment capabilities.Under complex scenes,multi-modal fusion technology utilizes the complementary characteristics of multiple data streams to fuse different data types and achieve more accurate predictions.However,achieving outstanding performance is challenging because of equipment performance limitations,missing information,and data noise.This paper comprehensively reviews existing methods based onmulti-modal fusion techniques and completes a detailed and in-depth analysis.According to the data fusion stage,multi-modal fusion has four primary methods:early fusion,deep fusion,late fusion,and hybrid fusion.The paper surveys the three majormulti-modal fusion technologies that can significantly enhance the effect of data fusion and further explore the applications of multi-modal fusion technology in various fields.Finally,it discusses the challenges and explores potential research opportunities.Multi-modal tasks still need intensive study because of data heterogeneity and quality.Preserving complementary information and eliminating redundant information between modalities is critical in multi-modal technology.Invalid data fusion methods may introduce extra noise and lead to worse results.This paper provides a comprehensive and detailed summary in response to these challenges.

Towards trustworthy multi-modal motion prediction:Holistic evaluation and interpretability of outputs

Predicting the motion of other road agents enables autonomous vehicles to perform safe and efficient path planning.This task is very complex,as the behaviour of road agents depends on many factors and the number of possible future trajectories can be consid-erable(multi-modal).Most prior approaches proposed to address multi-modal motion prediction are based on complex machine learning systems that have limited interpret-ability.Moreover,the metrics used in current benchmarks do not evaluate all aspects of the problem,such as the diversity and admissibility of the output.The authors aim to advance towards the design of trustworthy motion prediction systems,based on some of the re-quirements for the design of Trustworthy Artificial Intelligence.The focus is on evaluation criteria,robustness,and interpretability of outputs.First,the evaluation metrics are comprehensively analysed,the main gaps of current benchmarks are identified,and a new holistic evaluation framework is proposed.Then,a method for the assessment of spatial and temporal robustness is introduced by simulating noise in the perception system.To enhance the interpretability of the outputs and generate more balanced results in the proposed evaluation framework,an intent prediction layer that can be attached to multi-modal motion prediction models is proposed.The effectiveness of this approach is assessed through a survey that explores different elements in the visualisation of the multi-modal trajectories and intentions.The proposed approach and findings make a significant contribution to the development of trustworthy motion prediction systems for autono-mous vehicles,advancing the field towards greater safety and reliability.

Fine-Grained Ship Recognition Based on Visible and Near-Infrared Multimodal Remote Sensing Images: Dataset,Methodology and Evaluation

Fine-grained recognition of ships based on remote sensing images is crucial to safeguarding maritime rights and interests and maintaining national security.Currently,with the emergence of massive high-resolution multi-modality images,the use of multi-modality images for fine-grained recognition has become a promising technology.Fine-grained recognition of multi-modality images imposes higher requirements on the dataset samples.The key to the problem is how to extract and fuse the complementary features of multi-modality images to obtain more discriminative fusion features.The attention mechanism helps the model to pinpoint the key information in the image,resulting in a significant improvement in the model’s performance.In this paper,a dataset for fine-grained recognition of ships based on visible and near-infrared multi-modality remote sensing images has been proposed first,named Dataset for Multimodal Fine-grained Recognition of Ships(DMFGRS).It includes 1,635 pairs of visible and near-infrared remote sensing images divided into 20 categories,collated from digital orthophotos model provided by commercial remote sensing satellites.DMFGRS provides two types of annotation format files,as well as segmentation mask images corresponding to the ship targets.Then,a Multimodal Information Cross-Enhancement Network(MICE-Net)fusing features of visible and near-infrared remote sensing images,has been proposed.In the network,a dual-branch feature extraction and fusion module has been designed to obtain more expressive features.The Feature Cross Enhancement Module(FCEM)achieves the fusion enhancement of the two modal features by making the channel attention and spatial attention work cross-functionally on the feature map.A benchmark is established by evaluating state-of-the-art object recognition algorithms on DMFGRS.MICE-Net conducted experiments on DMFGRS,and the precision,recall,mAP0.5 and mAP0.5:0.95 reached 87%,77.1%,83.8%and 63.9%,respectively.Extensive experiments demonstrate that the proposed MICE-Net has more excellent performance on DMFGRS.Built on lightweight network YOLO,the model has excellent generalizability,and thus has good potential for application in real-life scenarios.

Multi-dimension and multi-modal rolling mill vibration prediction model based on multi-level network fusion

Mill vibration is a common problem in rolling production,which directly affects the thickness accuracy of the strip and may even lead to strip fracture accidents in serious cases.The existing vibration prediction models do not consider the features contained in the data,resulting in limited improvement of model accuracy.To address these challenges,this paper proposes a multi-dimensional multi-modal cold rolling vibration time series prediction model(MDMMVPM)based on the deep fusion of multi-level networks.In the model,the long-term and short-term modal features of multi-dimensional data are considered,and the appropriate prediction algorithms are selected for different data features.Based on the established prediction model,the effects of tension and rolling force on mill vibration are analyzed.Taking the 5th stand of a cold mill in a steel mill as the research object,the innovative model is applied to predict the mill vibration for the first time.The experimental results show that the correlation coefficient(R^(2))of the model proposed in this paper is 92.5%,and the root-mean-square error(RMSE)is 0.0011,which significantly improves the modeling accuracy compared with the existing models.The proposed model is also suitable for the hot rolling process,which provides a new method for the prediction of strip rolling vibration.

Multi-modal knowledge graph inference via media convergence and logic rule

Media convergence works by processing information from different modalities and applying them to different domains.It is difficult for the conventional knowledge graph to utilise multi-media features because the introduction of a large amount of information from other modalities reduces the effectiveness of representation learning and makes knowledge graph inference less effective.To address the issue,an inference method based on Media Convergence and Rule-guided Joint Inference model(MCRJI)has been pro-posed.The authors not only converge multi-media features of entities but also introduce logic rules to improve the accuracy and interpretability of link prediction.First,a multi-headed self-attention approach is used to obtain the attention of different media features of entities during semantic synthesis.Second,logic rules of different lengths are mined from knowledge graph to learn new entity representations.Finally,knowledge graph inference is performed based on representing entities that converge multi-media features.Numerous experimental results show that MCRJI outperforms other advanced baselines in using multi-media features and knowledge graph inference,demonstrating that MCRJI provides an excellent approach for knowledge graph inference with converged multi-media features.

Temperature and Salinity Dual-parameter Sensing Based on Forward Brillouin Scattering in 1060-XP SMF

A novel temperature and salinity discriminative sensing method based on forward Brillouin scattering(FBS)in 1060-XP single-mode fiber(SMF)is proposed.The measured frequency shifts corresponding to different radial acoustic modes in 1060-XP SMF show different sensitivities to temperature and salinity.Based on the new phenomenon that different radial acoustic modes have different frequency shift-temperature and frequency shift-salinity coefficients,we propose a novel method for simultaneously measuring temperature and salinity by measuring the frequency shift changes of two FBS scattering peaks.In a proof-of-concept experiment,the temperature and salinity measurement errors are 0.12℃and 0.29%,respectively.The proposed method for simultaneously measuring temperature and salinity has the potential applications such as ocean surveying,food manufacturing and pharmaceutical engineering.

Research on Multi-modal In-Vehicle Intelligent Personal Assistant Design

Intelligent personal assistants play a pivotal role in in-vehicle systems,significantly enhancing life efficiency,driving safety,and decision-making support.In this study,the multi-modal design elements of intelligent personal assistants within the context of visual,auditory,and somatosensory interactions with drivers were discussed.Their impact on the driver’s psychological state through various modes such as visual imagery,voice interaction,and gesture interaction were explored.The study also introduced innovative designs for in-vehicle intelligent personal assistants,incorporating design principles such as driver-centricity,prioritizing passenger safety,and utilizing timely feedback as a criterion.Additionally,the study employed design methods like driver behavior research and driving situation analysis to enhance the emotional connection between drivers and their vehicles,ultimately improving driver satisfaction and trust.

Generative Multi-Modal Mutual Enhancement Video Semantic Communications

Recently,there have been significant advancements in the study of semantic communication in single-modal scenarios.However,the ability to process information in multi-modal environments remains limited.Inspired by the research and applications of natural language processing across different modalities,our goal is to accurately extract frame-level semantic information from videos and ultimately transmit high-quality videos.Specifically,we propose a deep learning-basedMulti-ModalMutual Enhancement Video Semantic Communication system,called M3E-VSC.Built upon a VectorQuantized Generative AdversarialNetwork(VQGAN),our systemaims to leverage mutual enhancement among different modalities by using text as the main carrier of transmission.With it,the semantic information can be extracted fromkey-frame images and audio of the video and performdifferential value to ensure that the extracted text conveys accurate semantic information with fewer bits,thus improving the capacity of the system.Furthermore,a multi-frame semantic detection module is designed to facilitate semantic transitions during video generation.Simulation results demonstrate that our proposed model maintains high robustness in complex noise environments,particularly in low signal-to-noise ratio conditions,significantly improving the accuracy and speed of semantic transmission in video communication by approximately 50 percent.

CrossFormer Embedding DeepLabv3+ for Remote Sensing Images Semantic Segmentation

High-resolution remote sensing image segmentation is a challenging task. In urban remote sensing, the presenceof occlusions and shadows often results in blurred or invisible object boundaries, thereby increasing the difficultyof segmentation. In this paper, an improved network with a cross-region self-attention mechanism for multi-scalefeatures based onDeepLabv3+is designed to address the difficulties of small object segmentation and blurred targetedge segmentation. First,we use CrossFormer as the backbone feature extraction network to achieve the interactionbetween large- and small-scale features, and establish self-attention associations between features at both large andsmall scales to capture global contextual feature information. Next, an improved atrous spatial pyramid poolingmodule is introduced to establish multi-scale feature maps with large- and small-scale feature associations, andattention vectors are added in the channel direction to enable adaptive adjustment of multi-scale channel features.The proposed networkmodel is validated using the PotsdamandVaihingen datasets. The experimental results showthat, compared with existing techniques, the network model designed in this paper can extract and fuse multiscaleinformation, more clearly extract edge information and small-scale information, and segment boundariesmore smoothly. Experimental results on public datasets demonstrate the superiority of ourmethod compared withseveral state-of-the-art networks.

On-chip quantum NOON state sensing for temperature and humidity

A maximal photon number entangled state,namely NOON state,can be adopted for sensing with a quantum enhancedprecision.In this work,we designed silicon quantum photonic chips containing two types of Mach-Zehnder interferometerswherein the two-photon NOON state,sensing element for temperature or humidity,is generated.Compared with classicallight or single photon case,two-photon NOON state sensing shows a solid enhancement in the sensing resolution andprecision.As the first demonstration of on-chip quantum photonic sensing,it reveals the advantages of photonic chips forhigh integration density,small-size,stability for multiple-parameter sensing serviceability.A higher sensing precision isexpected to beat the standard quantum limit with a higher photon number NOON state.

Remote sensing of quality traits in cereal and arable production systems:A review

Cereal is an essential source of calories and protein for the global population.Accurately predicting cereal quality before harvest is highly desirable in order to optimise management for farmers,grading harvest and categorised storage for enterprises,future trading prices,and policy planning.The use of remote sensing data with extensive spatial coverage demonstrates some potential in predicting crop quality traits.Many studies have also proposed models and methods for predicting such traits based on multiplatform remote sensing data.In this paper,the key quality traits that are of interest to producers and consumers are introduced.The literature related to grain quality prediction was analyzed in detail,and a review was conducted on remote sensing platforms,commonly used methods,potential gaps,and future trends in crop quality prediction.This review recommends new research directions that go beyond the traditional methods and discusses grain quality retrieval and the associated challenges from the perspective of remote sensing data.

Chaotic CS Encryption:An Efficient Image Encryption Algorithm Based on Chebyshev Chaotic System and Compressive Sensing

Images are the most important carrier of human information. Moreover, how to safely transmit digital imagesthrough public channels has become an urgent problem. In this paper, we propose a novel image encryptionalgorithm, called chaotic compressive sensing (CS) encryption (CCSE), which can not only improve the efficiencyof image transmission but also introduce the high security of the chaotic system. Specifically, the proposed CCSEcan fully leverage the advantages of the Chebyshev chaotic system and CS, enabling it to withstand various attacks,such as differential attacks, and exhibit robustness. First, we use a sparse trans-form to sparse the plaintext imageand then use theArnold transformto perturb the image pixels. After that,we elaborate aChebyshev Toeplitz chaoticsensing matrix for CCSE. By using this Toeplitz matrix, the perturbed image is compressed and sampled to reducethe transmission bandwidth and the amount of data. Finally, a bilateral diffusion operator and a chaotic encryptionoperator are used to perturb and expand the image pixels to change the pixel position and value of the compressedimage, and ultimately obtain an encrypted image. Experimental results show that our method can be resistant tovarious attacks, such as the statistical attack and noise attack, and can outperform its current competitors.

Using ontology and rules to retrieve the semantics of disaster remote sensing data

Remote sensing data plays an important role in natural disaster management.However,with the increase of the variety and quantity of remote sensors,the problem of“knowledge barriers”arises when data users in disaster field retrieve remote sensing data.To improve this problem,this paper proposes an ontology and rule based retrieval(ORR)method to retrieve disaster remote sensing data,and this method introduces ontology technology to express earthquake disaster and remote sensing knowledge,on this basis,and realizes the task suitability reasoning of earthquake disaster remote sensing data,mining the semantic relationship between remote sensing metadata and disasters.The prototype system is built according to the ORR method,which is compared with the traditional method,using the ORR method to retrieve disaster remote sensing data can reduce the knowledge requirements of data users in the retrieval process and improve data retrieval efficiency.