Construction of Human Digital Twin Model Based on Multimodal Data and Its Application in Locomotion Mode Identifcation

With the increasing attention to the state and role of people in intelligent manufacturing, there is a strong demand for human-cyber-physical systems (HCPS) that focus on human-robot interaction. The existing intelligent manufacturing system cannot satisfy efcient human-robot collaborative work. However, unlike machines equipped with sensors, human characteristic information is difcult to be perceived and digitized instantly. In view of the high complexity and uncertainty of the human body, this paper proposes a framework for building a human digital twin (HDT) model based on multimodal data and expounds on the key technologies. Data acquisition system is built to dynamically acquire and update the body state data and physiological data of the human body and realize the digital expression of multi-source heterogeneous human body information. A bidirectional long short-term memory and convolutional neural network (BiLSTM-CNN) based network is devised to fuse multimodal human data and extract the spatiotemporal features, and the human locomotion mode identifcation is taken as an application case. A series of optimization experiments are carried out to improve the performance of the proposed BiLSTM-CNN-based network model. The proposed model is compared with traditional locomotion mode identifcation models. The experimental results proved the superiority of the HDT framework for human locomotion mode identifcation.

Analysis of Emotions Using Multimodal Data: A Case Study

In this case study, we hypothesized that sympathetic nerve activity would be higher during conversation with PALRO robot, and that conversation would result in an increase in cerebral blood flow near the Broca’s area. The facial expressions of a human subject were recorded, and cerebral blood flow and heart rate variability were measured during interactions with the humanoid robot. These multimodal data were time-synchronized to quantitatively verify the change from the resting baseline by testing facial expression analysis, cerebral blood flow, and heart rate variability. In conclusion, this subject indicated that sympathetic nervous activity was dominant, suggesting that the subject may have enjoyed and been excited while talking to the robot (normalized High Frequency < normalized Low Frequency: 0.22 ± 0.16 < 0.78 ± 0.16). Cerebral blood flow values were higher during conversation and in the resting state after the experiment than in the resting state before the experiment. Talking increased cerebral blood flow in the frontal region. As the subject was left-handed, it was confirmed that the right side of the brain, where the Broca’s area is located, was particularly activated (Left < right: 0.15 ± 0.21 < 1.25 ± 0.17). In the sections where a “happy” facial emotion was recognized, the examiner-judged “happy” faces and the MTCNN “happy” results were also generally consistent.

Early warning of emerging infectious diseases based on multimodal data

The coronavirus disease 2019 (COVID-19) pandemic has dramatically increased the awareness of emerging infectious diseases. The advancement of multiomics analysis technology has resulted in the development of several databases containing virus information. Several scientists have integrated existing data on viruses to construct phylogenetic trees and predict virus mutation and transmission in different ways, providing prospective technical support for epidemic prevention and control. This review summarized the databases of known emerging infectious viruses and techniques focusing on virus variant forecasting and early warning. It focuses on the multi-dimensional information integration and database construction of emerging infectious viruses, virus mutation spectrum construction and variant forecast model, analysis of the affinity between mutation antigen and the receptor, propagation model of virus dynamic evolution, and monitoring and early warning for variants. As people have suffered from COVID-19 and repeated flu outbreaks, we focused on the research results of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and influenza viruses. This review comprehensively viewed the latest virus research and provided a reference for future virus prevention and control research.

Fusion of color and hallucinated depth features for enhanced multimodal deep learning-based damage segmentation

Recent advances in computer vision and deep learning have shown that the fusion of depth information can significantly enhance the performance of RGB-based damage detection and segmentation models.However,alongside the advantages,depth-sensing also presents many practical challenges.For instance,the depth sensors impose an additional payload burden on the robotic inspection platforms limiting the operation time and increasing the inspection cost.Additionally,some lidar-based depth sensors have poor outdoor performance due to sunlight contamination during the daytime.In this context,this study investigates the feasibility of abolishing depth-sensing at test time without compromising the segmentation performance.An autonomous damage segmentation framework is developed,based on recent advancements in vision-based multi-modal sensing such as modality hallucination(MH)and monocular depth estimation(MDE),which require depth data only during the model training.At the time of deployment,depth data becomes expendable as it can be simulated from the corresponding RGB frames.This makes it possible to reap the benefits of depth fusion without any depth perception per se.This study explored two different depth encoding techniques and three different fusion strategies in addition to a baseline RGB-based model.The proposed approach is validated on computer-generated RGB-D data of reinforced concrete buildings subjected to seismic damage.It was observed that the surrogate techniques can increase the segmentation IoU by up to 20.1%with a negligible increase in the computation cost.Overall,this study is believed to make a positive contribution to enhancing the resilience of critical civil infrastructure.

Brain-inspired multimodal approach for effluent quality prediction using wastewater surface images and water quality data

Efficiently predicting effluent quality through data-driven analysis presents a significant advancement for consistent wastewater treatment operations.In this study,we aimed to develop an integrated method for predicting effluent COD and NH3 levels.We employed a 200 L pilot-scale sequencing batch reactor(SBR)to gather multimodal data from urban sewage over 40 d.Then we collected data on critical parameters like COD,DO,pH,NH_(3),EC,ORP,SS,and water temperature,alongside wastewater surface images,resulting in a data set of approximately 40246 points.Then we proposed a brain-inspired image and temporal fusion model integrated with a CNN-LSTM network(BITF-CL)using this data.This innovative model synergized sewage imagery with water quality data,enhancing prediction accuracy.As a result,the BITF-CL model reduced prediction error by over 23%compared to traditional methods and still performed comparably to conventional techniques even without using DO and SS sensor data.Consequently,this research presents a cost-effective and precise prediction system for sewage treatment,demonstrating the potential of brain-inspired models.

Deep Multimodal Learning and Fusion Based Intelligent Fault Diagnosis Approach

Industrial Internet of Things(IoT)connecting society and industrial systems represents a tremendous and promising paradigm shift.With IoT,multimodal and heterogeneous data from industrial devices can be easily collected,and further analyzed to discover device maintenance and health related potential knowledge behind.IoT data-based fault diagnosis for industrial devices is very helpful to the sustainability and applicability of an IoT ecosystem.But how to efficiently use and fuse this multimodal heterogeneous data to realize intelligent fault diagnosis is still a challenge.In this paper,a novel Deep Multimodal Learning and Fusion(DMLF)based fault diagnosis method is proposed for addressing heterogeneous data from IoT environments where industrial devices coexist.First,a DMLF model is designed by combining a Convolution Neural Network(CNN)and Stacked Denoising Autoencoder(SDAE)together to capture more comprehensive fault knowledge and extract features from different modal data.Second,these multimodal features are seamlessly integrated at a fusion layer and the resulting fused features are further used to train a classifier for recognizing potential faults.Third,a two-stage training algorithm is proposed by combining supervised pre-training and fine-tuning to simplify the training process for deep structure models.A series of experiments are conducted over multimodal heterogeneous data from a gear device to verify our proposed fault diagnosis method.The experimental results show that our method outperforms the benchmarking ones in fault diagnosis accuracy.

O'Hare Airport roadway traffic prediction via data fusion and Gaussian process regression

This study proposes an approach of leveraging information gathered from multiple traffic data sources at different resolutions to obtain approximate inference on the traffic distribution of Chicago's O'Hare Airport area.Specifically,it proposes the ingestion of traffic datasets at different resolutions to build spatiotemporal models for predicting the distribution of traffic volume on the road network.Due to its good adaptability and flexibility for spatiotemporal data,the Gaussian process(GP)regression was employed to provide short-term forecasts using data collected by loop detectors(sensors)and supplemented by telematics data.The GP regression is used to make predictions of the distribution of the proportion of sensor data traffic volume represented by the telematics data for each location of the sensors.Consequently,the fitted GP model can be used to determine the approximate traffic distribution for a testing location outside of the training points.Policymakers in the transportation sector can find the results of this work helpful for making informed decisions relating to current and future transportation conditions in the area.

iCEMIGE: Integration of CEll-morphometrics, MIcrobiome, and GEne biomarker signatures for risk stratification in breast cancers

BACKGROUND The development of precision medicine is essential for personalized treatment and improved clinical outcome,whereas biomarkers are critical for the success of precision therapies.AIM To investigate whether iCEMIGE(integration of CEll-morphometrics,MIcro-biome,and GEne biomarker signatures)improves risk stratification of breast cancer(BC)patients.METHODS We used our recently developed machine learning technique to identify cellular morphometric biomarkers(CMBs)from the whole histological slide images in The Cancer Genome Atlas(TCGA)breast cancer(TCGA-BRCA)cohort.Multivariate Cox regression was used to assess whether cell-morphometrics prognosis score(CMPS)and our previously reported 12-gene expression prognosis score(GEPS)and 15-microbe abundance prognosis score(MAPS)were independent prognostic factors.iCEMIGE was built upon the sparse representation learning technique.The iCEMIGE scoring model performance was measured by the area under the receiver operating characteristic curve compared to CMPS,GEPS,or MAPS alone.Nomogram models were created to predict overall survival(OS)and progress-free survival(PFS)rates at 5-and 10-year in the TCGA-BRCA cohort.RESULTS We identified 39 CMBs that were used to create a CMPS system in BCs.CMPS,GEPS,and MAPS were found to be significantly independently associated with OS.We then established an iCEMIGE scoring system for risk stratification of BC patients.The iGEMIGE score has a significant prognostic value for OS and PFS independent of clinical factors(age,stage,and estrogen and progesterone receptor status)and PAM50-based molecular subtype.Importantly,the iCEMIGE score significantly increased the power to predict OS and PFS compared to CMPS,GEPS,or MAPS alone.CONCLUSION Our study demonstrates a novel and generic artificial intelligence framework for multimodal data integration toward improving prognosis risk stratification of BC patients,which can be extended to other types of cancer.

Design and Implementation of Key-Value Database for Ship Virtual Test Platform Based on Distributed System

The virtual test platform is a vital tool for ship simulation and testing.However,the numerical pool ship virtual test platform is a complex system that comprises multiple heterogeneous data types,such as relational data,files,text,images,and animations.The analysis,evaluation,and decision-making processes heavily depend on data,which continue to increase in size and complexity.As a result,there is an increasing need for a distributed database system to manage these data.In this paper,we propose a Key-Value database based on a distributed system that can operate on any type of data,regardless of its size or type.This database architecture supports class column storage and load balancing and optimizes the efficiency of I/O bandwidth and CPU resource utilization.Moreover,it is specif-ically designed to handle the storage and access of largefiles.Additionally,we propose a multimodal data fusion mechanism that can connect various descrip-tions of the same substance,enabling the fusion and retrieval of heterogeneous multimodal data to facilitate data analysis.Our approach focuses on indexing and storage,and we compare our solution with Redis,MongoDB,and MySQL through experiments.We demonstrate the performance,scalability,and reliability of our proposed database system while also analysing its architecture’s defects and providing optimization solutions and future research directions.In conclu-sion,our database system provides an efficient and reliable solution for the data management of the virtual test platform of numerical pool ships.