Depressive semantic awareness from vlog facial and vocal streams via spatio-temporal transformer

With the rapid growth of information transmission via the Internet,efforts have been made to reduce network load to promote efficiency.One such application is semantic computing,which can extract and process semantic communication.Social media has enabled users to share their current emotions,opinions,and life events through their mobile devices.Notably,people suffering from mental health problems are more willing to share their feelings on social networks.Therefore,it is necessary to extract semantic information from social media(vlog data)to identify abnormal emotional states to facilitate early identification and intervention.Most studies do not consider spatio-temporal information when fusing multimodal information to identify abnormal emotional states such as depression.To solve this problem,this paper proposes a spatio-temporal squeeze transformer method for the extraction of semantic features of depression.First,a module with spatio-temporal data is embedded into the transformer encoder,which is utilized to obtain a representation of spatio-temporal features.Second,a classifier with a voting mechanism is designed to encourage the model to classify depression and non-depression effec-tively.Experiments are conducted on the D-Vlog dataset.The results show that the method is effective,and the accuracy rate can reach 70.70%.This work provides scaffolding for future work in the detection of affect recognition in semantic communication based on social media vlog data.

SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness

Image fusion aims to integrate complementary information in source images to synthesize a fused image comprehensively characterizing the imaging scene. However, existing image fusion algorithms are only applicable to strictly aligned source images and cause severe artifacts in the fusion results when input images have slight shifts or deformations. In addition,the fusion results typically only have good visual effect, but neglect the semantic requirements of high-level vision tasks.This study incorporates image registration, image fusion, and semantic requirements of high-level vision tasks into a single framework and proposes a novel image registration and fusion method, named Super Fusion. Specifically, we design a registration network to estimate bidirectional deformation fields to rectify geometric distortions of input images under the supervision of both photometric and end-point constraints. The registration and fusion are combined in a symmetric scheme, in which while mutual promotion can be achieved by optimizing the naive fusion loss, it is further enhanced by the mono-modal consistent constraint on symmetric fusion outputs. In addition, the image fusion network is equipped with the global spatial attention mechanism to achieve adaptive feature integration. Moreover, the semantic constraint based on the pre-trained segmentation model and Lovasz-Softmax loss is deployed to guide the fusion network to focus more on the semantic requirements of high-level vision tasks. Extensive experiments on image registration, image fusion,and semantic segmentation tasks demonstrate the superiority of our Super Fusion compared to the state-of-the-art alternatives.The source code and pre-trained model are publicly available at https://github.com/Linfeng-Tang/Super Fusion.

A Semantic-Aware Method for Polymorphic Signature Generation

It is difficult for security experts to generate polymorphic signatures by using traditional string mining and matching techniques.A semantic-aware method is presented to generate a kind of two-level signature that includes both polymorphic semantics and string patterns.It first analyzes the characteristics of polymorphic engines and categorizes the data flows into different clusters and then uses static data flow methods to extract invariable semantic instructions.And then,it combines traditional string methods to generate the signature.In comparison with other methods,experimental results show that it may effectively reduce false positives and false negatives.

Building a Distributed Infrastructure for Scalable Triple Stores

Built specifically for the Semantic Web, triple stores are required to accommodate a large number of RDF triples and remain primarily centralized. As triple stores grow and evolve with time, there is a demanding need for scalable techniques to remove resource and performance bottlenecks in such systems. To this end, we propose a fully decentralized peer-to-peer architecture for large scale triple stores in which triples are maintained by individual stakeholders, and a semantics-directed search protocol, mediated by topology reorganization, for locating triples of interest. We test our design through simulations and the results show anticipated improvements over existing techniques for distributed triple stores. In addition to engineering future large scale triple stores, our work will in particular benefit the federation of stand-alone triple stores of today to achieve desired scalability.