A Robust Framework for Multimodal Sentiment Analysis with Noisy Labels Generated from Distributed Data Annotation

Multimodal sentiment analysis utilizes multimodal data such as text,facial expressions and voice to detect people’s attitudes.With the advent of distributed data collection and annotation,we can easily obtain and share such multimodal data.However,due to professional discrepancies among annotators and lax quality control,noisy labels might be introduced.Recent research suggests that deep neural networks(DNNs)will overfit noisy labels,leading to the poor performance of the DNNs.To address this challenging problem,we present a Multimodal Robust Meta Learning framework(MRML)for multimodal sentiment analysis to resist noisy labels and correlate distinct modalities simultaneously.Specifically,we propose a two-layer fusion net to deeply fuse different modalities and improve the quality of the multimodal data features for label correction and network training.Besides,a multiple meta-learner(label corrector)strategy is proposed to enhance the label correction approach and prevent models from overfitting to noisy labels.We conducted experiments on three popular multimodal datasets to verify the superiority of ourmethod by comparing it with four baselines.

A Multimodal Critical Discourse Analysis of Lingnan Cultural Promotional Videos on Social Media

In recent years,more and more directors of culture and tourism have taken part in the promotion of local cultural tourism by cross-dressing,talent shows,and pushing their limits on self-media platforms.This study investigates short videos of Lingnan culture promoted by directors general and deputy directors general of the Culture,Radio,Television,Tourism,and Sports Bureau of counties and cities in Guangdong Province on social media by the method of multimodal critical discourse analysis.The analysis of 33 videos shows that Lingnan culture is a domineering and confident culture,historical culture,graceful and elegant culture,and vibrant and active culture.Domineering and confident culture is embedded in the utterances and behaviors of the directors general or deputy directors general in the video.Historical culture is realized through the conversation with historical figures through time travel.Graceful and elegant culture is constructed in the depiction of sceneries and the depiction of characters’manners.Vibrant and active culture is represented in the depiction of the characters’actional process and analytical process.

A Study of the English Translation of Characteristic Cuisines of the Hexi Corridor from the Perspective of Multimodal Discourse Analysis

Given the diversity of geographical and cultural backgrounds,it is difficult to find precise counterparts of many food-related vocabularies in the unique dietary culture of the Hexi Corridor in the target language,making it challenging to fully convey the essence of its local dietary culture in translation.To achieve the purpose of translation,the theory of multimodal discourse analysis provides an ideal translation approach through the language mode,image mode,sound mode,and the synergistic effect of multimodalities,which can accurately address the deficiencies in the translation of dietary culture.Through this research,it is not only possible to highlight the unique charm of the dietary culture of the Hexi Corridor but also to build a bridge for its international dissemination,promoting cross-border cultural exchanges and mutual understanding.

Multimodal sentiment analysis for social media contents during public emergencies

Purpose:Nowadays,public opinions during public emergencies involve not only textual contents but also contain images.However,the existing works mainly focus on textual contents and they do not provide a satisfactory accuracy of sentiment analysis,lacking the combination of multimodal contents.In this paper,we propose to combine texts and images generated in the social media to perform sentiment analysis.Design/methodology/approach:We propose a Deep Multimodal Fusion Model(DMFM),which combines textual and visual sentiment analysis.We first train word2vec model on a large-scale public emergency corpus to obtain semantic-rich word vectors as the input of textual sentiment analysis.BiLSTM is employed to generate encoded textual embeddings.To fully excavate visual information from images,a modified pretrained VGG16-based sentiment analysis network is used with the best-performed fine-tuning strategy.A multimodal fusion method is implemented to fuse textual and visual embeddings completely,producing predicted labels.Findings:We performed extensive experiments on Weibo and Twitter public emergency datasets,to evaluate the performance of our proposed model.Experimental results demonstrate that the DMFM provides higher accuracy compared with baseline models.The introduction of images can boost the performance of sentiment analysis during public emergencies.Research limitations:In the future,we will test our model in a wider dataset.We will also consider a better way to learn the multimodal fusion information.Practical implications:We build an efficient multimodal sentiment analysis model for the social media contents during public emergencies.Originality/value:We consider the images posted by online users during public emergencies on social platforms.The proposed method can present a novel scope for sentiment analysis during public emergencies and provide the decision support for the government when formulating policies in public emergencies.

Leveraging Vision-Language Pre-Trained Model and Contrastive Learning for Enhanced Multimodal Sentiment Analysis

Multimodal sentiment analysis is an essential area of research in artificial intelligence that combines multiple modes,such as text and image,to accurately assess sentiment.However,conventional approaches that rely on unimodal pre-trained models for feature extraction from each modality often overlook the intrinsic connections of semantic information between modalities.This limitation is attributed to their training on unimodal data,and necessitates the use of complex fusion mechanisms for sentiment analysis.In this study,we present a novel approach that combines a vision-language pre-trained model with a proposed multimodal contrastive learning method.Our approach harnesses the power of transfer learning by utilizing a vision-language pre-trained model to extract both visual and textual representations in a unified framework.We employ a Transformer architecture to integrate these representations,thereby enabling the capture of rich semantic infor-mation in image-text pairs.To further enhance the representation learning of these pairs,we introduce our proposed multimodal contrastive learning method,which leads to improved performance in sentiment analysis tasks.Our approach is evaluated through extensive experiments on two publicly accessible datasets,where we demonstrate its effectiveness.We achieve a significant improvement in sentiment analysis accuracy,indicating the supe-riority of our approach over existing techniques.These results highlight the potential of multimodal sentiment analysis and underscore the importance of considering the intrinsic semantic connections between modalities for accurate sentiment assessment.

Analysis of Emotions Using Multimodal Data: A Case Study

In this case study, we hypothesized that sympathetic nerve activity would be higher during conversation with PALRO robot, and that conversation would result in an increase in cerebral blood flow near the Broca’s area. The facial expressions of a human subject were recorded, and cerebral blood flow and heart rate variability were measured during interactions with the humanoid robot. These multimodal data were time-synchronized to quantitatively verify the change from the resting baseline by testing facial expression analysis, cerebral blood flow, and heart rate variability. In conclusion, this subject indicated that sympathetic nervous activity was dominant, suggesting that the subject may have enjoyed and been excited while talking to the robot (normalized High Frequency < normalized Low Frequency: 0.22 ± 0.16 < 0.78 ± 0.16). Cerebral blood flow values were higher during conversation and in the resting state after the experiment than in the resting state before the experiment. Talking increased cerebral blood flow in the frontal region. As the subject was left-handed, it was confirmed that the right side of the brain, where the Broca’s area is located, was particularly activated (Left < right: 0.15 ± 0.21 < 1.25 ± 0.17). In the sections where a “happy” facial emotion was recognized, the examiner-judged “happy” faces and the MTCNN “happy” results were also generally consistent.

Real-Time Multimodal Biometric Authentication of Human Using Face Feature Analysis

As multimedia data sharing increases,data security in mobile devices and its mechanism can be seen as critical.Biometrics combines the physiological and behavioral qualities of an individual to validate their character in real-time.Humans incorporate physiological attributes like a fingerprint,face,iris,palm print,finger knuckle print,Deoxyribonucleic Acid(DNA),and behavioral qualities like walk,voice,mark,or keystroke.The main goal of this paper is to design a robust framework for automatic face recognition.Scale Invariant Feature Transform(SIFT)and Speeded-up Robust Features(SURF)are employed for face recognition.Also,we propose a modified Gabor Wavelet Transform for SIFT/SURF(GWT-SIFT/GWT-SURF)to increase the recognition accuracy of human faces.The proposed scheme is composed of three steps.First,the entropy of the image is removed using Discrete Wavelet Transform(DWT).Second,the computational complexity of the SIFT/SURF is reduced.Third,the accuracy is increased for authentication by the proposed GWT-SIFT/GWT-SURF algorithm.A comparative analysis of the proposed scheme is done on real-time Olivetti Research Laboratory(ORL)and Poznan University of Technology(PUT)databases.When compared to the traditional SIFT/SURF methods,we verify that the GWT-SIFT achieves the better accuracy of 99.32%and the better approach is the GWT-SURF as the run time of the GWT-SURF for 100 images is 3.4 seconds when compared to the GWT-SIFT which has a run time of 4.9 seconds for 100 images.

Literature Review of Multimodal Discourse Analysis of Textbooks

With the rapid development of modern technology,textbooks have been undergoing great changes from the traditionallypaper-based form to the multimodal form.Multimodality has become a major feature of textbook discourse.Multimodal discourseanalysis is a new approach to discourse analysis based on the semiotic theory of systemic functional linguistics.This paper providesa detailed review of multimodal discourse analysis(hereafter MDA) in modern linguistics and some previous researches on textbookdiscourse.Then it further outlines what have already been done on textbook discourse from the perspective of MDA.

Tobacco Control Intervention:A Comparative Multimodal Discourse Analysis of Video Advertisements in China and Australia

Significant differences in tobacco-control initiatives and achievements between Australia and China have been witnessed in the world,despite the fact that thecountries share the Asia-pacific region and have intimate relationships.The smoking rate in Australia has been on a downward trend since the 1980s and therefore Australia has been portrayed as a“dark market”by global tobacco industry companies.However in China,cigarette smoking is the chief killer that causes deadly health risks to both direct users of tobacco and people exposed to second-hand smoke.Non-communicable diseases due to this have caused great burden to the national medical resources.As a vital intervention tool of mass media campaign,public health advertisements on tobacco-control have played an important role in influencing the cognition and behavior of recipients.This study aims to compare the strategies adopted by the advertisements in Australia and China through corpus-driven multimodal discourse analysis to conduct quantitative analysis and qualitative case study.The results of comparative analysis indicate different pathways of tobacco-control advertisements constructions in the two countries that Australian tobacco-control advertisements achieved better in terms of health communication intervention to encourage cessation among smokers and reduce smoking initiation among potential smokers at the population level.To be compared,Chinese tobacco-control advertisements placed a heavier emphasis on the health hazards caused by second-hand smoking to protect people from the attributable morbidity and mortality.From the perspective of content theme,this study argues that Chinese tobacco-control advertisements should shift the focus to the health damage of smoking to smokers which is more targeted and fundamental.From the perspective of multimodal presentations of the video advertisements,a more prominent demonstration of the health damage is recommended.Moreover,this study argues that cultural-oriented strategies should be developed in the tobacco-control advertisements if considering the cigarette sharing and gifting social phenomenon in the Chinese society.

Multimodal Discourse Analysis-A Corpus-based Study

According to multimodal discourse analysis(MDA),different kinds of semiotic modes work together to conduct meaning.Previous studies on MDA only involve case analysis.Nowadays,scholars tend to establish multimodal corpora to provide a more fuller and comprehensive study on multimodal discourses.In this paper,the author will look at some grand thesis about multimodal corpora and summarize the development of multimodal corpora.

The Media Images of Old Influencers on TikTok:A Multimodal Critical Discourse Analysis

Drawing on the insights from multimodal critical discourse analysis,this study mainly focuses on two super-gerontic celebrities by selecting 60 videos to elucidate how their images are represented through linguistic(e.g.,captions,and hashtags)and other semiotic resources(e.g.,accessories,semiotic tools,and etc.).The findings suggest that images of the old celebrities are characterized as being attractive,wise,stylish,healthy and young,indicating that they are breaking up the stereotypical images of old people who are unattractive,dependent and senile.The phenomenon of old influencers making themselves visible on the social media suggests that they are moving towards successful aging with the backdrop of aging society in China.This present study suggests that multimodal construction of old celebrities can inform relevant practice to combat ageism in the media and helps to empower the old people in the media discourse.In doing so,the pedagogical implication of present study lies in its education about how we view aging in the aging society.

A Multimodal Discourse Analysis of the Publicity Film of Qingdao, the Host City of the SCO Summit

With the innovation of information technology,the communication mode which is dominated by language has changed,and image,sound,color and other semiotic resources are playing an increasingly important role in the meaning construction of com?munication.Thus,the multimodal discourse whose communicative meaning is achieved by a variety of semiotic resources aroused the attention of scholars,and multimodal discourse analysis emerged as required.Therefore,based on Kress and van Leeuwen’s vi?sual grammar,this thesis makes a multimodal discourse analysis(MDA)of the publicity film of Qingdao published before the SCO summit,exploring the meaning-making of visual modality.

A Multimodal Discourse Analysis of the Movie Poster: A.I. Artificial Intelligence from the Perspective of Visual Grammar

This paper carries out a multimodal discourse analysis of a movie poster:A.I.Artificial Intelligence based on the Kress and van Leeuwen’s visual grammar.The result of analysis shows that a movie poster as a special discourse demonstrates greatly the cultural value and artistic charm itself.The analysis might give insight to the design of movie posters.

A Multimodal Analysis of Texts That Represent Stereotypical Gender Identity

This paper is about how stereotypical gender identity is reflected in Chinese gender education textbooks by adopting multimodal analysis from the perspective of essentialism.The stereotypical gender identity mainly refers to femininity and masculinity,and gender difference in everyday communication.

A Multimodal Analysis of Social Class in College English Textbook

As a required course at higher education level,English has permeated the everyday life of Chinese students from all so-cial backgrounds.Social class,deciding one’s access to various social and educational recourses,has a great influence on thelearning process of college students inside and outside the classroom.However,few studies have placed their focus on social classand college English textbooks.Thus,this paper attempts to adopt a multimodal analysis to find how the speaking tasks in collegeEnglish textbook represents the social identity of Chinese college students and to offer pedagogic implications.

Meaning Construction of 2019 Beijing International Horticultural Exposition Emblem from the Perspective of Multimodal Discourse Analysis

The multimodal discourse has been playing a significant role in modern society since its springing-up from the 1990s.Based on the theoretical framework of Visual Grammar(VG)put forward by Kress&van Leeuwen(1996),this study,from the per⁃spective of multimodal discourse analysis,aims at exploring the meaning construction in the emblem of 2019 Beijing International Horticultural Exposition.It is found that an emblem can be interpreted with other modes such as visual image,color and pattern de⁃sign apart from verbal language,which will not only work to participate in the overall meaning construction with mutual interaction,but also analyze how sociocultural background,local customs and humanity spirit are reflected through these forms.

A Multimodal Discourse Analysis of Meaning Construction of Real Estate Posters

Posters have been one of the main research subjects in the Systemic-Functional Linguistics. Nowadays, posters are full of various types of semiotic modes with their wide applications of multimodality. This phenomenon is reflected in real estate posters as well, therefore, this thesis hopes to explore their essence of using multimodality and discussing meaning construction of the real estate posters.

Discourse Analysis of Taylor Swift’s“Look What You Made Me Do”—From the Perspective of Multimodal Metaphor and the Formal Level of SFL Basis

Multimodal discourse analysis is a relatively new area of study and has received great attention these years.Relative studies have also made development in recent years concerning advertisement,classroom education,political speech,and commer⁃cial events,but few have stepped into the music domain.This paper made use of Taylor Swift’s hit single“Look What You Made Me Do”to explore multimodal discourse analysis on music from the perspective of multimodal metaphor and the formal level of sys⁃temic functional linguistics(SFL),and to reveal the relationship between its lyric,tune,and music video,which shows how these multimodal discourse interrelated and coordinated to express the producer’s meaning,attitude,and ideas.

A Multimodal Discourse Analysis of the Cosmetic Advertisement: Takingthe Poster of STEEROREAR as an Example

Because of the high speed of the development of technology,the traditional way to transmit information through language is gradually replaced by the newly appeared multimodal way of analyzing.This paper takes one cosmetic advertisement of STEERO-REAR as an example,aiming to analyze those adopted modalities in the poster.After analyzing,it is found that the elements,like color and position of the participants,contribute to the meaning construction in an organic way.The meaning and purpose of the producer are realized through the construction of representational meaning,interactive meaning and compositional meaning.

The Subtitle Translation of Movie from the Perspective of Multimodal Discourse Analysis

With the development of science and technology,especially the development of digital technology,mankind has entered the age of multimedia,and the mode of human life and communication have undergone profound changes.As a single communicative mode,language has been gradually replaced by complex communicative mode composed of language,image and sound.Multimodal discourse analysis provides a new perspective for discourse analysis composed of a variety of symbols,which can help readers understand how symbols such as images and music work together and form meanings.Firm analysis is often analyzed from the perspective of psychology,aesthetics and other macro aspects,but seldom from the perspective of linguistics.The paper analyzes how the theory of multimodal discourse analysis affects the translation of film by discussing the interaction between film translation and multimodal modes in the film Pride and Prejudice.