A Modified CycleGAN for Multi-Organ Ultrasound Image Enhancement via Unpaired Pre-Training

Handheld ultrasound devices are known for their portability and affordability,making them widely utilized in underdeveloped areas and community healthcare for rapid diagnosis and early screening.However,the image quality of handheld ultrasound devices is not always satisfactory due to the limited equipment size,which hinders accurate diagnoses by doctors.At the same time,paired ultrasound images are difficult to obtain from the clinic because imaging process is complicated.Therefore,we propose a modified cycle generative adversarial network(cycleGAN) for ultrasound image enhancement from multiple organs via unpaired pre-training.We introduce an ultrasound image pre-training method that does not require paired images,alleviating the requirement for large-scale paired datasets.We also propose an enhanced block with different structures in the pre-training and fine-tuning phases,which can help achieve the goals of different training phases.To improve the robustness of the model,we add Gaussian noise to the training images as data augmentation.Our approach is effective in obtaining the best quantitative evaluation results using a small number of parameters and less training costs to improve the quality of handheld ultrasound devices.

Multimodal fusion recognition for digital twin

The digital twin is the concept of transcending reality,which is the reverse feedback from the real physical space to the virtual digital space.People hold great prospects for this emerging technology.In order to realize the upgrading of the digital twin industrial chain,it is urgent to introduce more modalities,such as vision,haptics,hearing and smell,into the virtual digital space,which assists physical entities and virtual objects in creating a closer connection.Therefore,perceptual understanding and object recognition have become an urgent hot topic in the digital twin.Existing surface material classification schemes often achieve recognition through machine learning or deep learning in a single modality,ignoring the complementarity between multiple modalities.In order to overcome this dilemma,we propose a multimodal fusion network in our article that combines two modalities,visual and haptic,for surface material recognition.On the one hand,the network makes full use of the potential correlations between multiple modalities to deeply mine the modal semantics and complete the data mapping.On the other hand,the network is extensible and can be used as a universal architecture to include more modalities.Experiments show that the constructed multimodal fusion network can achieve 99.42%classification accuracy while reducing complexity.

Choosing Multimodal Freight Mix: An Integrated Multi-Objective Multicriteria Approach

Multimodal freight transportation emerges as the go-to strategy for cost-effectively and sustainably moving goods over long distances. In a multimodal freight system, where a single contract includes various transportation methods, businesses aiming for economic success must make well-informed decisions about which modes of transport to use. These decisions prioritize secure deliveries, competitive cost advantages, and the minimization of environmental footprints associated with transportation-related pollution. Within the dynamic landscape of logistics innovation, various multicriteria decision-making (MCDM) approaches empower businesses to evaluate freight transport options thoroughly. In this study, we utilize a case study to demonstrate the application of the Technique for Order Preference by Similarity to the Ideal Solution (TOPSIS) algorithm for MCDM decision-making in freight mode selection. We further enhance the TOPSIS framework by integrating the entropy weight coefficient method. This enhancement aids in assigning precise weights to each criterion involved in mode selection, leading to a more reliable decision-making process. The proposed model provides cost-effective and timely deliveries, minimizing environmental footprint and meeting consumers’ needs. Our findings reveal that freight carbon footprint is the primary concern, followed by freight cost, time sensitivity, and service reliability. The study identifies the combination of Rail/Truck as the ideal mode of transport and containers in flat cars (COFC) as the next best option for the selected case. The proposed algorithm, incorporating the enhanced TOPSIS framework, benefits companies navigating the complexities of multimodal transport. It empowers making more strategic and informed transportation decisions. This demonstration will be increasingly valuable as companies navigate the ever-growing trade within the global supply chains.

Multimodal Social Media Fake News Detection Based on Similarity Inference and Adversarial Networks

As social networks become increasingly complex, contemporary fake news often includes textual descriptionsof events accompanied by corresponding images or videos. Fake news in multiple modalities is more likely tocreate a misleading perception among users. While early research primarily focused on text-based features forfake news detection mechanisms, there has been relatively limited exploration of learning shared representationsin multimodal (text and visual) contexts. To address these limitations, this paper introduces a multimodal modelfor detecting fake news, which relies on similarity reasoning and adversarial networks. The model employsBidirectional Encoder Representation from Transformers (BERT) and Text Convolutional Neural Network (Text-CNN) for extracting textual features while utilizing the pre-trained Visual Geometry Group 19-layer (VGG-19) toextract visual features. Subsequently, the model establishes similarity representations between the textual featuresextracted by Text-CNN and visual features through similarity learning and reasoning. Finally, these features arefused to enhance the accuracy of fake news detection, and adversarial networks have been employed to investigatethe relationship between fake news and events. This paper validates the proposed model using publicly availablemultimodal datasets from Weibo and Twitter. Experimental results demonstrate that our proposed approachachieves superior performance on Twitter, with an accuracy of 86%, surpassing traditional unimodalmodalmodelsand existing multimodal models. In contrast, the overall better performance of our model on the Weibo datasetsurpasses the benchmark models across multiple metrics. The application of similarity reasoning and adversarialnetworks in multimodal fake news detection significantly enhances detection effectiveness in this paper. However,current research is limited to the fusion of only text and image modalities. Future research directions should aimto further integrate features fromadditionalmodalities to comprehensively represent themultifaceted informationof fake news.

Multiobjective Differential Evolution for Higher-Dimensional Multimodal Multiobjective Optimization

In multimodal multiobjective optimization problems(MMOPs),there are several Pareto optimal solutions corre-sponding to the identical objective vector.This paper proposes a new differential evolution algorithm to solve MMOPs with higher-dimensional decision variables.Due to the increase in the dimensions of decision variables in real-world MMOPs,it is diffi-cult for current multimodal multiobjective optimization evolu-tionary algorithms(MMOEAs)to find multiple Pareto optimal solutions.The proposed algorithm adopts a dual-population framework and an improved environmental selection method.It utilizes a convergence archive to help the first population improve the quality of solutions.The improved environmental selection method enables the other population to search the remaining decision space and reserve more Pareto optimal solutions through the information of the first population.The combination of these two strategies helps to effectively balance and enhance conver-gence and diversity performance.In addition,to study the per-formance of the proposed algorithm,a novel set of multimodal multiobjective optimization test functions with extensible decision variables is designed.The proposed MMOEA is certified to be effective through comparison with six state-of-the-art MMOEAs on the test functions.

Effect of different anesthetic modalities with multimodal analgesia on postoperative pain level in colorectal tumor patients

BACKGROUND According to clinical data,a significant percentage of patients experience pain after surgery,highlighting the importance of alleviating postoperative pain.The current approach involves intravenous self-control analgesia,often utilizing opioid analgesics such as morphine,sufentanil,and fentanyl.Surgery for colo-rectal cancer typically involves general anesthesia.Therefore,optimizing anes-thetic management and postoperative analgesic programs can effectively reduce perioperative stress and enhance postoperative recovery.The study aims to analyze the impact of different anesthesia modalities with multimodal analgesia on patients'postoperative pain.AIM To explore the effects of different anesthesia methods coupled with multi-mode analgesia on postoperative pain in patients with colorectal cancer.METHODS Following the inclusion criteria and exclusion criteria,a total of 126 patients with colorectal cancer admitted to our hospital from January 2020 to December 2022 were included,of which 63 received general anesthesia coupled with multi-mode labor pain and were set as the control group,and 63 received general anesthesia associated with epidural anesthesia coupled with multi-mode labor pain and were set as the research group.After data collection,the effects of postoperative analgesia,sedation,and recovery were compared.RESULTS Compared to the control group,the research group had shorter recovery times for orientation,extubation,eye-opening,and spontaneous respiration(P<0.05).The research group also showed lower Visual analog scale scores at 24 h and 48 h,higher Ramany scores at 6 h and 12 h,and improved cognitive function at 24 h,48 h,and 72 h(P<0.05).Additionally,interleukin-6 and interleukin-10 levels were significantly reduced at various time points in the research group compared to the control group(P<0.05).Levels of CD3+,CD4+,and CD4+/CD8+were also lower in the research group at multiple time points(P<0.05).CONCLUSION For patients with colorectal cancer,general anesthesia coupled with epidural anesthesia and multi-mode analgesia can achieve better postoperative analgesia and sedation effects,promote postoperative rehabilitation of patients,improve inflammatory stress and immune status,and have higher safety.

An Immune-Inspired Approach with Interval Allocation in Solving Multimodal Multi-Objective Optimization Problems with Local Pareto Sets

In practical engineering,multi-objective optimization often encounters situations where multiple Pareto sets(PS)in the decision space correspond to the same Pareto front(PF)in the objective space,known as Multi-Modal Multi-Objective Optimization Problems(MMOP).Locating multiple equivalent global PSs poses a significant challenge in real-world applications,especially considering the existence of local PSs.Effectively identifying and locating both global and local PSs is a major challenge.To tackle this issue,we introduce an immune-inspired reproduction strategy designed to produce more offspring in less crowded,promising regions and regulate the number of offspring in areas that have been thoroughly explored.This approach achieves a balanced trade-off between exploration and exploitation.Furthermore,we present an interval allocation strategy that adaptively assigns fitness levels to each antibody.This strategy ensures a broader survival margin for solutions in their initial stages and progressively amplifies the differences in individual fitness values as the population matures,thus fostering better population convergence.Additionally,we incorporate a multi-population mechanism that precisely manages each subpopulation through the interval allocation strategy,ensuring the preservation of both global and local PSs.Experimental results on 21 test problems,encompassing both global and local PSs,are compared with eight state-of-the-art multimodal multi-objective optimization algorithms.The results demonstrate the effectiveness of our proposed algorithm in simultaneously identifying global Pareto sets and locally high-quality PSs.

A Robust Framework for Multimodal Sentiment Analysis with Noisy Labels Generated from Distributed Data Annotation

Multimodal sentiment analysis utilizes multimodal data such as text,facial expressions and voice to detect people’s attitudes.With the advent of distributed data collection and annotation,we can easily obtain and share such multimodal data.However,due to professional discrepancies among annotators and lax quality control,noisy labels might be introduced.Recent research suggests that deep neural networks(DNNs)will overfit noisy labels,leading to the poor performance of the DNNs.To address this challenging problem,we present a Multimodal Robust Meta Learning framework(MRML)for multimodal sentiment analysis to resist noisy labels and correlate distinct modalities simultaneously.Specifically,we propose a two-layer fusion net to deeply fuse different modalities and improve the quality of the multimodal data features for label correction and network training.Besides,a multiple meta-learner(label corrector)strategy is proposed to enhance the label correction approach and prevent models from overfitting to noisy labels.We conducted experiments on three popular multimodal datasets to verify the superiority of ourmethod by comparing it with four baselines.

Enhancing Cross-Lingual Image Description: A Multimodal Approach for Semantic Relevance and Stylistic Alignment

Cross-lingual image description,the task of generating image captions in a target language from images and descriptions in a source language,is addressed in this study through a novel approach that combines neural network models and semantic matching techniques.Experiments conducted on the Flickr8k and AraImg2k benchmark datasets,featuring images and descriptions in English and Arabic,showcase remarkable performance improvements over state-of-the-art methods.Our model,equipped with the Image&Cross-Language Semantic Matching module and the Target Language Domain Evaluation module,significantly enhances the semantic relevance of generated image descriptions.For English-to-Arabic and Arabic-to-English cross-language image descriptions,our approach achieves a CIDEr score for English and Arabic of 87.9%and 81.7%,respectively,emphasizing the substantial contributions of our methodology.Comparative analyses with previous works further affirm the superior performance of our approach,and visual results underscore that our model generates image captions that are both semantically accurate and stylistically consistent with the target language.In summary,this study advances the field of cross-lingual image description,offering an effective solution for generating image captions across languages,with the potential to impact multilingual communication and accessibility.Future research directions include expanding to more languages and incorporating diverse visual and textual data sources.

Multimodal imaging diagnosis and analysis of prognostic factors in patients with adult-onset Coats disease

AIM:To describe the multimodal imaging features,treatment,and outcomes of patients diagnosed with adultonset Coats disease.METHODS:This retrospective study included patients first diagnosed with Coats disease at≥18 years of age between September 2017 and September 2021.Some patients received anti-vascular endothelial growth factor(VEGF)therapy(conbercept,0.5 mg)as the initial treatment,which was combined with laser photocoagulation as needed.All the patients underwent best corrected visual acuity(BCVA)and intraocular pressure examinations,fundus color photography,spontaneous fluorescence tests,fundus fluorescein angiography,optical coherence tomography(OCT),OCT angiography,and other examinations.BCVA alterations and multimodal image findings in the affected eyes following treatment were compared and the prognostic factors were analyzed.RESULTS:The study included 15 patients who were aged 24-72(57.33±12.61)y at presentation.Systemic hypertension was the most common associated systemic condition,occurring in 13(86.7%)patients.Baseline BCVA ranged from 2.0 to 5.0(4.0±1.1),which showed improvement following treatment(4.2±1.0).Multimodal imaging revealed retinal telangiectasis in 13 patients(86.7%),patchy hemorrhage in 5 patients(33.3%),and stage 2B disease(Shield’s staging criteria)in 11 patients(73.3%).OCT revealed that the baseline central macular thickness(CMT)ranged from 129 to 964μm(473.0±230.1μm),with 13 patients(86.7%)exhibiting a baseline CMT exceeding 250μm.Furthermore,8 patients(53.3%)presented with an epiretinal membrane at baseline or during follow-up.Hyper-reflective scars were observed on OCT in five patients(33.3%)with poor visual prognosis.Vision deteriorated in one patient who did not receive treatment.Final vision was stable in three patients who received laser treatment,whereas improvement was observed in one of two patients who received anti-VEGF therapy alone.In addition,8 of 9 patients(88.9%)who received laser treatment and conbercept exhibited stable or improved BCVA.CONCLUSION:Multimodal imaging can help diagnose adult-onset Coats disease.Anti-VEGF treatment combined with laser therapy can be an option for improving or maintaining BCVA and resolving macular edema.The final visual outcome depends on macular involvement and the disease stage.

Audio-Text Multimodal Speech Recognition via Dual-Tower Architecture for Mandarin Air Traffic Control Communications

In air traffic control communications (ATCC), misunderstandings between pilots and controllers could result in fatal aviation accidents. Fortunately, advanced automatic speech recognition technology has emerged as a promising means of preventing miscommunications and enhancing aviation safety. However, most existing speech recognition methods merely incorporate external language models on the decoder side, leading to insufficient semantic alignment between speech and text modalities during the encoding phase. Furthermore, it is challenging to model acoustic context dependencies over long distances due to the longer speech sequences than text, especially for the extended ATCC data. To address these issues, we propose a speech-text multimodal dual-tower architecture for speech recognition. It employs cross-modal interactions to achieve close semantic alignment during the encoding stage and strengthen its capabilities in modeling auditory long-distance context dependencies. In addition, a two-stage training strategy is elaborately devised to derive semantics-aware acoustic representations effectively. The first stage focuses on pre-training the speech-text multimodal encoding module to enhance inter-modal semantic alignment and aural long-distance context dependencies. The second stage fine-tunes the entire network to bridge the input modality variation gap between the training and inference phases and boost generalization performance. Extensive experiments demonstrate the effectiveness of the proposed speech-text multimodal speech recognition method on the ATCC and AISHELL-1 datasets. It reduces the character error rate to 6.54% and 8.73%, respectively, and exhibits substantial performance gains of 28.76% and 23.82% compared with the best baseline model. The case studies indicate that the obtained semantics-aware acoustic representations aid in accurately recognizing terms with similar pronunciations but distinctive semantics. The research provides a novel modeling paradigm for semantics-aware speech recognition in air traffic control communications, which could contribute to the advancement of intelligent and efficient aviation safety management.

FusionNN:A Semantic Feature Fusion Model Based on Multimodal for Web Anomaly Detection

With the rapid development of the mobile communication and the Internet,the previous web anomaly detectionand identificationmodels were built relying on security experts’empirical knowledge and attack features.Althoughthis approach can achieve higher detection performance,it requires huge human labor and resources to maintainthe feature library.In contrast,semantic feature engineering can dynamically discover new semantic featuresand optimize feature selection by automatically analyzing the semantic information contained in the data itself,thus reducing dependence on prior knowledge.However,current semantic features still have the problem ofsemantic expression singularity,as they are extracted from a single semantic mode such as word segmentation,character segmentation,or arbitrary semantic feature extraction.This paper extracts features of web requestsfrom dual semantic granularity,and proposes a semantic feature fusion method to solve the above problems.Themethod first preprocesses web requests,and extracts word-level and character-level semantic features of URLs viaconvolutional neural network(CNN),respectively.By constructing three loss functions to reduce losses betweenfeatures,labels and categories.Experiments on the HTTP CSIC 2010,Malicious URLs and HttpParams datasetsverify the proposedmethod.Results show that compared withmachine learning,deep learningmethods and BERTmodel,the proposed method has better detection performance.And it achieved the best detection rate of 99.16%in the dataset HttpParams.

A deep multimodal fusion and multitasking trajectory prediction model for typhoon trajectory prediction to reduce flight scheduling cancellation

Natural events have had a significant impact on overall flight activity,and the aviation industry plays a vital role in helping society cope with the impact of these events.As one of the most impactful weather typhoon seasons appears and continues,airlines operating in threatened areas and passengers having travel plans during this time period will pay close attention to the development of tropical storms.This paper proposes a deep multimodal fusion and multitasking trajectory prediction model that can improve the reliability of typhoon trajectory prediction and reduce the quantity of flight scheduling cancellation.The deep multimodal fusion module is formed by deep fusion of the feature output by multiple submodal fusion modules,and the multitask generation module uses longitude and latitude as two related tasks for simultaneous prediction.With more dependable data accuracy,problems can be analysed rapidly and more efficiently,enabling better decision-making with a proactive versus reactive posture.When multiple modalities coexist,features can be extracted from them simultaneously to supplement each other’s information.An actual case study,the typhoon Lichma that swept China in 2019,has demonstrated that the algorithm can effectively reduce the number of unnecessary flight cancellations compared to existing flight scheduling and assist the new generation of flight scheduling systems under extreme weather.

Multimodal imaging in the diagnosis of bone giant cell tumors:A retrospective study

BACKGROUND Giant cell tumor of bone is a locally aggressive and rarely metastasizing tumor,and also a potential malignant tumor that may develop into a primary malignant giant cell tumor.AIM To evaluate the role of multimodal imaging in the diagnosis of giant cell tumors of bone.METHODS The data of 32 patients with giant cell tumor of bone confirmed by core-needle biopsy or surgical pathology at our hospital between March 2018 and March 2023 were retrospectively selected.All the patients with giant cell tumors of the bone were examined by X-ray,computed tomography(CT)and magnetic resonance imaging(MRI),and 7 of them were examined by positron emission tomography(PET)-CT.RESULTS X-ray imaging can provide overall information on giant cell tumor lesions.CT and MRI can reveal the characteristics of the internal structure of the tumor as well as the adjacent relationships of the tumor,and these methods have unique advantages for diagnosing tumors and determining the scope of surgery.PET-CT can detect small lesions and is highly valuable for identifying benign and malignant tumors to aid in the early diagnosis of metastasis.CONCLUSION Multimodal imaging plays an important role in the diagnosis of giant cell tumor of bone and can provide a reference for the treatment of giant cell tumors.

Nomogram based on multimodal magnetic resonance combined with B7-H3mRNA for preoperative lymph node prediction in esophagus cancer

Accurate preoperative prediction of lymph node metastasis(LNM)in esophageal cancer(EC)patients is of crucial clinical significance for treatment planning and prognosis.AIM To develop a clinical radiomics nomogram that can predict the preoperative lymph node(LN)status in EC patients.METHODS A total of 32 EC patients confirmed by clinical pathology(who underwent surgical treatment)were included.Real-time fluorescent quantitative reverse transcription-polymerase chain reaction was used to detect the expression of B7-H3 mRNA in EC tissue obtained during preoperative gastroscopy,and its correlation with LNM was analyzed.Radiomics features were extracted from multi-modal magnetic resonance imaging of EC using Pyradiomics in Python.Feature extraction,data dimensionality reduction,and feature selection were performed using XGBoost model and leave-one-out cross-validation.Multivariable logistic regression analysis was used to establish the prediction model,which included radiomics features,LN status from computed tomography(CT)reports,and B7-H3 mRNA expression,represented by a radiomics nomogram.Receiver operating characteristic area under the curve(AUC)and decision curve analysis(DCA)were used to evaluate the predictive performance and clinical application value of the model.RESULTS The relative expression of B7-H3 mRNA in EC patients with LNM was higher than in those without metastasis,and the difference was statistically significant(P<0.05).The AUC value in the receiver operating characteristic(ROC)curve was 0.718(95%CI:0.528-0.907),with a sensitivity of 0.733 and specificity of 0.706,indicating good diagnostic performance.The individualized clinical prediction nomogram included radiomics features,LN status from CT reports,and B7-H3 mRNA expression.The ROC curve demonstrated good diagnostic value,with an AUC value of 0.765(95%CI:0.598-0.931),sensitivity of 0.800,and specificity of 0.706.DCA indicated the practical value of the radiomics nomogram in clinical practice.CONCLUSION This study developed a radiomics nomogram that includes radiomics features,LN status from CT reports,and B7-H3 mRNA expression,enabling convenient preoperative individualized prediction of LNM in EC patients.

National Image Construction from the Perspective of Multimodal Metaphor-A Case Study of the Opening Ceremony of the Beijing Winter Olympics

The national image is a comprehensive concept with a distinct political feature,including the international image presented to the outside world,and also encompassing the national identity of the people.With the development of globalization,international cultural communication has become a crucial part of shaping the national image,and the opening ceremony of the Beijing Winter Olympics has become an important opportunity for China to showcase its national image to the world in the post-pandemic era.Based on Forceville’s multimodal metaphor theory,this paper examines the metaphorical phenomena contained in the performance and their functions,effects,and purposes in the construction of the national image.It is found that there are many scenes,images,and narratives in the opening ceremony,including war metaphor,competition metaphor,personification metaphor,and other conceptual metaphors.The focus of this paper is on multimodal metaphor in a broad sense,mainly encompassing auditory and visual modes.Through the use of these multimodal metaphors,the opening ceremony of the Winter Olympics builds an image of a vibrant,peace-loving,and responsible country,which not only demonstrates China’s cultural self-confidence,but also expresses the Chinese people’s beautiful vision for the early reunification of the motherland.

Cultivation of Translators for Multimodal Translation in the New Media Age

In the 21st century,the development of digital and new media technologies has ushered in an age of pervasive multimodal communication,which has significantly amplified the role of multimodal translation in facilitating crosscultural exchanges.Despite the profound impact of these developments,the prevailing translation pedagogy remains predominantly focused on the enhancement of linguistic translation skills,with noticeable neglect of the imperative to cultivate students’competencies in multimodal translation.Based on the distinctive characteristics and challenges that multimodal translation presents in the context of new media,this study delves into the formulation of educational objectives and curriculum design for the training of multimodal translators.The intent is to propose a framework that can guide the preparation of translators who are adept and equipped to navigate the complexities and demands of the contemporary age.

Future of neurocritical care:Integrating neurophysics,multimodal monitoring,and machine learning

Multimodal monitoring(MMM)in the intensive care unit(ICU)has become increasingly sophisticated with the integration of neurophysical principles.However,the challenge remains to select and interpret the most appropriate combination of neuromonitoring modalities to optimize patient outcomes.This manuscript reviewed current neuromonitoring tools,focusing on intracranial pressure,cerebral electrical activity,metabolism,and invasive and noninvasive autoregulation moni-toring.In addition,the integration of advanced machine learning and data science tools within the ICU were discussed.Invasive monitoring includes analysis of intracranial pressure waveforms,jugular venous oximetry,monitoring of brain tissue oxygenation,thermal diffusion flowmetry,electrocorticography,depth electroencephalography,and cerebral microdialysis.Noninvasive measures include transcranial Doppler,tympanic membrane displacement,near-infrared spectroscopy,optic nerve sheath diameter,positron emission tomography,and systemic hemodynamic monitoring including heart rate variability analysis.The neurophysical basis and clinical relevance of each method within the ICU setting were examined.Machine learning algorithms have shown promise by helping to analyze and interpret data in real time from continuous MMM tools,helping clinicians make more accurate and timely decisions.These algorithms can integrate diverse data streams to generate predictive models for patient outcomes and optimize treatment strategies.MMM,grounded in neurophysics,offers a more nuanced understanding of cerebral physiology and disease in the ICU.Although each modality has its strengths and limitations,its integrated use,especially in combination with machine learning algorithms,can offer invaluable information for individualized patient care.

A Multimodal Critical Discourse Analysis of Lingnan Cultural Promotional Videos on Social Media

In recent years,more and more directors of culture and tourism have taken part in the promotion of local cultural tourism by cross-dressing,talent shows,and pushing their limits on self-media platforms.This study investigates short videos of Lingnan culture promoted by directors general and deputy directors general of the Culture,Radio,Television,Tourism,and Sports Bureau of counties and cities in Guangdong Province on social media by the method of multimodal critical discourse analysis.The analysis of 33 videos shows that Lingnan culture is a domineering and confident culture,historical culture,graceful and elegant culture,and vibrant and active culture.Domineering and confident culture is embedded in the utterances and behaviors of the directors general or deputy directors general in the video.Historical culture is realized through the conversation with historical figures through time travel.Graceful and elegant culture is constructed in the depiction of sceneries and the depiction of characters’manners.Vibrant and active culture is represented in the depiction of the characters’actional process and analytical process.

Implicit Modality Mining: An End-to-End Method for Multimodal Information Extraction

Multimodal named entity recognition(MNER)and relation extraction(MRE)are key in social media analysis but face challenges like inefficient visual processing and non-optimal modality interaction.(1)Heavy visual embedding:the process of visual embedding is both time and computationally expensive due to the prerequisite extraction of explicit visual cues from the original image before input into the multimodal model.Consequently,these approaches cannot achieve efficient online reasoning;(2)suboptimal interaction handling:the prevalent method of managing interaction between different modalities typically relies on the alternation of self-attention and cross-attention mechanisms or excessive dependence on the gating mechanism.This explicit modeling method may fail to capture some nuanced relations between image and text,ultimately undermining the model’s capability to extract optimal information.To address these challenges,we introduce Implicit Modality Mining(IMM),a novel end-to-end framework for fine-grained image-text correlation without heavy visual embedders.IMM uses an Implicit Semantic Alignment module with a Transformer for cross-modal clues and an Insert-Activation module to effectively utilize these clues.Our approach achieves state-of-the-art performance on three datasets.