Adequate alignment and interaction for cross-modal retrieval

Background Cross-modal retrieval has attracted widespread attention in many cross-media similarity search applications,particularly image-text retrieval in the fields of computer vision and natural language processing.Recently,visual and semantic embedding(VSE)learning has shown promising improvements in image text retrieval tasks.Most existing VSE models employ two unrelated encoders to extract features and then use complex methods to contextualize and aggregate these features into holistic embeddings.Despite recent advances,existing approaches still suffer from two limitations:(1)without considering intermediate interactions and adequate alignment between different modalities,these models cannot guarantee the discriminative ability of representations;and(2)existing feature aggregators are susceptible to certain noisy regions,which may lead to unreasonable pooling coefficients and affect the quality of the final aggregated features.Methods To address these challenges,we propose a novel cross-modal retrieval model containing a well-designed alignment module and a novel multimodal fusion encoder that aims to learn the adequate alignment and interaction of aggregated features to effectively bridge the modality gap.Results Experiments on the Microsoft COCO and Flickr30k datasets demonstrated the superiority of our model over state-of-the-art methods.

Multimodal Sentiment Analysis Based on a Cross-Modal Multihead Attention Mechanism

Multimodal sentiment analysis aims to understand people’s emotions and opinions from diverse data.Concate-nating or multiplying various modalities is a traditional multi-modal sentiment analysis fusion method.This fusion method does not utilize the correlation information between modalities.To solve this problem,this paper proposes amodel based on amulti-head attention mechanism.First,after preprocessing the original data.Then,the feature representation is converted into a sequence of word vectors and positional encoding is introduced to better understand the semantic and sequential information in the input sequence.Next,the input coding sequence is fed into the transformer model for further processing and learning.At the transformer layer,a cross-modal attention consisting of a pair of multi-head attention modules is employed to reflect the correlation between modalities.Finally,the processed results are input into the feedforward neural network to obtain the emotional output through the classification layer.Through the above processing flow,the model can capture semantic information and contextual relationships and achieve good results in various natural language processing tasks.Our model was tested on the CMU Multimodal Opinion Sentiment and Emotion Intensity(CMU-MOSEI)and Multimodal EmotionLines Dataset(MELD),achieving an accuracy of 82.04% and F1 parameters reached 80.59% on the former dataset.

Cross-Modal Consistency with Aesthetic Similarity for Multimodal False Information Detection

With the explosive growth of false information on social media platforms, the automatic detection of multimodalfalse information has received increasing attention. Recent research has significantly contributed to multimodalinformation exchange and fusion, with many methods attempting to integrate unimodal features to generatemultimodal news representations. However, they still need to fully explore the hierarchical and complex semanticcorrelations between different modal contents, severely limiting their performance detecting multimodal falseinformation. This work proposes a two-stage detection framework for multimodal false information detection,called ASMFD, which is based on image aesthetic similarity to segment and explores the consistency andinconsistency features of images and texts. Specifically, we first use the Contrastive Language-Image Pre-training(CLIP) model to learn the relationship between text and images through label awareness and train an imageaesthetic attribute scorer using an aesthetic attribute dataset. Then, we calculate the aesthetic similarity betweenthe image and related images and use this similarity as a threshold to divide the multimodal correlation matrixinto consistency and inconsistencymatrices. Finally, the fusionmodule is designed to identify essential features fordetectingmultimodal false information. In extensive experiments on four datasets, the performance of the ASMFDis superior to state-of-the-art baseline methods.

Joint alignment and steering to manage interference

In wireless communication networks,mobile users in overlapping areas may experience severe interference,therefore,designing effective Interference Management(IM)methods is crucial to improving network performance.However,when managing multiple disturbances from the same source,it may not be feasible to use existing IM methods such as Interference Alignment(IA)and Interference Steering(IS)exclusively.It is because with IA,the aligned interference becomes indistinguishable at its desired Receiver(Rx)under the cost constraint of Degrees-of-Freedom(DoF),while with IS,more transmit power will be consumed in the direct and repeated application of IS to each interference.To remedy these deficiencies,Interference Alignment Steering(IAS)is proposed by incorporating IA and IS and exploiting their advantages in IM.With IAS,the interfering Transmitter(Tx)first aligns one interference incurred by the transmission of one data stream to a one-dimensional subspace orthogonal to the desired transmission at the interfered Rx,and then the remaining interferences are treated as a whole and steered to the same subspace as the aligned interference.Moreover,two improved versions of IAS,i.e.,IAS with Full Adjustment at the Interfering Tx(IAS-FAIT)and Interference Steering and Alignment(ISA),are presented.The former considers the influence of IA on the interfering user-pair's performance.The orthogonality between the desired signals at the interfered Rx can be maintained by adjusting the spatial characteristics of all interferences and the aligned interference components,thus ensuring the Spectral Efficiency(SE)of the interfering communication pairs.Under ISA,the power cost for IS at the interfered Tx is minimized,hence improving SE performance of the interfered communication-pairs.Since the proposed methods are realized at the interfering and interfered Txs cooperatively,the expenses of IM are shared by both communication-pairs.Our in-depth simulation results show that joint use of IA and IS can effectively manage multiple disturbances from the same source and improve the system's SE.

Improved spatio-temporal alignment measurement method for hull deformation

In this paper,an improved spatio-temporal alignment measurement method is presented to address the inertial matching measurement of hull deformation under the coexistence of time delay and large misalignment angle.Large misalignment angle and time delay often occur simultaneously and bring great challenges to the accurate measurement of hull deformation in space and time.The proposed method utilizes coarse alignment with large misalignment angle and time delay estimation of inertial measurement unit modeling to establish a brand-new spatiotemporal aligned hull deformation measurement model.In addition,two-step loop control is designed to ensure the accurate description of dynamic deformation angle and static deformation angle by the time-space alignment method of hull deformation.The experiments illustrate that the proposed method can effectively measure the hull deformation angle when time delay and large misalignment angle coexist.

Beam based alignment using a neural network

Beams typically do not travel through the magnet centers because of errors in storage rings.The beam deviating from the quadrupole centers is affected by additional dipole fields due to magnetic field feed-down.Beam-based alignment(BBA)is often performed to determine a golden orbit where the beam circulates around the quadrupole center axes.For storage rings with many quadrupoles,the conventional BBA procedure is time-consuming,particularly in the commissioning phase,because of the necessary iterative process.In addition,the conventional BBA method can be affected by strong coupling and the nonlinearity of the storage ring optics.In this study,a novel method based on a neural network was proposed to determine the golden orbit in a much shorter time with reasonable accuracy.This golden orbit can be used directly for operation or adopted as a starting point for conventional BBA.The method was demonstrated in the HLS-II storage ring for the first time through simulations and online experiments.The results of the experiments showed that the golden orbit obtained using this new method was consistent with that obtained using the conventional BBA.The development of this new method and the corresponding experiments are reported in this paper.

A Time Series Short-Term Prediction Method Based on Multi-Granularity Event Matching and Alignment

Accurate forecasting of time series is crucial across various domains.Many prediction tasks rely on effectively segmenting,matching,and time series data alignment.For instance,regardless of time series with the same granularity,segmenting them into different granularity events can effectively mitigate the impact of varying time scales on prediction accuracy.However,these events of varying granularity frequently intersect with each other,which may possess unequal durations.Even minor differences can result in significant errors when matching time series with future trends.Besides,directly using matched events but unaligned events as state vectors in machine learning-based prediction models can lead to insufficient prediction accuracy.Therefore,this paper proposes a short-term forecasting method for time series based on a multi-granularity event,MGE-SP(multi-granularity event-based short-termprediction).First,amethodological framework for MGE-SP established guides the implementation steps.The framework consists of three key steps,including multi-granularity event matching based on the LTF(latest time first)strategy,multi-granularity event alignment using a piecewise aggregate approximation based on the compression ratio,and a short-term prediction model based on XGBoost.The data from a nationwide online car-hailing service in China ensures the method’s reliability.The average RMSE(root mean square error)and MAE(mean absolute error)of the proposed method are 3.204 and 2.360,lower than the respective values of 4.056 and 3.101 obtained using theARIMA(autoregressive integratedmoving average)method,as well as the values of 4.278 and 2.994 obtained using k-means-SVR(support vector regression)method.The other experiment is conducted on stock data froma public data set.The proposed method achieved an average RMSE and MAE of 0.836 and 0.696,lower than the respective values of 1.019 and 0.844 obtained using the ARIMA method,as well as the values of 1.350 and 1.172 obtained using the k-means-SVR method.

On The Cauchy Problem for A 1D Euler-Alignment System in Besov Spaces

In this paper, we investigate a 1D pressureless Euler-alignment system with a non-local alignment term, describing a kind of self-organizing problem for flocking. As a result, by the transport equation theory and Lagrange coordinate transformation, the local well-posedness of the solutions for the 1D pressureless Euler-alignment in Besov spaces with 1≤p∞ is established. Next, the ill-posedness of the solutions for this model in Besov spaces with 1≤p and is also deduced. Finally, the precise blow-up criteria of the solutions for this system is presented in Besov spaces with 1≤p .

Vector meson spin alignment by the strong force field

Observation of unexpectedly large global spin alignment of φ vector mesons in non-central heavy-ion collisions by STAR experiment may reveal the non-perturbative nature of quark interaction in hot matter through fluctuating strong force field with short correlation length.

A Multi-Level Circulant Cross-Modal Transformer for Multimodal Speech Emotion Recognition

Speech emotion recognition,as an important component of humancomputer interaction technology,has received increasing attention.Recent studies have treated emotion recognition of speech signals as a multimodal task,due to its inclusion of the semantic features of two different modalities,i.e.,audio and text.However,existing methods often fail in effectively represent features and capture correlations.This paper presents a multi-level circulant cross-modal Transformer(MLCCT)formultimodal speech emotion recognition.The proposed model can be divided into three steps,feature extraction,interaction and fusion.Self-supervised embedding models are introduced for feature extraction,which give a more powerful representation of the original data than those using spectrograms or audio features such as Mel-frequency cepstral coefficients(MFCCs)and low-level descriptors(LLDs).In particular,MLCCT contains two types of feature interaction processes,where a bidirectional Long Short-term Memory(Bi-LSTM)with circulant interaction mechanism is proposed for low-level features,while a two-stream residual cross-modal Transformer block is appliedwhen high-level features are involved.Finally,we choose self-attention blocks for fusion and a fully connected layer to make predictions.To evaluate the performance of our proposed model,comprehensive experiments are conducted on three widely used benchmark datasets including IEMOCAP,MELD and CMU-MOSEI.The competitive results verify the effectiveness of our approach.

Precision measurement and suppression of low-frequency noise in a current source with double-resonance alignment magnetometers

Low-noise high-stability current sources have essential applications such as neutron electric dipole moment measurement and high-stability magnetometers. Previous studies mainly focused on frequency noise above 0.1 Hz while less on the low-frequency noise/drift. We use double resonance alignment magnetometers(DRAMs) to measure and suppress the low-frequency noise of a homemade current source(CS) board. The CS board noise level is suppressed by about 10 times in the range of 0.001-0.1 Hz and is reduced to 100 n A/√Hz at 0.001 Hz. The relative stability of CS board can reach2.2 × 10^(-8). In addition, the DRAM shows a better resolution and accuracy than a commercial 7.5-digit multimeter when measuring our homemade CS board. Further, by combining the DRAM with a double resonance orientation magnetometer,we may realize a low-noise CS in the 0.001-1000 Hz range.

Functionalized alginate-based bioinks for microscale electrohydrodynamic bioprinting of living tissue constructs with improved cellular spreading and alignment

Bioprinting has been widely investigated for tissue engineering and regenerative medicine applications.However,it is still difficult to reconstruct the complex native cell arrangement due to the limited printing resolution of conventional bioprinting techniques such as extrusion-and inkjet-based printing.Recently,an electrohydrodynamic(EHD)bioprinting strategy was reported for the precise deposition of well-organized cell-laden constructs with microscale filament size,whereas few studies have been devoted to developing bioinks that can be applied for EHD bioprinting and simultaneously support cell spreading.This study describes functionalized alginate-based bioinks for microscale EHD bioprinting using peptide grafting and fibrin incorporation,which leads to high cell viability(>90%)and cell spreading.The printed filaments can be further refined to as small as 30μm by incorporating polyoxyethylene and remained stable over one week when exposed to an aqueous environment.By utilizing the presented alginate-based bioinks,layer-specific cell alignment along the printing struts could be observed inside the EHD-printed microscale filaments,which allows fabricating living constructs with cell-scale filament resolution for guided cellular orientation.

Photo-induced flexible semiconductor CdSe/CdS quantum rods alignment

The anisotropic absorption and emission from semiconductor CdSe/CdS quantum rods(QRs)provide extra benefits among other photoluminescence nanocrystals.Using photo-induced alignment technique,the QRs can be oriented in liquid crystal polymer matrix at a large scale.In this article,a 2D Dammann grating pattern,within“SKL”characters domains aligned QRs in composite film,was fabricated by multi-step photo exposure using several photo masks,and a continuous geometric lens profile pattern aligned QRs was realized by the single step polarization converting holographic irradiation method.Both polarized optical microscope and fluorescence microscope are employed to determine the liquid crystal director profiles and QRs anisotropic excitation properties.We have been able to orient the QRs in fine binary and continuous patterns that confirms the strong quantum rod aligning ability of the proposed method.Thus,the proposed approach paves a way for photoinduced flexible QRs alignments to provide a highly specific and difficult-to-replicate security application at a large scale.

TECMH:Transformer-Based Cross-Modal Hashing For Fine-Grained Image-Text Retrieval

In recent years,cross-modal hash retrieval has become a popular research field because of its advantages of high efficiency and low storage.Cross-modal retrieval technology can be applied to search engines,crossmodalmedical processing,etc.The existing main method is to use amulti-label matching paradigm to finish the retrieval tasks.However,such methods do not use fine-grained information in the multi-modal data,which may lead to suboptimal results.To avoid cross-modal matching turning into label matching,this paper proposes an end-to-end fine-grained cross-modal hash retrieval method,which can focus more on the fine-grained semantic information of multi-modal data.First,the method refines the image features and no longer uses multiple labels to represent text features but uses BERT for processing.Second,this method uses the inference capabilities of the transformer encoder to generate global fine-grained features.Finally,in order to better judge the effect of the fine-grained model,this paper uses the datasets in the image text matching field instead of the traditional label-matching datasets.This article experiment on Microsoft COCO(MS-COCO)and Flickr30K datasets and compare it with the previous classicalmethods.The experimental results show that this method can obtain more advanced results in the cross-modal hash retrieval field.

ViT2CMH:Vision Transformer Cross-Modal Hashing for Fine-Grained Vision-Text Retrieval

In recent years,the development of deep learning has further improved hash retrieval technology.Most of the existing hashing methods currently use Convolutional Neural Networks(CNNs)and Recurrent Neural Networks(RNNs)to process image and text information,respectively.This makes images or texts subject to local constraints,and inherent label matching cannot capture finegrained information,often leading to suboptimal results.Driven by the development of the transformer model,we propose a framework called ViT2CMH mainly based on the Vision Transformer to handle deep Cross-modal Hashing tasks rather than CNNs or RNNs.Specifically,we use a BERT network to extract text features and use the vision transformer as the image network of the model.Finally,the features are transformed into hash codes for efficient and fast retrieval.We conduct extensive experiments on Microsoft COCO(MS-COCO)and Flickr30K,comparing with baselines of some hashing methods and image-text matching methods,showing that our method has better performance.

Deep learning for estimation of Kirkpatrick-Baez mirror alignment errors

A deep learning-based automated Kirkpatrick-Baez mirror alignment method is proposed for synchrotron radiation.We trained a convolutional neural network(CNN)on simulated and experimental imaging data of a focusing system.Instead of learning directly from bypass images,we use a scatterer for X-ray modulation and speckle generation for image feature enhancement.The smallest normalized root-mean-square error on the validation set was 4%.Compared with conventional alignment methods based on motor scanning and analyzer setups,the present method simplified the optical layout and estimated alignment errors using a single-exposure experiment.Single-shot misalignment error estimation only took 0.13 s,significantly outperforming conventional methods.We also demonstrated the effects of the beam quality and pretraining using experimental data.The proposed method exhibited strong robustness,can handle high-precision focusing systems with complex or dynamic wavefront errors,and provides an important basis for intelligent control of future synchrotron radiation beamlines.

Modeling and Analysis of UAV-Assisted Mobile Network with Imperfect Beam Alignment

With the rapid development of emerging 5G and beyond(B5G),Unmanned Aerial Vehicles(UAVs)are increasingly important to improve the performance of dense cellular networks.As a conventional metric,coverage probability has been widely studied in communication systems due to the increasing density of users and complexity of the heterogeneous environment.In recent years,stochastic geometry has attracted more attention as a mathematical tool for modeling mobile network systems.In this paper,an analytical approach to the coverage probability analysis of UAV-assisted cellular networks with imperfect beam alignment has been proposed.An assumption was considered that all users are distributed according to Poisson Cluster Process(PCP)around base stations,in particular,Thomas Cluster Process(TCP).Using thismodel,the impact of beam alignment errors on the coverage probabilitywas investigated.Initially,the ProbabilityDensity Function(PDF)of directional antenna gain between the user and its serving base station was obtained.Then,association probability with each tier was achieved.A tractable expression was derived for coverage probability in both Line-of-Sight(LoS)andNon-Line-of-Sight(NLoS)condition links.Numerical results demonstrated that at low UAVs altitude,beam alignment errors significantly degrade coverage performance.Moreover,for a small cluster size,alignment errors do not necessarily affect the coverage performance.

Anomaly Detection of UAV State Data Based on Single-Class Triangular Global Alignment Kernel Extreme Learning Machine

Unmanned Aerial Vehicles(UAVs)are widely used and meet many demands in military and civilian fields.With the continuous enrichment and extensive expansion of application scenarios,the safety of UAVs is constantly being challenged.To address this challenge,we propose algorithms to detect anomalous data collected from drones to improve drone safety.We deployed a one-class kernel extreme learning machine(OCKELM)to detect anomalies in drone data.By default,OCKELM uses the radial basis(RBF)kernel function as the kernel function of themodel.To improve the performance ofOCKELM,we choose a TriangularGlobalAlignmentKernel(TGAK)instead of anRBF Kernel and introduce the Fast Independent Component Analysis(FastICA)algorithm to reconstruct UAV data.Based on the above improvements,we create a novel anomaly detection strategy FastICA-TGAK-OCELM.The method is finally validated on the UCI dataset and detected on the Aeronautical Laboratory Failures and Anomalies(ALFA)dataset.The experimental results show that compared with other methods,the accuracy of this method is improved by more than 30%,and point anomalies are effectively detected.

Review of Visible-Infrared Cross-Modality Person Re-Identification

Person re-identification(ReID)is a sub-problem under image retrieval.It is a technology that uses computer vision to identify a specific pedestrian in a collection of pictures or videos.The pedestrian image under cross-device is taken from a monitored pedestrian image.At present,most ReID methods deal with the matching between visible and visible images,but with the continuous improvement of security monitoring system,more and more infrared cameras are used to monitor at night or in dim light.Due to the image differences between infrared camera and RGB camera,there is a huge visual difference between cross-modality images,so the traditional ReID method is difficult to apply in this scene.In view of this situation,studying the pedestrian matching between visible and infrared modalities is particularly crucial.Visible-infrared person re-identification(VI-ReID)was first proposed in 2017,and then attracted more and more attention,and many advanced methods emerged.

Anti-interference self-alignment algorithm by attitude optimization estimation for SINS on a rocking base

The performance of a strapdown inertial navigation system(SINS)largely depends on the accuracy and rapidness of the initial alignment.A novel anti-interference self-alignment algorithm by attitude optimization estimation for SINS on a rocking base is presented in this paper.The algorithm transforms the initial alignment into the initial attitude determination problem by using infinite vector observations to remove the angular motions,the SINS alignment is heuristically established as an optimiza-tion problem of finding the minimum eigenvector.In order to further improve the alignment precision,an adaptive recursive weighted least squares(ARWLS)curve fitting algorithm is used to fit the translational motion interference-contaminated reference vectors according to their time domain characteristics.Simulation studies and experimental results favorably demonstrate its rapidness,accuracy and robustness.