Word Embeddings and Semantic Spaces in Natural Language Processing

One of the critical hurdles, and breakthroughs, in the field of Natural Language Processing (NLP) in the last two decades has been the development of techniques for text representation that solves the so-called curse of dimensionality, a problem which plagues NLP in general given that the feature set for learning starts as a function of the size of the language in question, upwards of hundreds of thousands of terms typically. As such, much of the research and development in NLP in the last two decades has been in finding and optimizing solutions to this problem, to feature selection in NLP effectively. This paper looks at the development of these various techniques, leveraging a variety of statistical methods which rest on linguistic theories that were advanced in the middle of the last century, namely the distributional hypothesis which suggests that words that are found in similar contexts generally have similar meanings. In this survey paper we look at the development of some of the most popular of these techniques from a mathematical as well as data structure perspective, from Latent Semantic Analysis to Vector Space Models to their more modern variants which are typically referred to as word embeddings. In this review of algoriths such as Word2Vec, GloVe, ELMo and BERT, we explore the idea of semantic spaces more generally beyond applicability to NLP.

An Unsupervised Writer Identification Based on Generating Clusterable Embeddings

The writer identification system identifies individuals based on their handwriting is a frequent topic in biometric authentication and verification systems.Due to its importance,numerous studies have been conducted in various languages.Researchers have established several learning methods for writer identification including supervised and unsupervised learning.However,supervised methods require a large amount of annotation data,which is impossible in most scenarios.On the other hand,unsupervised writer identification methods may be limited and dependent on feature extraction that cannot provide the proper objectives to the architecture and be misinterpreted.This paper introduces an unsupervised writer identification system that analyzes the data and recognizes the writer based on the inter-feature relations of the data to resolve the uncertainty of the features.A pairwise architecturebased Autoembedder was applied to generate clusterable embeddings for handwritten text images.Furthermore,the trained baseline architecture generates the embedding of the data image,and the K-means algorithm is used to distinguish the embedding of individual writers.The proposed model utilized the IAM dataset for the experiment as it is inconsistent with contributions from the authors but is easily accessible for writer identification tasks.In addition,traditional evaluation metrics are used in the proposed model.Finally,the proposed model is compared with a few unsupervised models,and it outperformed the state-of-the-art deep convolutional architectures in recognizing writers based on unlabeled data.

基于BM25、文本Embeddings与交叉编码器的民航客服知识库检索研究

随着民航经济的发展和人民生活水平的提高,旅客出行的服务要求越来越高,而当前传统的民航客服知识库检索普遍存在检索准确率以及效率低的问题,已经不能满足旅客的服务需求。文章通过结合Best Match 25算法、文本Embeddings和交叉编码器对知识库进行检索,高效检索出符合座席意图的答案,进而提升民航客服知识库查找效率,缩短座席通话查询时长,提升旅客服务体验,助力实现民航客服数字化、智能化转型。

An Automated System to Predict Popular Cybersecurity News Using Document Embeddings

The substantial competition among the news industries puts editors under the pressure of posting news articleswhich are likely to gain more user attention. Anticipating the popularity of news articles can help the editorial teamsin making decisions about posting a news article. Article similarity extracted from the articles posted within a smallperiod of time is found to be a useful feature in existing popularity prediction approaches. This work proposesa new approach to estimate the popularity of news articles by adding semantics in the article similarity basedapproach of popularity estimation. A semantically enriched model is proposed which estimates news popularity bymeasuring cosine similarity between document embeddings of the news articles. Word2vec model has been used togenerate distributed representations of the news content. In this work, we define popularity as the number of timesa news article is posted on different websites. We collect data from different websites that post news concerning thedomain of cybersecurity and estimate the popularity of cybersecurity news. The proposed approach is comparedwith different models and it is shown that it outperforms the other models.

Learning Context-based Embeddings for Knowledge Graph Completion

Purpose:Due to the incompleteness nature of knowledge graphs(KGs),the task of predicting missing links between entities becomes important.Many previous approaches are static,this posed a notable problem that all meanings of a polysemous entity share one embedding vector.This study aims to propose a polysemous embedding approach,named KG embedding under relational contexts(ContE for short),for missing link prediction.Design/methodology/approach:ContE models and infers different relationship patterns by considering the context of the relationship,which is implicit in the local neighborhood of the relationship.The forward and backward impacts of the relationship in ContE are mapped to two different embedding vectors,which represent the contextual information of the relationship.Then,according to the position of the entity,the entity’s polysemous representation is obtained by adding its static embedding vector to the corresponding context vector of the relationship.Findings:ContE is a fully expressive,that is,given any ground truth over the triples,there are embedding assignments to entities and relations that can precisely separate the true triples from false ones.ContE is capable of modeling four connectivity patterns such as symmetry,antisymmetry,inversion and composition.Research limitations:ContE needs to do a grid search to find best parameters to get best performance in practice,which is a time-consuming task.Sometimes,it requires longer entity vectors to get better performance than some other models.Practical implications:ContE is a bilinear model,which is a quite simple model that could be applied to large-scale KGs.By considering contexts of relations,ContE can distinguish the exact meaning of an entity in different triples so that when performing compositional reasoning,it is capable to infer the connectivity patterns of relations and achieves good performance on link prediction tasks.Originality/value:ContE considers the contexts of entities in terms of their positions in triples and the relationships they link to.It decomposes a relation vector into two vectors,namely,forward impact vector and backward impact vector in order to capture the relational contexts.ContE has the same low computational complexity as TransE.Therefore,it provides a new approach for contextualized knowledge graph embedding.

STRONG EMBEDDINGS OF PLANAR GRAPHS ON HIGHER SURFACES

In this paper, the authors discuss the upper bound for the genus of strong embeddings for 3-connected planar graphs on higher surfaces. It is shown that the problem of determining the upper bound for the strong embedding of 3-connected planar near-triangulations on higher non-orientable surfaces is NP-hard. As a corollary, a theorem of Richter, Seymour and Siran about the strong embedding of 3-connected planar graphs is generalized to orientable surface.

A NOTE ON STRONG EMBEDDINGS OF MAXIMAL PLANAR GRAPHS ON NON ORIENTABLE SURFACES

In this paper, it is shown that for every maximal planar graph G=(V,E) , a strong embedding on some non orientable surface with genus at most |V(G)|-22 is admitted such that the surface dual of G is also a planar graph. As a corollary, an interpolation theorem for strong embeddings of G on non orientable surfaces is obtained.

Flexibility of Embeddings of a Halin Graph in the Torus

In this paper we show that the face-width of any embedding of a Halin graph（a type of planar graph） in the torus is one, and give a formula for determining the number of all nonequivalent embeddings of a Halin graph in the torus.

Embeddings Among Quantum Affine sl_(n)

We establish an explicit embedding of a quantum affine sl_(n) into a quantum affine sl_(n)+1.This embedding serves as a common generalization of two natural,but seemingly unrelated,embed-dings,one on the quantum affine Schur algebra level and the other on the non-quantum level.The embedding on the quantum affine Schur algebras is used extensively in the analysis of canonical bases of quantum affine sln and gl_(n).The embedding on the non-quantum level is used crucially in a work of Riche and Williamson on the study of modular representation theory of general linear groups over a finite field.The same embedding is also used in a work of Maksimau on the categorical representations of affine general linear algebras.We further provide a more natural compatibility statement of the em-bedding on the idempotent version with that on the quantum affine Schur algebra level.A gl_(n)-variant of the embedding is also established.

Erratum to “Time-Series Embeddings from Language Models:A Tool for Wind Direction Nowcasting”

The article “Time-series embeddings from language models: A tool for wind direction nowcasting”, written by Décio ALVES, Fábio MENDON?A, Sheikh Shanawaz MOSTAFA, and Fernando MORGADO-DIAS was originally published electronically on the publisher's internet portal on 9 July 2024 without open access.

Sobolev embeddings in infinite dimensions

In this paper,we study Sobolev spaces in infinite dimensions and the corresponding embedding theorems.Our underlying spaces areℓ^(r)for r ∈[1,∞),equipped with corresponding probability measures.For the weighted Sobolev space W_(b)^(1,p)(ℓ^(r),γa)with a weight a∈ℓ^(r)of the Gaussian measureγa and a gradient weight b∈l^(∞),we characterize the relation between the weights(a and b)and the continuous(resp.compact)log-Sobolev embedding for p∈[1,∞)(resp.p∈(1,∞)).Several counterexamples are also constructed,which are of independent interest.

A Universal Inequality for Stability of Coarse Lipschitz Embeddings

Let X and Y be two pointed metric spaces.In this article,we give a generalization of the Cheng-Dong-Zhang theorem for coarse Lipschitz embeddings as follows:If f:X→Y is a standard coarse Lipschitz embedding,then for each x^(*)∈Lip_(0)(X)there existα,γ>0 depending only on f and Q_(x)*∈Lip_(0)(Y)with‖Q_(x)*‖_(Lip)≤α‖x^(*)‖_(Lip)such that|Q_(x)*f(x)-x^(*)(x)|≤γ‖x^(*)‖_(Lip),for all x∈X.Coarse stability for a pair of metric spaces is studied.This can be considered as a coarse version of Qian Problem.As an application,we give candidate negative answers to a 58-year old problem by Lindenstrauss asking whether every Banach space is a Lipschitz retract of its bidual.Indeed,we show that X is not a Lipschitz retract of its bidual if X is a universally left-coarsely stable space but not an absolute cardinality-Lipschitz retract.If there exists a universally right-coarsely stable Banach space with the RNP but not isomorphic to any Hilbert space,then the problem also has a negative answer for a separable space.

基于Transformer结构增强的神经网络架构搜索性能预测器

神经网络架构搜索(Neural Architecture Search,NAS)作为一种通过搜索算法设计神经网络架构的方法,在计算机视觉和自然语言处理等领域得到广泛应用,相较于人工设计网络,NAS方法可以减少设计成本并提高模型性能.但是NAS的性能评估需要对候选架构进行大量训练,由此带来的计算量占整个NAS的80%以上.为降低计算开销和时间成本,近年来已提出许多基于Transformer的NAS预测器,由于Transformer出色的结构编码能力可以更好地表示拓扑信息,因而得到广泛应用.但是,现有基于Transformer的NAS预测器依然存在三个问题:其一是在预处理阶段,传统的One-hot编码方式描述节点特征的能力较弱,只能区分不同操作节点类型,而难以表达操作的细节特征,如卷积核尺寸等.其二是在编码阶段,Transformer的自注意力机制导致模型结构信息缺失;其三是在评估阶段,现有的Transformer预测器仅使用多层感知机(Multilayer Perceptron,MLP)对前向传播图进行精度预测,忽略了反向传播梯度流对预测精度的影响,因此难以真正拟合NAS评估中的正、反向交替信息流图,导致预测器精度与实际运行精度误差波动极大(10%~90%).为解决上述问题,本文提出了一种基于Transformer结构增强的NAS性能预测方法.首先,在预处理阶段,本文提出了一种超维嵌入方法增加输入数据维度以强化节点操作的参数描述能力,其次,在编码阶段将Transformer编码后的信息与图结构信息共同输人一个图卷积网络(Graph Convolutional Network,GCN),弥补由自注意力机制引起的结构缺失.最后,在性能评估阶段,本文构建了同时包含前向传播和反向传播的全训练图,并将数据集信息、图结构编码与梯度编码共同输入到GCN网络预测器中,使预测结果更贴近模型真实性能.实验结果表明,本方法与目前最先进方法相比,肯德尔相关系数提高了7.45%,训练时间减少了1.55倍。

Oxygen functionalization-assisted anionic exchange toward unique construction of flower-like transition metal chalcogenide embedded carbon fabric for ultra-long life flexible energy storage and conversion

The metal-organic framework(MOF)derived Ni–Co–C–N composite alloys(NiCCZ)were“embedded”inside the carbon cloth(CC)strands as opposed to the popular idea of growing them upward to realize ultrastable energy storage and conversion application.The NiCCZ was then oxygen functionalized,facilitating the next step of stoichiometric sulfur anion diffusion during hydrothermal sulfurization,generating a flower-like metal hydroxysulfide structure(NiCCZOS)with strong partial implantation inside CC.Thus obtained NiCCZOS shows an excellent capacity when tested as a supercapacitor electrode in a three-electrode configuration.Moreover,when paired with the biomass-derived nitrogen-rich activated carbon,the asymmetric supercapacitor device shows almost 100%capacity retention even after 45,000 charge–discharge cycles with remarkable energy density(59.4 Wh kg^(-1)/263.8μWh cm^(–2))owing to a uniquely designed cathode.Furthermore,the same electrode performed as an excellent bifunctional water-splitting electrocatalyst with an overpotential of 271 mV for oxygen evolution reaction(OER)and 168.4 mV for hydrogen evolution reaction(HER)at 10 mA cm−2 current density along with 30 h of unhinged chronopotentiometric stability performance for both HER and OER.Hence,a unique metal chalcogenide composite electrode/substrate configuration has been proposed as a highly stable electrode material for flexible energy storage and conversion applications.

Hate speech detection in Twitter using hybrid embeddings and improved cuckoo search-based neural networks

Purpose-Hate speech is an expression of intense hatred.Twitter has become a popular analytical tool for the prediction and monitoring of abusive behaviors.Hate speech detection with social media data has witnessed special research attention in recent studies,hence,the need to design a generic metadata architecture and efficient feature extraction technique to enhance hate speech detection.Design/methodology/approach-This study proposes a hybrid embeddings enhanced with a topic inference method and an improved cuckoo search neural network for hate speech detection in Twitter data.The proposed method uses a hybrid embeddings technique that includes Term Frequency-Inverse Document Frequency(TF-IDF)for word-level feature extraction and Long Short Term Memory(LSTM)which is a variant of recurrent neural networks architecture for sentence-level feature extraction.The extracted features from the hybrid embeddings then serve as input into the improved cuckoo search neural network for the prediction of a tweet as hate speech,offensive language or neither.Findings-The proposed method showed better results when tested on the collected Twitter datasets compared to other related methods.In order to validate the performances of the proposed method,t-test and post hoc multiple comparisons were used to compare the significance and means of the proposed method with other related methods for hate speech detection.Furthermore,Paired Sample t-Test was also conducted to validate the performances of the proposed method with other related methods.Research limitations/implications-Finally,the evaluation results showed that the proposed method outperforms other related methods with mean F1-score of 91.3.Originality/value-The main novelty of this study is the use of an automatic topic spotting measure based on na€ıve Bayes model to improve features representation.

Continuous Embeddings of Besov-Morrey Function Spaces

We study embeddings of spaces of Besov-Morrey type, MB p1^s1 q1^r1（R^d）→MB p2^s2 q2^r2（R^d） and obtain necessary and sufficient conditions for this. Moreover, we can also charaeterise the special weighted situation Bp1^s1 （R^d ,w）→MB p2^s2 q2^r2（R^d） for a Muekenhoupt A∞ weight w, with wα（x） = ｜x｜^a, 〉 -d, as a typical example.

Embeddings of weighted Sobolev spaces and degenerate elliptic problems

New embeddings of some weighted Sobolev spaces with weights a(x)and b(x)are established.The weights a(x)and b(x)can be singular.Some applications of these embeddings to a class of degenerate elliptic problems of the form-div(a(x)?u)=b(x)f(x,u)in?,u=0 on??,where?is a bounded or unbounded domain in RN,N 2,are presented.The main results of this paper also give some generalizations of the well-known Caffarelli-Kohn-Nirenberg inequality.

A Linked List Encryption Scheme for Image Steganography without Embedding

Information steganography has received more and more attention from scholars nowadays,especially in the area of image steganography,which uses image content to transmit information and makes the existence of secret information undetectable.To enhance concealment and security,the Steganography without Embedding(SWE)method has proven effective in avoiding image distortion resulting from cover modification.In this paper,a novel encrypted communication scheme for image SWE is proposed.It reconstructs the image into a multi-linked list structure consisting of numerous nodes,where each pixel is transformed into a single node with data and pointer domains.By employing a special addressing algorithm,the optimal linked list corresponding to the secret information can be identified.The receiver can restore the secretmessage fromthe received image using only the list header position information.The scheme is based on the concept of coverless steganography,eliminating the need for any modifications to the cover image.It boasts high concealment and security,along with a complete message restoration rate,making it resistant to steganalysis.Furthermore,this paper proposes linked-list construction schemeswithin theproposedframework,which caneffectively resist a variety of attacks,includingnoise attacks and image compression,demonstrating a certain degree of robustness.To validate the proposed framework,practical tests and comparisons are conducted using multiple datasets.The results affirm the framework’s commendable performance in terms of message reduction rate,hidden writing capacity,and robustness against diverse attacks.

Enhancing visual security: An image encryption scheme based on parallel compressive sensing and edge detection embedding

A novel image encryption scheme based on parallel compressive sensing and edge detection embedding technology is proposed to improve visual security. Firstly, the plain image is sparsely represented using the discrete wavelet transform.Then, the coefficient matrix is scrambled and compressed to obtain a size-reduced image using the Fisher–Yates shuffle and parallel compressive sensing. Subsequently, to increase the security of the proposed algorithm, the compressed image is re-encrypted through permutation and diffusion to obtain a noise-like secret image. Finally, an adaptive embedding method based on edge detection for different carrier images is proposed to generate a visually meaningful cipher image. To improve the plaintext sensitivity of the algorithm, the counter mode is combined with the hash function to generate keys for chaotic systems. Additionally, an effective permutation method is designed to scramble the pixels of the compressed image in the re-encryption stage. The simulation results and analyses demonstrate that the proposed algorithm performs well in terms of visual security and decryption quality.

Randomization Strategies in Image Steganography Techniques:A Review

Image steganography is one of the prominent technologies in data hiding standards.Steganographic system performance mostly depends on the embedding strategy.Its goal is to embed strictly confidential information into images without causing perceptible changes in the original image.The randomization strategies in data embedding techniques may utilize random domains,pixels,or region-of-interest for concealing secrets into a cover image,preventing information from being discovered by an attacker.The implementation of an appropriate embedding technique can achieve a fair balance between embedding capability and stego image imperceptibility,but it is challenging.A systematic approach is used with a standard methodology to carry out this study.This review concentrates on the critical examination of several embedding strategies,incorporating experimental results with state-of-the-art methods emphasizing the robustness,security,payload capacity,and visual quality metrics of the stego images.The fundamental ideas of steganography are presented in this work,along with a unique viewpoint that sets it apart from previous works by highlighting research gaps,important problems,and difficulties.Additionally,it offers a discussion of suggested directions for future study to advance and investigate uncharted territory in image steganography.