Towards privacy-preserving and efficient word vector learning for lightweight IoT devices

Nowadays,Internet of Things(IoT)is widely deployed and brings great opportunities to change people's daily life.To realize more effective human-computer interaction in the IoT applications,the Question Answering(QA)systems implanted in the IoT services are supposed to improve the ability to understand natural language.Therefore,the distributed representation of words,which contains more semantic or syntactic information,has been playing a more and more important role in the QA systems.However,learning high-quality distributed word vectors requires lots of storage and computing resources,hence it cannot be deployed on the resource-constrained IoT devices.It is a good choice to outsource the data and computation to the cloud servers.Nevertheless,it could cause privacy risks to directly upload private data to the untrusted cloud.Therefore,realizing the word vector learning process over untrusted cloud servers without privacy leakage is an urgent and challenging task.In this paper,we present a novel efficient word vector learning scheme over encrypted data.We first design a series of arithmetic computation protocols.Then we use two non-colluding cloud servers to implement high-quality word vectors learning over encrypted data.The proposed scheme allows us to perform training word vectors on the remote cloud servers while protecting privacy.Security analysis and experiments over real data sets demonstrate that our scheme is more secure and efficient than existing privacy-preserving word vector learning schemes.

Research on Cross-Language Retrieval Using Bilingual Word Vectors in Different Languages

Bilingual word vectors have been exploited a lot in cross-language information retrieval research. However, most of the research is currently focused on similar language pairs. There are very few studies exploring the impact of using bilingual word vectors for cross-language information retrieval in long-distance language pairs. In this paper, it systematically analyzes the retrieval performance of various European languages (English, German, Italian, French, Finnish, Dutch) as well as Asian languages (Chinese, Japanese) in the adhoc task of CLEF 2002–2003 campaign. Genetic proximity was used to visually represent the relationships between languages and compare their crosslingual retrieval performance in various settings. The results show that the differences in language vocabulary would dramatically affect the retrieval performance. At the same time, the term by term translation retrieval method performs slightly better than the simple vector addition retrieval methods. It proves that the translation-based retrieval model can still maintain its advantage under the new semantic scheme.

Word2Vec新闻推荐系统设计与实现——基于Attention机制与Embedding优化

[目的/意义]设计基于Attention机制与Embedding优化的Word2Vec新闻推荐系统,通过词向量计算上的性能提升改善推荐成效。[方法/过程]重点解决改进Word2Vec的新闻推荐系统建设中的3个关键技术难点:(1)基于Attention机制与Embedding优化的Word2Vec模型构建,为系统提供词向量计算神经网络;(2)MongoDB及Redis数据库的可用性提升,提升分布式框架下的数据库架构鲁棒性;(3)构建智能监控与运维平台。[结果/结论]对比Word2Vec,基于Attention机制与Embedding优化的Word2Vec在损失值和准确率方面明显提升,数据库层优化及智能监控与运维平台提升系统可靠性和稳定性。

基于Word2vec的二语教学“基本形式库”构建方法初探

二语教学中,重视表达取向的“基本形式”观致力于构建一个“基本形式库”。本文以微博语料库为例,将其中高频词设置为检索词,依据Word2vec训练的词向量进行检索,围绕检索词查找近似词来构成(准)等义组,继而进一步确定该组的“基本形式”。本文初步提出了一种兼具可操作性和效率性的建设方法,作为人工建设“基本形式库”的辅助工具,并检索出了一部分基本形式(准)等义组作为前人研究的补充,为“基本形式”观理论进一步发展提供工具与思考。

基于Word2vector的文本特征化表示方法

针对基于词语统计的特征化表示无法有效提取文本的词义特征的问题,提出一种基于上下文关系的文本特征化表示方法。该方法利用Word2vector提取词义特征,获得词向量;再对词向量进行"最优适应度划分"的聚类,并根据聚类结果将词语替代表示为聚类质心;根据质心及其所代表的词语的词频,构成词向量聚类质心频率模型(semantic frequency-inverse document frequency,SF-IDF),用于特征化表示文本。在不依赖语义规则的情况下,分别以路透社文本集Reuter-21578、维基百科(extensible markup language,XML)数据为文本数据集,采用神经网络语言模型(neural network language model,NNLM)算法进行文本分类实验,并采用F1-measure标准进行样本分类的效果评估,词向量聚类质心频率模型SF-IDF(semantic frequency-inverse document frequency,SF-IDF)向量与现有技术中词频-逆向文件频率(term frequency-inverse document frequency,TF-IDF)向量的分类效果对比,与TF-IDF模型进行对比实验;在Reuter-21578数据集上平均准确率由原有的57.1%提高到63.3%,在Wikipedia XML数据集上平均准确率由原有的48.7%提高到59.2%。SF-IDF模型可适用于现行的基于特征向量的信息检索算法,且较TF-IDF模型有更高的文本相似性分析效率,可提升文本分类准确率。

Paragraph Vector Representation Based on Word to Vector and CNN Learning

Document processing in natural language includes retrieval,sentiment analysis,theme extraction,etc.Classical methods for handling these tasks are based on models of probability,semantics and networks for machine learning.The probability model is loss of semantic information in essential,and it influences the processing accuracy.Machine learning approaches include supervised,unsupervised,and semi-supervised approaches,labeled corpora is necessary for semantics model and supervised learning.The method for achieving a reliably labeled corpus is done manually,it is costly and time-consuming because people have to read each document and annotate the label of each document.Recently,the continuous CBOW model is efficient for learning high-quality distributed vector representations,and it can capture a large number of precise syntactic and semantic word relationships,this model can be easily extended to learn paragraph vector,but it is not precise.Towards these problems,this paper is devoted to developing a new model for learning paragraph vector,we combine the CBOW model and CNNs to establish a new deep learning model.Experimental results show that paragraph vector generated by the new model is better than the paragraph vector generated by CBOW model in semantic relativeness and accuracy.

一种基于Word2Vector与编辑距离的句子相似度计算方法

随着各种问答系统的流行与聊天机器人的火热,对句子相似性的比较和处理越来越成为各类类似系统的核心部分。因此,研究并设计出一种好的句子相似性比较方法变得越来越关键。该文基于一种深度学习模型Word2Vector并且结合编辑距离算法提出了一种句子相似度计算方法,给出了具体的设计思路,并且通过实验验证了该方法的有效性,最后总结了该方法的优缺点。

Improve Neural Machine Translation by Building Word Vector with Part of Speech

Neural Machine Translation(NMT)based system is an important technology for translation applications.However,there is plenty of rooms for the improvement of NMT.In the process of NMT,traditional word vector cannot distinguish the same words under different parts of speech(POS).Aiming to alleviate this problem,this paper proposed a new word vector training method based on POS feature.It can efficiently improve the quality of translation by adding POS feature to the training process of word vectors.In the experiments,we conducted extensive experiments to evaluate our methods.The experimental result shows that the proposed method is beneficial to improve the quality of translation from English into Chinese.

Word Embeddings and Semantic Spaces in Natural Language Processing

One of the critical hurdles, and breakthroughs, in the field of Natural Language Processing (NLP) in the last two decades has been the development of techniques for text representation that solves the so-called curse of dimensionality, a problem which plagues NLP in general given that the feature set for learning starts as a function of the size of the language in question, upwards of hundreds of thousands of terms typically. As such, much of the research and development in NLP in the last two decades has been in finding and optimizing solutions to this problem, to feature selection in NLP effectively. This paper looks at the development of these various techniques, leveraging a variety of statistical methods which rest on linguistic theories that were advanced in the middle of the last century, namely the distributional hypothesis which suggests that words that are found in similar contexts generally have similar meanings. In this survey paper we look at the development of some of the most popular of these techniques from a mathematical as well as data structure perspective, from Latent Semantic Analysis to Vector Space Models to their more modern variants which are typically referred to as word embeddings. In this review of algoriths such as Word2Vec, GloVe, ELMo and BERT, we explore the idea of semantic spaces more generally beyond applicability to NLP.

自然语言处理领域中的词嵌入方法综述

词嵌入作为自然语言处理任务的第一步,其目的是将输入的自然语言文本转换为模型可以处理的数值向量,即词向量,也称词的分布式表示。词向量作为自然语言处理任务的根基,是完成一切自然语言处理任务的前提。然而,国内外针对词嵌入方法的综述文献大多只关注于不同词嵌入方法本身的技术路线,而未能将词嵌入的前置分词方法以及词嵌入方法完整的演变趋势进行分析与概述。以word2vec模型和Transformer模型作为划分点,从生成的词向量是否能够动态地改变其内隐的语义信息来适配输入句子的整体语义这一角度,将词嵌入方法划分为静态词嵌入方法和动态词嵌入方法,并对此展开讨论。同时,针对词嵌入中的分词方法,包括整词切分和子词切分,进行了对比和分析;针对训练词向量所使用的语言模型,从概率语言模型到神经概率语言模型再到如今的深度上下文语言模型的演化,进行了详细列举和阐述;针对预训练语言模型时使用的训练策略进行了总结和探讨。最后,总结词向量质量的评估方法,分析词嵌入方法的当前现状并对其未来发展方向进行展望。

基于Word2Vec的中文文本零水印算法

经典的文本鲁棒水印会修改文本内容或格式,从而降低文本的保真性和可用性,文章提出了一种基于Word2Vec的中文文本零水印算法,能够在不修改文本信息的前提下实现水印的生成和检测。首先对文本数据进行分词,统计词频并提取特征词,运用Word2Vec生成相应的特征词向量;然后采用SVD(奇异值分解)算法对其进行降维,并结合AES(高级加密标准)加密生成最终的零水印。水印检测时,通过对比SVD分解产生的特征值和特征向量判断版权归属。基于理论概述和实验结果综合分析,文章提出的零水印算法不需要对原始文本做任何修改,能够抵抗一定程度的增删、句型转换、同义词替换等攻击,具有一定的鲁棒性,切实有效地解决了文本的版权保护问题。

An Optimized Chinese Filtering Model Using Value Scale Extended Text Vector

With the development of Internet technology,the explosive growth of Internet information presentation has led to difficulty in filtering effective information.Finding a model with high accuracy for text classification has become a critical problem to be solved by text filtering,especially for Chinese texts.This paper selected the manually calibrated Douban movie website comment data for research.First,a text filtering model based on the BP neural network has been built;Second,based on the Term Frequency-Inverse Document Frequency(TF-IDF)vector space model and the doc2vec method,the text word frequency vector and the text semantic vector were obtained respectively,and the text word frequency vector was linearly reduced by the Principal Component Analysis(PCA)method.Third,the text word frequency vector after dimensionality reduction and the text semantic vector were combined,add the text value degree,and the text synthesis vector was constructed.Experiments show that the model combined with text word frequency vector degree after dimensionality reduction,text semantic vector,and text value has reached the highest accuracy of 84.67%.

Word Sense Disambiguation Based Sentiment Classification Using Linear Kernel Learning Scheme

Word Sense Disambiguation has been a trending topic of research in Natural Language Processing and Machine Learning.Mining core features and performing the text classification still exist as a challenging task.Here the features of the context such as neighboring words like adjective provide the evidence for classification using machine learning approach.This paper presented the text document classification that has wide applications in information retrieval,which uses movie review datasets.Here the document indexing based on controlled vocabulary,adjective,word sense disambiguation,generating hierarchical cate-gorization of web pages,spam detection,topic labeling,web search,document summarization,etc.Here the kernel support vector machine learning algorithm helps to classify the text and feature extract is performed by cuckoo search opti-mization.Positive review and negative review of movie dataset is presented to get the better classification accuracy.Experimental results focused with context mining,feature analysis and classification.By comparing with the previous work,proposed work designed to achieve the efficient results.Overall design is per-formed with MATLAB 2020a tool.

基于字词向量融合的民航智慧监管短文本分类

为解决民航监管事项所产生的检查记录仅依靠人工进行分类分析导致效率低的问题,提出一种基于数据增强与字词向量融合的双通道特征提取的短文本分类模型,探讨民航监管事项的分类,包括与人、设备设施环境、制度程序和机构职责等相关问题。为解决类别不平衡问题,采用数据增强算法在原始文本上进行变换,生成新的样本,使各个类别的样本数量更加均衡。将字向量和词向量按字融合拼接,得到具有词特征信息的字向量。将字词融合的向量分别送入到文本卷积神经网络(TextCNN)和双向长短期记忆(BiLSTM)模型中进行不同维度的特征提取,从局部的角度和全局的角度分别提取特征,并在民航监管事项检查记录数据集上进行试验。结果表明:该模型准确率为0.9837,F 1值为0.9836。与一些字嵌入模型和词嵌入模型相对比,准确率提升0.4%。和一些常用的单通道模型相比,准确率提升3%,验证了双通道模型提取的特征具有全面性和有效性。

基于煤矿井下不安全行为知识图谱构建方法

虽然知识图谱已广泛应用于各个领域,但在煤矿安全方面,尤其在煤矿井下不安全行为方面的研究较少。构建了一种自底向上的煤矿井下不安全行为知识图谱。首先,采用传统机器学习和深度学习算法相结合的方法进行命名实体识别,采用RoBERTa进行词语向量化,采用双向长短时记忆网络(BiLSTM)对向量进行标注,提高网络模型对上下文特征的捕捉能力,通过多层感知机(MLP)解决煤矿井下不安全行为数据集数据量不足的问题,采用条件随机场(CRF)模型解决前面存在的单词关系不识别问题,并捕获全文信息和预测结果。其次,根据语句的结构特点,设计了基于知识“实体-关系-实体”三元组的依存句法树结构,对井下不安全行为领域的知识资源进行知识抽取与表示。最后,构建面向井下不安全行为的知识图谱。实验结果表明:(1) RoBERTaBiLSTM-MLP-CRF模型对于导致结果、违反性行为、错误性行为及粗心性行为4类实体类别具有较好的识别效果,其准确率分别为86.7%,80.3%,80.7%,77.4%。(2)在相同的数据集下,RoBERTa-BiLSTM-MLP-CRF模型训练的准确率、召回率、F1值较RoBERTa-BiLSTM-CRF模型分别提高了1.6%,1.5%,1.6%。

基于用户偏好的多媒体视频个性化智能推荐方法研究

为了满足用户个性化需求,提出基于用户偏好的多媒体视频个性化智能推荐方法。采集用户历史浏览行为数据,对提取的视频特征向量作相似度计算,生成视频候选集,建立视频浏览兴趣点(PoI)关联图,根据用户浏览时间计算其对不同类别视频PoI的兴趣偏好度,选择排名靠前的N个视频生成推荐列表,实现多媒体视频个性化智能推荐。实验结果表明,该方法可实现多媒体视频个性化推荐,当其维度为180时,视频推荐的Top-1指标最高;兴趣吻合度指标均值达到94.9%,Top-1均值为95.51%。

基于主题词向量中心点的K-means文本聚类算法

K-means由于其时间复杂度低运行速度快一直是最为流行的聚类算法之一,但是该算法在进行聚类时需要预先给出聚类个数和初始类中心点,其选取得合适与否会直接影响最终聚类效果。该文对初始类中心和迭代类中心的选取进行大量研究,根据决策图进行初始类中心的选择,利用每个类簇的主题词向量替代均值作为迭代类中心。实验表明,该文的初始点选取方法能够准确地选取初始点,且利用主题词向量作为迭代类中心能够很好地避免噪声点和噪声特征的影响,很大程度上地提高了K-means算法的性能。

多类表情符号短文本情感分析模型研究

相关研究数据表明在使用社交网络进行社交活动的95后中有69.8%的网民倾向于使用各类表情符号表述情感倾向。表情符号的高频使用与其自身带有的鲜明情感倾向使得表情符号成为文本情感分析的重要语料资源。基于此,提出了一种多类表情符号的短文本情感模型EMME。模型针对Twitter语料库以5类表情符号融入文本语言进行情感分析,首先利用CBOW模型构建词向量,继而使用卷积对拼接的词向量进行特征融合,后使用MLP实现文本正负情感分类,并针对5类表情符号与文本情感概率进行线性回归。实验数据表明对含各类表情符号短文本情感倾向判别中,EMME模型相比于MNB模型、SVM模型以及EMB模型的MacroF1值分别提高了14.81%、10.42%与9.01%;且EMME模型在不同样本容量规模中均取得了最好的分类准确率。

基于A-BiLSTM和CNN的文本分类

为解决单一神经网络不能获取准确全局文本信息的问题,提出一种基于A-BiLSTM双通道和优化CNN的文本分类模型。A-BiLSTM双通道层使用注意力机制关注对文本分类贡献值较大的部分,并用BiLSTM提取文本中上下文语义信息;A-BiLSTM双通道层中将两者输出的特征信息融合,得到高级语义;A-BiLSTM双通道层后,使用优化CNN的强学习能力提取关键局部特征,得到最终文本特征表示。分类器输出文本信息的类别。实验结果表明,该模型分类效果优于其它对比模型,具有良好的泛化能力。

基于Word2Vec和Bi-GRU的高职线上教评情感分析法试探

为提高高职线上教评情感分析的准确度和效率,提出基于双向门控循环单元(Bi-GRU)网络的教评情感分析法。该方法利用Skip-gram神经网络学习教育领域特定的词嵌入向量,再利用两个相同架构的Bi-GRU网络,从不同角度实现对学生反馈的细粒度分析。实验结果表明,该方法内容分类和情感分类的准确度分别达到97%和95%,显著优于支持向量机(SVM)、长短时记忆网络(LSTM)等其他方法。