Sentiment Analysis on Twitter Data Using Term Frequency-Inverse Document Frequency

This study is an exploratory analysis of applying natural language processing techniques such as Term Frequency-Inverse Document Frequency and Sentiment Analysis on Twitter data. The uniqueness of this work is established by determining the overall sentiment of a politician’s tweets based on TF-IDF values of terms used in their published tweets. By calculating the TF-IDF value of terms from the corpus, this work displays the correlation between TF-IDF score and polarity. The results of this work show that calculating the TF-IDF score of the corpus allows for a more accurate representation of the overall polarity since terms are given a weight based on their uniqueness and relevance rather than just the frequency at which they appear in the corpus.

基于改进的TF-IDF算法及共现词的主题词抽取算法

信息主题的抽取是快速定位用户需求的基础任务,主题词抽取时主要存在三个问题:一是词语权重的计算,二是词语间关系的度量,三是数据维度灾难.在计算词权重时首先利用互信息确定共现词对,与词频、词性、词位置信息非线性组合,然后,根据词权重构建文档—共现词矩阵并建立潜在语义分析(Latent Semantic Analysis,LSA)模型.该方法借助LSA模型的奇异值分解(Singular Value Decomposition,SVD)将文档—共现词矩阵映射到潜在语义空间,不仅实现数据降维,而且获得低维度的文档相似矩阵.最后,对文档相似矩阵进行k-means聚类,在同类文档中选出词权重最大的前几对共现词,作为该类文章的主题词.对比基于TF-IDF(Term Frequency-Inverse Document Frequency)和共现词抽取主题词的实验,该算法的准确度分别提高了19%和10%.

An improved TF-IDF approach for text classification

This paper presents a new improved term frequency/inverse document frequency (TF-IDF) approach which uses confidence, support and characteristic words to enhance the recall and precision of text classification. Synonyms defined by a lexicon are processed in the improved TF-IDF approach. We detailedly discuss and analyze the relationship among confidence, recall and precision. The experiments based on science and technology gave promising results that the new TF-IDF approach improves the precision and recall of text classification compared with the conventional TF-IDF approach.

Environmental complaint insights through text mining based on the driver,pressure,state,impact,and response(DPSIR)framework:Evidence from an Italian environmental agency

Individuals,local communities,environmental associations,private organizations,and public representatives and bodies may all be aggrieved by environmental problems concerning poor air quality,illegal waste disposal,water contamination,and general pollution.Environmental complaints represent the expressions of dissatisfaction with these issues.As the timeconsuming of managing a large number of complaints,text mining may be useful for automatically extracting information on stakeholder priorities and concerns.The paper used text mining and semantic network analysis to crawl relevant keywords about environmental complaints from two online complaint submission systems:online claim submission system of Regional Agency for Prevention,Environment and Energy(Arpae)(“Contact Arpae”);and Arpae's internal platform for environmental pollution(“Environmental incident reporting portal”)in the Emilia-Romagna Region,Italy.We evaluated the total of 2477 records and classified this information based on the claim topic(air pollution,water pollution,noise pollution,waste,odor,soil,weather-climate,sea-coast,and electromagnetic radiation)and geographical distribution.Then,this paper used natural language processing to extract keywords from the dataset,and classified keywords ranking higher in Term Frequency-Inverse Document Frequency(TF-IDF)based on the driver,pressure,state,impact,and response(DPSIR)framework.This study provided a systemic approach to understanding the interaction between people and environment in different geographical contexts and builds sustainable and healthy communities.The results showed that most complaints are from the public and associated with air pollution and odor.Factories(particularly foundries and ceramic industries)and farms are identified as the drivers of environmental issues.Citizen believed that environmental issues mainly affect human well-being.Moreover,the keywords of“odor”,“report”,“request”,“presence”,“municipality”,and“hours”were the most influential and meaningful concepts,as demonstrated by their high degree and betweenness centrality values.Keywords connecting odor(classified as impacts)and air pollution(classified as state)were the most important(such as“odor-burnt plastic”and“odor-acrid”).Complainants perceived odor annoyance as a primary environmental concern,possibly related to two main drivers:“odor-factory”and“odorsfarms”.The proposed approach has several theoretical and practical implications:text mining may quickly and efficiently address citizen needs,providing the basis toward automating(even partially)the complaint process;and the DPSIR framework might support the planning and organization of information and the identification of stakeholder concerns and priorities,as well as metrics and indicators for their assessment.Therefore,integration of the DPSIR framework with the text mining of environmental complaints might generate a comprehensive environmental knowledge base as a prerequisite for a wider exploitation of analysis to support decision-making processes and environmental management activities.

Spark 平台下基于加权词向量的文本分类方法

针对Spark平台下文本分类中文本表示方法不够完善,导致分类准确率低的问题,结合SparkML下的TF-IDF算法和Word2vec模型,提出一种基于SparkML的加权词向量文本表示方法。首先对文本进行分词,去停用词等预处理,基于SparkML计算出每个词语的词频和逆文档频率,同时计算词语的词向量。使用词语的TF-IDF值作为词向量的权重,将文本表示为加权词向量,再使用SVM分类器进行分类。在THUNews数据集上进行实验。实验结果表明,该方法相比于传统的TF-IDF算法、平均Word2Vec词向量文本表示,可以提升分类的精度。

Hybrid Approach to Document Anomaly Detection:An Application to Facilitate RPA in Title Insurance

Anomaly detection(AD)is an important aspect of various domains and title insurance(TI)is no exception.Robotic process automation(RPA)is taking over manual tasks in TI business processes,but it has its limitations without the support of artificial intelligence(AI)and machine learning(ML).With increasing data dimensionality and in composite population scenarios,the complexity of detecting anomalies increases and AD in automated document management systems(ADMS)is the least explored domain.Deep learning,being the fastest maturing technology can be combined along with traditional anomaly detectors to facilitate and improve the RPAs in TI.We present a hybrid model for AD,using autoencoders(AE)and a one-class support vector machine(OSVM).In the present study,OSVM receives input features representing real-time documents from the TI business,orchestrated and with dimensions reduced by AE.The results obtained from multiple experiments are comparable with traditional methods and within a business acceptable range,regarding accuracy and performance.

基于大数据的文化遗产认知分析方法——以北京旧城中轴线为例

以北京旧城中轴文化遗产为例,利用2012、2015年的相关微博、报刊新闻、学术文献数据,通过提取关键词,抽取词频、tf-idf权重、互信息、后验概率等特征,从群体、时间、空间多个维度分析文化遗产的认知。在人群维度上,通过具有特征性人群的传媒信息,发现不同人群对文化遗产的认识存在异同:对于中轴文化遗产核心单元故宫、天安门、天坛的认知相对一致,而对于钟楼鼓楼、太庙、地安门的认识,官方偏向于行政管理,学者偏向于历史价值,大众则偏向于生活化。在时间维度上,提取文化遗产关注程度和认知变化。如相对于2015年,大众对故宫、天安门的关注程度相对提高,对太庙的历史价值认识更为丰富。大众相对于官方和学者对文化遗产的认知更容易发生变化,且对热点事件敏感。在空间维度上,挖掘文化遗产单元之间的认知转移和关联模式,一方面,空间上相连的天安门—正阳门—正阳门大街具有较高的双向认知;另一方面,中轴文化遗产中,故宫、天安门、天坛的后验概率较高,表现出跨空间的认知汇聚模式。基于大数据的认知分析方法,是问卷调查、文献调研、访谈分析等传统方法的重要补充方式,能够降低数据收集者的主观影响,增加分析维度和效率,有助于发现隐含的知识和模式。本文结论可为文化遗产价值挖掘、保护提供决策支持。

Fusion Model for Tentative Diagnosis Inference Based on Clinical Narratives

In general,physicians make a preliminary diagnosis based on patients’admission narratives and admission conditions,largely depending on their experiences and professional knowledge.An automatic and accurate tentative diagnosis based on clinical narratives would be of great importance to physicians,particularly in the shortage of medical resources.Despite its great value,little work has been conducted on this diagnosis method.Thus,in this study,we propose a fusion model that integrates the semantic and symptom features contained in the clinical text.The semantic features of the input text are initially captured by an attention-based Bidirectional Long Short-Term Memory(BiLSTM)network.The symptom concepts,recognized from the input text,are then vectorized by using the term frequency-inverse document frequency method based on the relations between symptoms and diseases.Finally,two fusion strategies are utilized to recommend the most potential candidate for the international classification of diseases code.Model training and evaluation are performed on a public clinical dataset.The results show that both fusion strategies achieved a promising performance,in which the best performance obtained a top-3 accuracy of 0.7412.

Spontaneous Language Analysis in Alzheimer’s Disease:Evaluation of Natural Language Processing Technique for Analyzing Lexical Performance

Language disorder,a common manifestation of Alzheimer’s disease(AD),has attracted widespread attention in recent years.This paper uses a novel natural language processing(NLP)method,compared with latest deep learning technology,to detect AD and explore the lexical performance.Our proposed approach is based on two stages.First,the dialogue contents are summarized into two categories with the same category.Second,term frequency—inverse document frequency(TF-IDF)algorithm is used to extract the keywords of transcripts,and the similarity of keywords between the groups was calculated separately by cosine distance.Several deep learning methods are used to compare the performance.In the meanwhile,keywords with the best performance are used to analyze AD patients’lexical performance.In the Predictive Challenge of Alzheimer’s Disease held by iFlytek in 2019,the proposed AD diagnosis model achieves a better performance in binary classification by adjusting the number of keywords.The F1 score of the model has a considerable improvement over the baseline of 75.4%,and the training process of which is simple and efficient.We analyze the keywords of the model and find that AD patients use less noun and verb than normal controls.A computer-assisted AD diagnosis model on small Chinese dataset is proposed in this paper,which provides a potential way for assisting diagnosis of AD and analyzing lexical performance in clinical setting.