To promote behavioral change among adolescents in Zambia, the National HIV/AIDS/STI/TB Council, in collaboration with UNICEF, developed the Zambia U-Report platform. This platform provides young people with improved a...To promote behavioral change among adolescents in Zambia, the National HIV/AIDS/STI/TB Council, in collaboration with UNICEF, developed the Zambia U-Report platform. This platform provides young people with improved access to information on various Sexual Reproductive Health topics through Short Messaging Service (SMS) messages. Over the years, the platform has accumulated millions of incoming and outgoing messages, which need to be categorized into key thematic areas for better tracking of sexual reproductive health knowledge gaps among young people. The current manual categorization process of these text messages is inefficient and time-consuming and this study aims to automate the process for improved analysis using text-mining techniques. Firstly, the study investigates the current text message categorization process and identifies a list of categories adopted by counselors over time which are then used to build and train a categorization model. Secondly, the study presents a proof of concept tool that automates the categorization of U-report messages into key thematic areas using the developed categorization model. Finally, it compares the performance and effectiveness of the developed proof of concept tool against the manual system. The study used a dataset comprising 206,625 text messages. The current process would take roughly 2.82 years to categorise this dataset whereas the trained SVM model would require only 6.4 minutes while achieving an accuracy of 70.4% demonstrating that the automated method is significantly faster, more scalable, and consistent when compared to the current manual categorization. These advantages make the SVM model a more efficient and effective tool for categorizing large unstructured text datasets. These results and the proof-of-concept tool developed demonstrate the potential for enhancing the efficiency and accuracy of message categorization on the Zambia U-report platform and other similar text messages-based platforms.展开更多
Purpose： The late Don R. Swanson was well appreciated during his lifetime as Dean of the Graduate Library School at University of Chicago, as winner of the American Society for Information Science Award of Merit for ...Purpose： The late Don R. Swanson was well appreciated during his lifetime as Dean of the Graduate Library School at University of Chicago, as winner of the American Society for Information Science Award of Merit for 2000, and as author of many seminal articles. In this informal essay, I will give my personal perspective on Don＇s contributions to science, and outline some current and future directions in literature-based discovery that are rooted in concepts that he developed.Design/methodology/approach： Personal recollections and literature review. Findings： The Swanson A-B-C model of literature-based discovery has been successfully used by laboratory investigators analyzing their findings and hypotheses. It continues to be a fertile area of research in a wide range of application areas including text mining, drug repurposing, studies of scientific innovation, knowledge discovery in databases, and bioinformatics. Recently, additional modes of discovery that do not follow the A-B-C model have also been proposed and explored （e.g. so-called storytelling, gaps, analogies, link prediction, negative consensus, outliers, and revival of neglected or discarded research questions）. Research limitations： This paper reflects the opinions of the author and is not a comprehensive nor technically based review of literature-based discovery. Practical implications： The general scientific public is still not aware of the availability of tools for literature-based discovery. Our Arrowsmith project site maintains a suite of discovery tools that are free and open to the public （http：//arrowsmith.psych.uic.edu）, as does BITOLA which is maintained by Dmitar Hristovski （http：// http：//ibmi.mf.uni-lj.si/bitola）, and Epiphanet which is maintained by Trevor Cohen （http：//epiphanet.uth.tme.edu/）. Bringing user-friendly tools to the public should be a high priority, since even more than advancing basic research in informatics, it is vital that we ensure that scientists actually use discovery tools and that these are actually able to help them make experimental discoveries in the lab and in the clinic. Originality/value： This paper discusses problems and issues which were inherent in Don＇s thoughts during his life, including those which have not yet been fully taken up and studied systematically.展开更多
With the dramatically development of Internet, the information processing and management technology onWWW have become a great important branch of data mining and data warehouse. Especially, nowadays, Text Miningis mar...With the dramatically development of Internet, the information processing and management technology onWWW have become a great important branch of data mining and data warehouse. Especially, nowadays, Text Miningis marvelously emerging and plays an important role in interrelated fields. So it is worth summarizing the contentabout text mining from its definition to relational methods and techniques. In this paper, combined to comparativelymature data mining technology, we present the definition of text mining and the multi-stage text mining process mod-el. Moreover, this paper roundly introduces the key areas of text mining and some of the powerful text analysis tech-niques, including: Word Automatic Segmenting, Feature Representation, Feature Extraction, Text Categorization,Text Clustering, Text Summarization, Information Extraction, Pattern Quality Evaluation, etc. These techniquescover the whole process from information preprocessing to knowledge obtaining.展开更多
1.引言大家熟知,所谓“数据丰富但知识缺乏“的现状导致了数据挖掘(Data Mining)技术研究的兴起,数据挖掘又称数据库知识发现(Knowledge Discovery in Databases)是从海量的结构化信息中抽取或挖掘隐含信息和知识的重要方法和途径。数...1.引言大家熟知,所谓“数据丰富但知识缺乏“的现状导致了数据挖掘(Data Mining)技术研究的兴起,数据挖掘又称数据库知识发现(Knowledge Discovery in Databases)是从海量的结构化信息中抽取或挖掘隐含信息和知识的重要方法和途径。数据挖掘技术已相当成熟。因为除了结构化的数据之外,在数字化信息中更多地存在大量自由、非结构化或半结构化的文本信息如新闻文章、电子书本、电子图书馆藏、Web页面内容、Email、文档数据库等,显然手工处理需要花费大量的人力物力,并且具有不确定性。展开更多
提出一种矩阵加权关联模式支持度计算方法及其相关定理,给出矩阵加权项集剪枝策略,基于该剪枝策略提出一种基于项权值变化的矩阵加权关联规则挖掘算法MWAR-Miner(matrix-weighted association rules-miner)。该算法克服现有的项无加权...提出一种矩阵加权关联模式支持度计算方法及其相关定理,给出矩阵加权项集剪枝策略,基于该剪枝策略提出一种基于项权值变化的矩阵加权关联规则挖掘算法MWAR-Miner(matrix-weighted association rules-miner)。该算法克服现有的项无加权和项权值固定条件下挖掘关联规则的缺陷,采用新的剪枝技术和模式支持度计算方法挖掘有效的矩阵加权关联规则,避免无效的和无趣的模式产生。以中文数据集CWT200g和英文数据集NTCIR-5为实验数据,理论分析和实验结果表明,与现有矩阵加权模式挖掘算法和基于无加权的挖掘算法比较,该算法挖掘的候选项集数量和挖掘时间明显减少,挖掘效率得到极大提高。展开更多
文摘To promote behavioral change among adolescents in Zambia, the National HIV/AIDS/STI/TB Council, in collaboration with UNICEF, developed the Zambia U-Report platform. This platform provides young people with improved access to information on various Sexual Reproductive Health topics through Short Messaging Service (SMS) messages. Over the years, the platform has accumulated millions of incoming and outgoing messages, which need to be categorized into key thematic areas for better tracking of sexual reproductive health knowledge gaps among young people. The current manual categorization process of these text messages is inefficient and time-consuming and this study aims to automate the process for improved analysis using text-mining techniques. Firstly, the study investigates the current text message categorization process and identifies a list of categories adopted by counselors over time which are then used to build and train a categorization model. Secondly, the study presents a proof of concept tool that automates the categorization of U-report messages into key thematic areas using the developed categorization model. Finally, it compares the performance and effectiveness of the developed proof of concept tool against the manual system. The study used a dataset comprising 206,625 text messages. The current process would take roughly 2.82 years to categorise this dataset whereas the trained SVM model would require only 6.4 minutes while achieving an accuracy of 70.4% demonstrating that the automated method is significantly faster, more scalable, and consistent when compared to the current manual categorization. These advantages make the SVM model a more efficient and effective tool for categorizing large unstructured text datasets. These results and the proof-of-concept tool developed demonstrate the potential for enhancing the efficiency and accuracy of message categorization on the Zambia U-report platform and other similar text messages-based platforms.
基金supported by NIH grants R01LM010817 and P01AG039347
文摘Purpose： The late Don R. Swanson was well appreciated during his lifetime as Dean of the Graduate Library School at University of Chicago, as winner of the American Society for Information Science Award of Merit for 2000, and as author of many seminal articles. In this informal essay, I will give my personal perspective on Don＇s contributions to science, and outline some current and future directions in literature-based discovery that are rooted in concepts that he developed.Design/methodology/approach： Personal recollections and literature review. Findings： The Swanson A-B-C model of literature-based discovery has been successfully used by laboratory investigators analyzing their findings and hypotheses. It continues to be a fertile area of research in a wide range of application areas including text mining, drug repurposing, studies of scientific innovation, knowledge discovery in databases, and bioinformatics. Recently, additional modes of discovery that do not follow the A-B-C model have also been proposed and explored （e.g. so-called storytelling, gaps, analogies, link prediction, negative consensus, outliers, and revival of neglected or discarded research questions）. Research limitations： This paper reflects the opinions of the author and is not a comprehensive nor technically based review of literature-based discovery. Practical implications： The general scientific public is still not aware of the availability of tools for literature-based discovery. Our Arrowsmith project site maintains a suite of discovery tools that are free and open to the public （http：//arrowsmith.psych.uic.edu）, as does BITOLA which is maintained by Dmitar Hristovski （http：// http：//ibmi.mf.uni-lj.si/bitola）, and Epiphanet which is maintained by Trevor Cohen （http：//epiphanet.uth.tme.edu/）. Bringing user-friendly tools to the public should be a high priority, since even more than advancing basic research in informatics, it is vital that we ensure that scientists actually use discovery tools and that these are actually able to help them make experimental discoveries in the lab and in the clinic. Originality/value： This paper discusses problems and issues which were inherent in Don＇s thoughts during his life, including those which have not yet been fully taken up and studied systematically.
文摘With the dramatically development of Internet, the information processing and management technology onWWW have become a great important branch of data mining and data warehouse. Especially, nowadays, Text Miningis marvelously emerging and plays an important role in interrelated fields. So it is worth summarizing the contentabout text mining from its definition to relational methods and techniques. In this paper, combined to comparativelymature data mining technology, we present the definition of text mining and the multi-stage text mining process mod-el. Moreover, this paper roundly introduces the key areas of text mining and some of the powerful text analysis tech-niques, including: Word Automatic Segmenting, Feature Representation, Feature Extraction, Text Categorization,Text Clustering, Text Summarization, Information Extraction, Pattern Quality Evaluation, etc. These techniquescover the whole process from information preprocessing to knowledge obtaining.
文摘1.引言大家熟知,所谓“数据丰富但知识缺乏“的现状导致了数据挖掘(Data Mining)技术研究的兴起,数据挖掘又称数据库知识发现(Knowledge Discovery in Databases)是从海量的结构化信息中抽取或挖掘隐含信息和知识的重要方法和途径。数据挖掘技术已相当成熟。因为除了结构化的数据之外,在数字化信息中更多地存在大量自由、非结构化或半结构化的文本信息如新闻文章、电子书本、电子图书馆藏、Web页面内容、Email、文档数据库等,显然手工处理需要花费大量的人力物力,并且具有不确定性。
文摘提出一种矩阵加权关联模式支持度计算方法及其相关定理,给出矩阵加权项集剪枝策略,基于该剪枝策略提出一种基于项权值变化的矩阵加权关联规则挖掘算法MWAR-Miner(matrix-weighted association rules-miner)。该算法克服现有的项无加权和项权值固定条件下挖掘关联规则的缺陷,采用新的剪枝技术和模式支持度计算方法挖掘有效的矩阵加权关联规则,避免无效的和无趣的模式产生。以中文数据集CWT200g和英文数据集NTCIR-5为实验数据,理论分析和实验结果表明,与现有矩阵加权模式挖掘算法和基于无加权的挖掘算法比较,该算法挖掘的候选项集数量和挖掘时间明显减少,挖掘效率得到极大提高。
