A New Speculative Execution Algorithm Based on C4.5 Decision Tree for Hadoop

As a distributed computing platform, Hadoop provides an effective way to handle big data. In Hadoop, the completion time of job will be delayed by a straggler. Although the definitive cause of the straggler is hard to detect, speculative execution is usually used for dealing with this problem, by simply backing up those stragglers on alternative nodes. In this paper, we design a new Speculative Execution algorithm based on C4.5 Decision Tree, SECDT, for Hadoop. In SECDT, we speculate completion time of stragglers and also of backup tasks, based on a kind of decision tree method: C4.5 decision tree. After we speculate the completion time, we compare the completion time of stragglers and of the backup tasks, calculating their differential value, and selecting the straggler with the maximum differential value to start the backup task.Experiment result shows that the SECDT can predict execution time more accurately than other speculative execution methods, hence reduce the job completion time.

Taiga: Performance Optimization of the C4.5 Decision Tree Construction Algorithm

Classification is an important machine learning problem, and decision tree construction algorithms are an important class of solutions to this problem. RainForest is a scalable way to implement decision tree construction algorithms. It consists of several algorithms, of which the best one is a hybrid between a traditional recursive implementation and an iterative implementation which uses more memory but involves less write operations. We propose an optimized algorithm inspired by RainForest. By using a more sophisticated switching criterion between the two algorithms, we are able to get a performance gain even when all statistical information fits in memory. Evaluations show that our method can achieve a performance boost of 2.8 times in average than the traditional recursive implementation.

Forecasting Model of Agro-meteorological Disaster Grade Based on Decision Tree

Based on the discuss of the basic concept of data mining technology and the decision tree method,combining with the data samples of wind and hailstorm disasters in some counties of Mudanjiang region,the forecasting model of agro-meteorological disaster grade was established by adopting the C4.5 classification algorithm of decision tree,which can forecast the direct economic loss degree to provide rational data mining model and obtain effective analysis results.

Effective use of FibroTest to generate decision trees in hepatitis C

AIM： To assess the usefulness of FibroTest to forecast scores by constructing decision trees in patients with chronic hepatitis C.METHODS： We used the C4.5 classification algorithm to construct decision trees with data from 261 patients with chronic hepatitis C without a liver biopsy. The FibroTest attributes of age, gender, bilirubin, apolipoprotein, haptoglobin, α2 macroglobulin, and γ-glutamyl transpeptidase were used as predictors, and the FibroTest score as the target. For testing, a 10-fold cross validation was used.RESULTS： The overall classification error was 14.9% （accuracy 85.1%）. FibroTest＇s cases with true scores of FO and F4 were classified with very high accuracy （18/20 for FO, 9/9 for FO-1 and 92/96 for F4） and the largest confusion centered on F3. The algorithm produced a set of compound rules out of the ten classification trees and was used to classify the 261 patients. The rules for the classification of patients in FO and F4 were effective in more than 75% of the cases in which they were tested.CONCLUSION： The recognition of clinical subgroups should help to enhance our ability to assess differences in fibrosis scores in clinical studies and improve our understanding of fibrosis progression,

A decision tree based decomposition method for oil refinery scheduling

Refinery scheduling attracts increasing concerns in both academic and industrial communities in recent years.However, due to the complexity of refinery processes, little has been reported for success use in real world refineries. In academic studies, refinery scheduling is usually treated as an integrated, large-scale optimization problem,though such complex optimization problems are extremely difficult to solve. In this paper, we proposed a way to exploit the prior knowledge existing in refineries, and developed a decision making system to guide the scheduling process. For a real world fuel oil oriented refinery, ten adjusting process scales are predetermined. A C4.5 decision tree works based on the finished oil demand plan to classify the corresponding category(i.e. adjusting scale). Then,a specific sub-scheduling problem with respect to the determined adjusting scale is solved. The proposed strategy is demonstrated with a scheduling case originated from a real world refinery.

Research on Scholarship Evaluation System based on Decision Tree Algorithm

Under the modern education system of China, the annual scholarship evaluation is a vital thing for many of the collegestudents. This paper adopts the classification algorithm of decision tree C4.5 based on the bettering of ID3 algorithm and constructa data set of the scholarship evaluation system through the analysis of the related attributes in scholarship evaluation information.And also having found some factors that plays a significant role in the growing up of the college students through analysis and re-search of moral education, intellectural education and culture&PE.

Data mining and well logging interpretation: application to a conglomerate reservoir

Data mining is the process of extracting implicit but potentially useful information from incomplete, noisy, and fuzzy data. Data mining offers excellent nonlinear modeling and self-organized learning, and it can play a vital role in the interpretation of well logging data of complex reservoirs. We used data mining to identify the lithologies in a complex reservoir. The reservoir lithologies served as the classification task target and were identified using feature extraction, feature selection, and modeling of data streams. We used independent component analysis to extract information from well curves. We then used the branch-and- bound algorithm to look for the optimal feature subsets and eliminate redundant information. Finally, we used the C5.0 decision-tree algorithm to set up disaggregated models of the well logging curves. The modeling and actual logging data were in good agreement, showing the usefulness of data mining methods in complex reservoirs.

Study on the Grouping of Patients with Chronic Infectious Diseases Based on Data Mining

Objective: According to RFM model theory of customer relationship management, data mining technology was used to group the chronic infectious disease patients to explore the effect of customer segmentation on the management of patients with different characteristics. Methods: 170,246 outpatient data was extracted from the hospital management information system (HIS) during January 2016 to July 2016, 43,448 data was formed after the data cleaning. K-Means clustering algorithm was used to classify patients with chronic infectious diseases, and then C5.0 decision tree algorithm was used to predict the situation of patients with chronic infectious diseases. Results: Male patients accounted for 58.7%, patients living in Shanghai accounted for 85.6%. The average age of patients is 45.88 years old, the high incidence age is 25 to 65 years old. Patients was gathered into three categories: 1) Clusters 1—Important patients (4786 people, 11.72%, R = 2.89, F = 11.72, M = 84,302.95);2) Clustering 2—Major patients (23,103, 53.2%, R = 5.22, F = 3.45, M = 9146.39);3) Cluster 3—Potential patients (15,559 people, 35.8%, R = 19.77, F = 1.55, M = 1739.09). C5.0 decision tree algorithm was used to predict the treatment situation of patients with chronic infectious diseases, the final treatment time (weeks) is an important predictor, the accuracy rate is 99.94% verified by the confusion model. Conclusion: Medical institutions should strengthen the adherence education for patients with chronic infectious diseases, establish the chronic infectious diseases and customer relationship management database, take the initiative to help them improve treatment adherence. Chinese governments at all levels should speed up the construction of hospital information, establish the chronic infectious disease database, strengthen the blocking of mother-to-child transmission, to effectively curb chronic infectious diseases, reduce disease burden and mortality.

An Active Rule Approach for Network Intrusion Detection with Enhanced C4.5 Algorithm

Intrusion detection systems provide additional defense capacity to a networked information system in addition to the security measures provided by the firewalls. This paper proposes an active rule based enhancement to the C4.5 algorithm for network intrusion detection in order to detect misuse behaviors of internal attackers through effective classification and decision making in computer networks. This enhanced C4.5 algorithm derives a set of classification rules from network audit data and then the generated rules are used to detect network intrusions in a real-time environment. Unlike most existing decision tree based approaches, the spawned rules generated and fired in this work are more effective because the information-theoretic approach minimizes the expected number of tests needed to classify an object and guarantees that a simple (but not necessarily the simplest) tree is found. The main advantage of this proposed algorithm is that the generalization ability of enhanced C4.5 decision trees is better than that of C4.5 decision trees. We have employed data from the third international knowledge discovery and data mining tools competition (KDDcup’99) to train and test the feasibility of this proposed model. By applying the enhanced C4.5 algorithm an average detection rate of 93.28 percent and a false positive rate of 0.7 percent have respectively been obtained in this work.

面向乳腺肿瘤的诊前问答系统决策模型构建研究

目的/意义运用决策树分类模型模拟专家问诊思路,预测潜在或已有乳腺肿瘤患者的疾病风险。方法/过程采用C 4.5经典分类算法和悲观剪枝法,对调研收集的病例数据进行患者预问诊的结果预测。结果/结论生成一棵以“术后化疗or放疗在院是否结束”为根节点、拥有76个叶子节点的C 4.5决策树,预测准确率达95%,并根据分类标签划分为3个风险等级。

基于SMOTE和决策树算法的电力变压器状态评估知识获取方法

提出基于合成少数过采样技术(SMOTE)算法和决策树算法的电力变压器状态评估知识获取方法,首先针对变压器非正常状态样本数量较少的情况,采用SMOTE算法补充非正常状态样本数量,解决了变压器样本集类别不平衡问题。然后将变压器状态评估过程视为分类过程,利用决策树模型为白箱模型的特点,将变压器状态评估知识获取问题转化为构建决策树的问题。最后采用C4.5决策树算法构建决策树,从中提取变压器状态评估知识,得到关键变压器状态量和评估规则。以某地市级供电公司110 kV电压等级油浸式变压器实际数据开展实例分析,结果表明所提出的方法能实现状态评估知识的自动化获取,可以为该地区110 kV油浸式变压器的状态评估工作提供决策支持。

基于决策树算法在人力资源推荐技术中的应用

研究基于决策树算法的人力资源推荐方法,提升人力资源推荐的综合质量与实际应用效果。采用流式分布式数据采集方式从海量数据源内采集人力资源数据,依据数据来源类别归类数据后,储存为原始人力资源数据集。针对原始人力资源数据集的缺陷,通过数据抽取、清洗转换及加载的预处理过程,实施数据预处理并构建数据仓库;将该数据仓库中的数据输入到通过改进ID3决策树算法改进的C4.5决策树算法中,该算法通过多层节点的反复分裂生成决策树,获得能够满足人力资源推荐预设终止条件的同类别分裂结果,实现人力资源推荐。结果表明,所提方法的最佳叶子节点数为15个,在此叶子节点数下该方法的召回率、F1值及覆盖率均较高,能够显著提升人力资源推荐的综合质量与推荐效果。

基于不平衡数据的公司破产预测研究

整合创新数据预处理技术与集成算法利用不平衡数据探讨了公司破产预测问题。首先,运用冗余信息处理方法、不同抽样方法等对不平衡数据进行预处理。其次,以5.0分类器(Classifier 5.0,C5.0)决策树和单隐层前馈神经网络作为基分类器,分别与三类重抽样数据预处理技术结合,择出最优抽样法。再次,结合自助汇聚法提升分类效果,并运用十折交叉验证的受试者操作特征曲线的下方面积进行评价,对比了两基分类器的集成模型。最后,运用加利福尼亚大学尔湾分校数据库中一万多家波兰制造业公司的实际数据进行实验验证。实验结果表明:欠抽样或人工少数类过采样法与神经网络结合的集成模型分类效果最优,为企业实施破产预测提供积极支撑。

数据挖掘技术在采摘机器人图像采集过程应用

针对采摘机器人果实识别速率较低导致采摘效率较低的问题,对数据挖掘技术在采摘机器人中图像采集过程的应用进行了分析。采摘机器人主要组成包括图像采集模块、运动控制模块、气压驱动模块、电源模块、微处理器模块和无线网传输模块。为了提升图像数据的处理速度,采用MR模型和决策树中的ID_(3)算法对图像数据进行处理,并构建决策树模型,对图像数据进行数据挖掘处理。为了验证该采摘机器人的性能,对其进行数据挖掘算法调试试验和采摘机器人性能试验,结果表明:该图像处理算法速度显著提升,采摘机器人性能稳定,采摘效果好。

数据挖掘技术在客户关系管理中的应用研究

在电子商务环境下,客户对企业起着至关重要的影响。新客户的获取无疑对企业的生存和发展起到很重要的作用。客户关系管理系统中,通过分析海量数据之间的联系,建立规范全面的信息模型。为了解决新客户的获取问题,采用数据挖掘技术对客户类别进行预测。通过对数据挖掘各种算法的比较,做了决策树算法编程实验,获得客户类别的预测结果。结果表明,数据挖掘技术能有效提高客户预测的准确率,提高了数据利用率。

Using AdaBoost Meta-Learning Algorithm for Medical News Multi-Document Summarization

Automatic text summarization involves reducing a text document or a larger corpus of multiple documents to a short set of sentences or paragraphs that convey the main meaning of the text. In this paper, we discuss about multi-document summarization that differs from the single one in which the issues of compression, speed, redundancy and passage selection are critical in the formation of useful summaries. Since the number and variety of online medical news make them difficult for experts in the medical field to read all of the medical news, an automatic multi-document summarization can be useful for easy study of information on the web. Hence we propose a new approach based on machine learning meta-learner algorithm called AdaBoost that is used for summarization. We treat a document as a set of sentences, and the learning algorithm must learn to classify as positive or negative examples of sentences based on the score of the sentences. For this learning task, we apply AdaBoost meta-learning algorithm where a C4.5 decision tree has been chosen as the base learner. In our experiment, we use 450 pieces of news that are downloaded from different medical websites. Then we compare our results with some existing approaches.

基于决策树的汽车配置规则预测系统研究

个性化汽车配置在欧美国家是一种流行的购车方案,目前配置规则判断系统研究领域尚属空白。本文提出了用于自动且有效率地判断客户特定需求的汽车订单是否有效的决策树模型,通过使用SQLServer提供的Analysis Service中的决策树算法和Weka的J48(C4.5算法)分别生成的决策树模型,对不同质量的配置订单群中训练数据和测试数据有效性进行了对比。本文在基于数据挖掘和决策树的研究基础之上,提出了基于决策树的汽车配置规则的预测系统,并且提出了软件系统的设计方案。实验结果表明该配置规则判断系统具有较好的实际应用价值。

基于决策树算法的心理健康智能评测研究

为了解决当前心理健康智能评测过程中存在的误评率高、工作效率低等问题,提出基于决策树算法的心理健康智能评测系统。首先分析心理健康智能评测的研究现状,构建心理健康智能评测系统框架;然后采集心理健康智能评测数据,采用决策树算法对心理健康智能评测数据进行分析和分类,得到心理健康智能评测结果;最后采用具体仿真实验分析了心理健康智能评测系统的可行性和优越性。结果表明,文中系统克服了当前心理健康智能评测系统存在的弊端,提高了心理健康智能评测精度,改善了心理健康智能评测效率,而且系统稳定性更好,可以满足当前心理健康智能评测的实际要求。

图书借阅热度的计算及预测研究

[目的/意义]研究如何将数据挖掘技术应用于图书采购决策。[方法/过程]以图书的有效借阅数为基础,以具有预测性质的决策指标——借阅热度为研究对象,运用决策树分类技术对历史借阅记录进行挖掘。[结果/结论]构建具有预测能力的模型,采用C4.5决策树算法预测候选采购图书借阅热度值,并进行实验验证。实验结果表明,决策树分类技术能够较为准确地预测候选图书借阅热度。

电网通信管理系统中电源数据信息处理方法研究

针对目前电网通信系统中通信电源管理的空白,引入大数据算法实现对电源数据信息的信息化管理。采用朴素贝叶斯分类算法来对数据进行分类,并融入C4.5决策树算法,实现不同种类数据分类。根据决策树模型计算每种属性的权重,将权重引入到朴素贝叶斯分类算法进行数据的分类,提高了分类精度。引入Xilinx XC7A200T型号的逻辑处理芯片、FPGA驱动、ADC HMCAD1520芯片对采集设备的上行端口进行逻辑设计,提高数据采集的效率和质量。试验表明,分类精度比朴素贝叶斯分类算法的分类精度提高了5%左右,数据采集方案的采集效率比传统的数据采集效率提高了15%左右。