A New Speculative Execution Algorithm Based on C4.5 Decision Tree for Hadoop

As a distributed computing platform, Hadoop provides an effective way to handle big data. In Hadoop, the completion time of job will be delayed by a straggler. Although the definitive cause of the straggler is hard to detect, speculative execution is usually used for dealing with this problem, by simply backing up those stragglers on alternative nodes. In this paper, we design a new Speculative Execution algorithm based on C4.5 Decision Tree, SECDT, for Hadoop. In SECDT, we speculate completion time of stragglers and also of backup tasks, based on a kind of decision tree method: C4.5 decision tree. After we speculate the completion time, we compare the completion time of stragglers and of the backup tasks, calculating their differential value, and selecting the straggler with the maximum differential value to start the backup task.Experiment result shows that the SECDT can predict execution time more accurately than other speculative execution methods, hence reduce the job completion time.

Land Evaluation Method Based on Decision Tree Produced by C4.5 and Fuzzy Decision

[Objective]The aim was to overcome the shortage of being difficult to build land evaluation model when the impact factors had continuous value in the traditional land evaluation process,as well as to improve the intelligibility of the land evaluation knowledge.[Method] The land evaluation method combining classification rule extracted by C4.5 algorithm with fuzzy decision was proposed in this study.[Result] The result of Second General Soil Survey of Guangdong Province had demonstrated that the method was convenient to extract classification rules,and by using only 100 rules,quantity correct rate 86.67% and area correct rate 84.80% of land evaluation could be obtained.[Conclusions] The use of C4.5 algorithm to obtain the rules,combined with fuzzy decision algorithm to build classifiers had got satisfactory results,which provided a practical algorithm for the land evaluation.

Research on Scholarship Evaluation System based on Decision Tree Algorithm

Under the modern education system of China, the annual scholarship evaluation is a vital thing for many of the collegestudents. This paper adopts the classification algorithm of decision tree C4.5 based on the bettering of ID3 algorithm and constructa data set of the scholarship evaluation system through the analysis of the related attributes in scholarship evaluation information.And also having found some factors that plays a significant role in the growing up of the college students through analysis and re-search of moral education, intellectural education and culture&PE.

Forecasting Model of Agro-meteorological Disaster Grade Based on Decision Tree

Based on the discuss of the basic concept of data mining technology and the decision tree method,combining with the data samples of wind and hailstorm disasters in some counties of Mudanjiang region,the forecasting model of agro-meteorological disaster grade was established by adopting the C4.5 classification algorithm of decision tree,which can forecast the direct economic loss degree to provide rational data mining model and obtain effective analysis results.

Effective use of FibroTest to generate decision trees in hepatitis C

AIM： To assess the usefulness of FibroTest to forecast scores by constructing decision trees in patients with chronic hepatitis C.METHODS： We used the C4.5 classification algorithm to construct decision trees with data from 261 patients with chronic hepatitis C without a liver biopsy. The FibroTest attributes of age, gender, bilirubin, apolipoprotein, haptoglobin, α2 macroglobulin, and γ-glutamyl transpeptidase were used as predictors, and the FibroTest score as the target. For testing, a 10-fold cross validation was used.RESULTS： The overall classification error was 14.9% （accuracy 85.1%）. FibroTest＇s cases with true scores of FO and F4 were classified with very high accuracy （18/20 for FO, 9/9 for FO-1 and 92/96 for F4） and the largest confusion centered on F3. The algorithm produced a set of compound rules out of the ten classification trees and was used to classify the 261 patients. The rules for the classification of patients in FO and F4 were effective in more than 75% of the cases in which they were tested.CONCLUSION： The recognition of clinical subgroups should help to enhance our ability to assess differences in fibrosis scores in clinical studies and improve our understanding of fibrosis progression,

A decision tree based decomposition method for oil refinery scheduling

Refinery scheduling attracts increasing concerns in both academic and industrial communities in recent years.However, due to the complexity of refinery processes, little has been reported for success use in real world refineries. In academic studies, refinery scheduling is usually treated as an integrated, large-scale optimization problem,though such complex optimization problems are extremely difficult to solve. In this paper, we proposed a way to exploit the prior knowledge existing in refineries, and developed a decision making system to guide the scheduling process. For a real world fuel oil oriented refinery, ten adjusting process scales are predetermined. A C4.5 decision tree works based on the finished oil demand plan to classify the corresponding category(i.e. adjusting scale). Then,a specific sub-scheduling problem with respect to the determined adjusting scale is solved. The proposed strategy is demonstrated with a scheduling case originated from a real world refinery.

An Active Rule Approach for Network Intrusion Detection with Enhanced C4.5 Algorithm

Intrusion detection systems provide additional defense capacity to a networked information system in addition to the security measures provided by the firewalls. This paper proposes an active rule based enhancement to the C4.5 algorithm for network intrusion detection in order to detect misuse behaviors of internal attackers through effective classification and decision making in computer networks. This enhanced C4.5 algorithm derives a set of classification rules from network audit data and then the generated rules are used to detect network intrusions in a real-time environment. Unlike most existing decision tree based approaches, the spawned rules generated and fired in this work are more effective because the information-theoretic approach minimizes the expected number of tests needed to classify an object and guarantees that a simple (but not necessarily the simplest) tree is found. The main advantage of this proposed algorithm is that the generalization ability of enhanced C4.5 decision trees is better than that of C4.5 decision trees. We have employed data from the third international knowledge discovery and data mining tools competition (KDDcup’99) to train and test the feasibility of this proposed model. By applying the enhanced C4.5 algorithm an average detection rate of 93.28 percent and a false positive rate of 0.7 percent have respectively been obtained in this work.

Application Comparison of Association Rules and C4.5 Rules in Land Evaluation

Association rules and C4.5 rules can overcome the shortage of the traditional land evaluation methods and improve the intelligibility and efficiency of the land evaluation knowledge.In order to compare these two kinds of classification rules in the application,two fuzzy classifiers were established by combining with fuzzy decision algorithm especially based on Second General Soil Survey of Guangdong Province.The results of experiments demonstrated that the fuzzy classifier based on association rules obtain a higher accuracy rate,but with more complex calculation process and more computational overhead;the fuzzy classifier based on C4.5 rules obtain a slightly lower accuracy,but with fast computation and simpler calculation.

Taiga: Performance Optimization of the C4.5 Decision Tree Construction Algorithm

Classification is an important machine learning problem, and decision tree construction algorithms are an important class of solutions to this problem. RainForest is a scalable way to implement decision tree construction algorithms. It consists of several algorithms, of which the best one is a hybrid between a traditional recursive implementation and an iterative implementation which uses more memory but involves less write operations. We propose an optimized algorithm inspired by RainForest. By using a more sophisticated switching criterion between the two algorithms, we are able to get a performance gain even when all statistical information fits in memory. Evaluations show that our method can achieve a performance boost of 2.8 times in average than the traditional recursive implementation.

Using AdaBoost Meta-Learning Algorithm for Medical News Multi-Document Summarization

Automatic text summarization involves reducing a text document or a larger corpus of multiple documents to a short set of sentences or paragraphs that convey the main meaning of the text. In this paper, we discuss about multi-document summarization that differs from the single one in which the issues of compression, speed, redundancy and passage selection are critical in the formation of useful summaries. Since the number and variety of online medical news make them difficult for experts in the medical field to read all of the medical news, an automatic multi-document summarization can be useful for easy study of information on the web. Hence we propose a new approach based on machine learning meta-learner algorithm called AdaBoost that is used for summarization. We treat a document as a set of sentences, and the learning algorithm must learn to classify as positive or negative examples of sentences based on the score of the sentences. For this learning task, we apply AdaBoost meta-learning algorithm where a C4.5 decision tree has been chosen as the base learner. In our experiment, we use 450 pieces of news that are downloaded from different medical websites. Then we compare our results with some existing approaches.

面向乳腺肿瘤的诊前问答系统决策模型构建研究

目的/意义运用决策树分类模型模拟专家问诊思路,预测潜在或已有乳腺肿瘤患者的疾病风险。方法/过程采用C 4.5经典分类算法和悲观剪枝法,对调研收集的病例数据进行患者预问诊的结果预测。结果/结论生成一棵以“术后化疗or放疗在院是否结束”为根节点、拥有76个叶子节点的C 4.5决策树,预测准确率达95%,并根据分类标签划分为3个风险等级。

Data mining and well logging interpretation: application to a conglomerate reservoir

Data mining is the process of extracting implicit but potentially useful information from incomplete, noisy, and fuzzy data. Data mining offers excellent nonlinear modeling and self-organized learning, and it can play a vital role in the interpretation of well logging data of complex reservoirs. We used data mining to identify the lithologies in a complex reservoir. The reservoir lithologies served as the classification task target and were identified using feature extraction, feature selection, and modeling of data streams. We used independent component analysis to extract information from well curves. We then used the branch-and- bound algorithm to look for the optimal feature subsets and eliminate redundant information. Finally, we used the C5.0 decision-tree algorithm to set up disaggregated models of the well logging curves. The modeling and actual logging data were in good agreement, showing the usefulness of data mining methods in complex reservoirs.

基于聚类与决策树的综合入侵检测算法研究

入侵检测是一种通过实时监测目标系统来发现入侵攻击行为的安全技术,传统的入侵检测系统在有效性、适应性和可扩展性方面都存在着不足。为了使模糊聚类算法获得的聚类结果为全局最优解,改进了传统的模糊C-均值算法,并且在每个聚类的数据集上建立一棵属于该聚类的C4.5决策树,构造了一种新的综合检测算法来确定是否存在入侵。通过实验结果分析,该检测算法降低了误报率,提高了入侵检测的检测性能以及可靠性。

Study on the Grouping of Patients with Chronic Infectious Diseases Based on Data Mining

Objective: According to RFM model theory of customer relationship management, data mining technology was used to group the chronic infectious disease patients to explore the effect of customer segmentation on the management of patients with different characteristics. Methods: 170,246 outpatient data was extracted from the hospital management information system (HIS) during January 2016 to July 2016, 43,448 data was formed after the data cleaning. K-Means clustering algorithm was used to classify patients with chronic infectious diseases, and then C5.0 decision tree algorithm was used to predict the situation of patients with chronic infectious diseases. Results: Male patients accounted for 58.7%, patients living in Shanghai accounted for 85.6%. The average age of patients is 45.88 years old, the high incidence age is 25 to 65 years old. Patients was gathered into three categories: 1) Clusters 1—Important patients (4786 people, 11.72%, R = 2.89, F = 11.72, M = 84,302.95);2) Clustering 2—Major patients (23,103, 53.2%, R = 5.22, F = 3.45, M = 9146.39);3) Cluster 3—Potential patients (15,559 people, 35.8%, R = 19.77, F = 1.55, M = 1739.09). C5.0 decision tree algorithm was used to predict the treatment situation of patients with chronic infectious diseases, the final treatment time (weeks) is an important predictor, the accuracy rate is 99.94% verified by the confusion model. Conclusion: Medical institutions should strengthen the adherence education for patients with chronic infectious diseases, establish the chronic infectious diseases and customer relationship management database, take the initiative to help them improve treatment adherence. Chinese governments at all levels should speed up the construction of hospital information, establish the chronic infectious disease database, strengthen the blocking of mother-to-child transmission, to effectively curb chronic infectious diseases, reduce disease burden and mortality.

一种有效的C4.5改进模型

介绍了一种有效的决策树改进模型:R-C 4.5及其简化版本,旨在构造一棵简单的树,同时提高决策树属性选择度量的可解释性,减少空枝和无意义分枝,以及过度拟合。该决策树模型基于著名的C 4.5决策树模型,但在属性的选取和分枝策略上进行了改进。在R-C 4.5中,通过合并分类效果差的分枝,有效避免了碎片等问题。实验表明,R-C 4.5决策树在保持模型预测准确率的同时,有效改进了树的健壮性。作为R-C 4.5的简化版本,R-C 4.5c和R-C 4.5s可生成更为简单的树,而且R-C 4.5s通过数据预处理阶段完成,易于实现。

Improving naive Bayes classifier by dividing its decision regions

Classification can be regarded as dividing the data space into decision regions separated by decision boundaries.In this paper we analyze decision tree algorithms and the NBTree algorithm from this perspective.Thus,a decision tree can be regarded as a classifier tree,in which each classifier on a non-root node is trained in decision regions of the classifier on the parent node.Meanwhile,the NBTree algorithm,which generates a classifier tree with the C4.5 algorithm and the naive Bayes classifier as the root and leaf classifiers respectively,can also be regarded as training naive Bayes classifiers in decision regions of the C4.5 algorithm.We propose a second division (SD) algorithm and three soft second division (SD-soft) algorithms to train classifiers in decision regions of the naive Bayes classifier.These four novel algorithms all generate two-level classifier trees with the naive Bayes classifier as root classifiers.The SD and three SD-soft algorithms can make good use of both the information contained in instances near decision boundaries,and those that may be ignored by the naive Bayes classifier.Finally,we conduct experiments on 30 data sets from the UC Irvine (UCI) repository.Experiment results show that the SD algorithm can obtain better generali-zation abilities than the NBTree and the averaged one-dependence estimators (AODE) algorithms when using the C4.5 algorithm and support vector machine (SVM) as leaf classifiers.Further experiments indicate that our three SD-soft algorithms can achieve better generalization abilities than the SD algorithm when argument values are selected appropriately.

A New Framework for Scholarship Predictor Using a Machine Learning Approach

Education is the base of the survival and growth of any state,but due to resource scarcity,students,particularly at the university level,are forced into a difficult situation.Scholarships are the most significant financial aid mechanisms developed to overcome such obstacles and assist the students in continuing with their higher studies.In this study,the convoluted situation of scholarship eligibility criteria,including parental income,responsibilities,and academic achievements,is addressed.In an attempt to maximize the scholarship selection process,numerous machine learning algorithms,including Support Vector Machines,Neural Networks,K-Nearest Neighbors,and the C4.5 algorithm,were applied.The C4.5 algorithm,owing to its efficiency in the prediction of scholarship beneficiaries based on extraneous factors,was capable of predicting a phenomenal 95.62%of predictions using extensive data of a well-esteemed government sector university from Pakistan.This percentage is 4%and 15%better than the remainder of the methods tested,and it depicts the extent of the potential for the technique to enhance the scholarship selection process.The Decision Support Systems(DSS)would not only save the administrative cost but would also create a fair and transparent process in place.In a world where accessibility to education is the key,this research provides data-oriented consolidation to ensure that deserving students are helped and allowed to get the financial assistance that they need to reach higher studies and bridge the gap between the demands of the day and the institutions of intellect.

C 5.0决策树对早期胃癌风险筛查研究

目的 C 5.0算法改进C 4.5算法以提高分类效率和准确性,越来越广泛地应用于处理分类问题。本研究拟根据患者问卷调查和血清学检查等资料,利用C 5.0决策树算法筛查早期胃癌风险,筛选对早期胃癌风险筛查影响较大的因素,进而辅助临床提高早期胃癌的诊断筛查。方法资料来自与广东药科大学附属第一医院的合作项目"基于云计算的早期胃癌筛查创新平台",对广东省6个市近30家医院消化内科就诊的618例胃病患者进行问卷调查,并收集其血清学检查和内镜检查及病理活组织检查资料。根据内镜检查和病理活组织检查结果将患者分为早期胃癌低危、中危及高危3类,用合成少数过采样技术(synthetic minority oversampling technique,SMOTE)方法处理样本分类不平衡问题,然后根据C 5.0算法建立早期胃癌风险筛查的决策树模型。结果产生1棵深度为11、共33个叶子节点的C 5.0决策树模型,对应有33条易于理解的分类规则,根据这些分类规则可快速评估患者的早期胃癌风险类型。建立的C 5.0决策树模型有较高的准确率,达73.28%,且增益图中曲线上凸明显,接近理想曲线,能较好地对早期胃癌风险进行分类预测。决策树模型计算各指标对早期胃癌风险预测的重要性,筛选出15个对早期胃癌风险筛查影响较大的因素,其中影响最大的因素是幽门螺旋杆菌(helicobacter pylori,Hp)抗体。结论基于患者问卷调查和血清学检查构建的C 5.0决策树模型对早期胃癌风险的预测效果较好,选出对早期胃癌风险筛查影响较大的因素,可辅助临床早期胃癌风险筛查。