SMART: Speedup Job Completion Time by Scheduling Reduce Tasks

Distributed computing systems have been widely used as the amount of data grows exponentially in the era of information explosion. Job completion time (JCT) is a major metric for assessing their effectiveness. How to reduce the JCT for these systems through reasonable scheduling has become a hot issue in both industry and academia. Data skew is a common phenomenon that can compromise the performance of such distributed computing systems. This paper proposes SMART, which can effectively reduce the JCT through handling the data skew during the reducing phase. SMART predicts the size of reduce tasks based on part of the completed map tasks and then enforces largest-first scheduling in the reducing phase according to the predicted reduce task size. SMART makes minimal modifications to the original Hadoop with only 20 additional lines of code and is readily deployable. The robustness and the effectiveness of SMART have been evaluated with a real-world cluster against a large number of datasets. Experiments show that SMART reduces JCT by up to 6.47%, 9.26%, and 13.66% for Terasort, WordCount and InvertedIndex respectively with the Purdue MapReduce benchmarks suite (PUMA) dataset.

Optimal Resource Allocation and Quality of Service Prediction in Cloud

In the present scenario,cloud computing service provides on-request access to a collection of resources available in remote system that can be shared by numerous clients.Resources are in self-administration;consequently,clients can adjust their usage according to their requirements.Resource usage is estimated and clients can pay according to their utilization.In literature,the existing method describes the usage of various hardware assets.Quality of Service(QoS)needs to be considered for ascertaining the schedule and the access of resources.Adhering with the security arrangement,any additional code is forbidden to ensure the usage of resources complying with QoS.Thus,all monitoring must be done from the hypervisor.To overcome the issues,Robust Resource Allocation and Utilization(RRAU)approach is developed for optimizing the management of its cloud resources.The work hosts a numerous virtual assets which could be expected under the circumstances and it enforces a controlled degree of QoS.The asset assignment calculation is heuristic,which is based on experimental evaluations,RRAU approach with J48 prediction model reduces Job Completion Time(JCT)by 4.75 s,Make Span(MS)6.25,and Monetary Cost(MC)4.25 for 15,25,35 and 45 resources are compared to the conventional methodologies in cloud environment.

Improved Heuristic Job Scheduling Method to Enhance Throughput for Big Data Analytics

Data-parallel computing platforms,such as Hadoop and Spark,are deployed in computing clusters for big data analytics.There is a general tendency that multiple users share the same computing cluster.The schedule of multiple jobs becomes a serious challenge.Over a long period in the past,the Shortest-Job-First(SJF)method has been considered as the optimal solution to minimize the average job completion time.However,the SJF method leads to a low system throughput in the case where a small number of short jobs consume a large amount of resources.This factor prolongs the average job completion time.We propose an improved heuristic job scheduling method,called the Densest-Job-Set-First(DJSF)method.The DJSF method schedules jobs by maximizing the number of completed jobs per unit time,aiming to decrease the average Job Completion Time(JCT)and improve the system throughput.We perform extensive simulations based on Google cluster data.Compared with the SJF method,the DJSF method decreases the average JCT by 23.19% and enhances the system throughput by 42.19%.Compared with Tetris,the job packing method improves the job completion efficiency by 55.4%,so that the computing platforms complete more jobs in a short time span.