Job schedulers for Big data processing in Hadoop environment： testing real-life schedulers using benchmark programs

At present, big data is very popular, because it has proved to be much successful in many fields such as social media, E-commerce transactions, etc. Big data describes the tools and technologies needed to capture, manage, store, distribute, and analyze petabyte or larger-sized datasets having different structures with high speed. Big data can be structured, unstructured, or semi structured. Hadoop is an open source framework that is used to process large amounts of data in an inexpensive and efficient way, and job scheduling is a key factor for achieving high performance in big data processing. This paper gives an overview of big data and highlights the problems and challenges in big data. It then highlights Hadoop Distributed File System （HDFS）, Hadoop MapReduce, and various parameters that affect the performance of job scheduling algorithms in big data such as Job Tracker, Task Tracker, Name Node, Data Node, etc. The primary purpose of this paper is to present a comparative study of job scheduling algorithms along with their experimental results in Hadoop environment. In addition, this paper describes the advantages, disadvantages, features, and drawbacks of various Hadoop job schedulers such as FIFO, Fair, capacity, Deadline Constraints, Delay, LATE, Resource Aware, etc, and provides a comparative study among these schedulers.

The Planning and Scheduling of Business Process in Agile Supply Chain System

The rapidly developing global competition is leadin g to the worldwide enterprise alliance, with which the geographical dispersion of production, assembly and distribution operations comes into being. Supply chain system is such a kind of enterprise alliance, managing the material and informat ion flows both in and between enterprises, such as vendors, manufacturing and assembly plants and distribution centers. In the present research work, we can see that supply chain system can quickly respond to customer needs and adapt to the dynamic change of the market so as to improve the competence of enterprises in the chain. Thus, in supply chain system, it’s most important to enhance the speed with which the products are produced and distributed to the customers who order them and reduce the operating costs at the same time. However, because of the special characteristics, such as dynamic and distributed , etc., often in Agile Supply Chain System (ASCS), there are many dynamic tasks, and many urgent changes of the processes, which make the planning work and mana gement become very difficult and complex. Thus, in Agile Supply Chain System, we first need an efficient planning work, which can program the processes properly to get a primary scheme. And then the local scheduling work based on the primar y scheme will play a important role to deal with the dynamic and distributed pro blems in the business process in ASCS. So, this paper will be organized as below . At the first of this paper, we will discuss the situation in which Agile Suppl y Chain System is applied, and then we will elaborate the characteristics of Agi le Supply Chain System. With that, the shortcomings of the process managements t hat are, at present, used in Supply Chain Systems will be displayed clearly. Sec ond, we will introduce the planning methods in our research work. And then, the local scheduling will be discussed in detail, based on the primarily planned wor kflow. To realize the goal, first we build the mathematic model to describe the scheduling goal of system optimum, based on the categories of the cooperating-r elation among the operation nodes, which we defined in our research work, in Agi le Supply Chain System. And then the optimized algorithm to solve the model woul d be introduced, in succession. At the final of this paper, we will introduce some knowledge of the process mana gement and the realization of ASCS and summarize our work.

Real-time Locally Optimal Schedule for Electric Vehicle Load via Diversity-maximization NSGA-II

As distributed energy storage equipments, electric vehicles (EVs) have great potential for applications in power systems. Meanwhile, reasonable optimization of the charging time of EVs can reduce the users’ expense. Thus, the schedule of the EV load requires multi-objective optimization. A diversity-maximization non-dominated sorting genetic algorithm (DM-NSGA)-II is developed to perform multi-objective optimization by considering the power load profile, the users’charging cost, and battery degradation. Furthermore, a real-time locally optimal schedule is adopted by utilizing a flexible time scale. The case study illustrates that the proposed DM-NSGA-II can prevent being trapped in a relatively limited region so as to diversify the optimal results and provide trade-off solutions to decision makers. The simulation analysis shows that the variable time scale can continuously involve the present EVs in the real-time optimization rather than rely on the forecasting data. The schedule of the EV load is more practical without the loss of accuracy.

Load Feedback-Based Resource Scheduling and Dynamic Migration-Based Data Locality for Virtual Hadoop Clusters in OpenStack-Based Clouds

With cloud computing technology becoming more mature, it is essential to combine the big data processing tool Hadoop with the Infrastructure as a Service（Iaa S） cloud platform. In this study, we first propose a new Dynamic Hadoop Cluster on Iaa S（DHCI） architecture, which includes four key modules： monitoring,scheduling, Virtual Machine（VM） management, and VM migration modules. The load of both physical hosts and VMs is collected by the monitoring module and can be used to design resource scheduling and data locality solutions. Second, we present a simple load feedback-based resource scheduling scheme. The resource allocation can be avoided on overburdened physical hosts or the strong scalability of virtual cluster can be achieved by fluctuating the number of VMs. To improve the flexibility, we adopt the separated deployment of the computation and storage VMs in the DHCI architecture, which negatively impacts the data locality. Third, we reuse the method of VM migration and propose a dynamic migration-based data locality scheme using parallel computing entropy. We migrate the computation nodes to different host（s） or rack（s） where the corresponding storage nodes are deployed to satisfy the requirement of data locality. We evaluate our solutions in a realistic scenario based on Open Stack.Substantial experimental results demonstrate the effectiveness of our solutions that contribute to balance the workload and performance improvement, even under heavy-loaded cloud system conditions.