Optimizing Big Data Retrieval and Job Scheduling Using Deep Learning Approaches

Big data analytics in business intelligence do not provide effective data retrieval methods and job scheduling that will cause execution inefficiency and low system throughput.This paper aims to enhance the capability of data retrieval and job scheduling to speed up the operation of big data analytics to overcome inefficiency and low throughput problems.First,integrating stacked sparse autoencoder and Elasticsearch indexing explored fast data searching and distributed indexing,which reduces the search scope of the database and dramatically speeds up data searching.Next,exploiting a deep neural network to predict the approximate execution time of a job gives prioritized job scheduling based on the shortest job first,which reduces the average waiting time of job execution.As a result,the proposed data retrieval approach outperforms the previous method using a deep autoencoder and Solr indexing,significantly improving the speed of data retrieval up to 53%and increasing system throughput by 53%.On the other hand,the proposed job scheduling algorithmdefeats both first-in-first-out andmemory-sensitive heterogeneous early finish time scheduling algorithms,effectively shortening the average waiting time up to 5%and average weighted turnaround time by 19%,respectively.

Modeling and performance evaluation of QoS-aware job scheduling of computational grids

To achieve high quality of service （QoS） on computational grids, the QoS-aware job scheduling is investigated for a hierarchical decentralized grid architecture that consists of multilevel schedulers. An integrated QoS-aware job dispatching policy is proposed, which correlates priorities of incoming jobs used for job selecting at the local scheduler of the grid node with the job dispatching policies at the global scheduler for computational grids. The stochastic high-level Petri net （SHLPN） model of a two-level hierarchy computational grid architecture is presented, and a model refinement is made to reduce the complexity of the model solution. A performance analysis technique based on the SHLPN is proposed to investigate the QoS-aware job scheduling policy. Numerical results show that the QoS-aware job dispatching policy outperforms the QoS-unaware job dispatching policy in balancing the high-priority jobs, and thus enables priority-based QoS.

Resource Intensity Aware Job Scheduling in A Distributed Cloud

A Dominant Resource Fairness （DRF） based scheme for job scheduling in distributed cloud computing systems which was modeled as multi-job scheduling and multi-resource allocation coupling problem is proposed, where the resource pool is constructed from a large number of distributed heterogeneous servers, representing different points in the configuration space of resources such as processing, memory, storage and bandwidth. By introducing dominant resource share of jobs and virtual machines, the multi-job scheduling and multi-resource allocation joint mechanism significantly improves the cloud system＇s resource utilization, yet with a substantial reduction of job completion times. We show through experiments and case studies the superior performance of the algorithms in practice.

A Dynamic Job Scheduling Algorithm for Parallel System

One of the fundamental problems in parallel and distributed systems is deciding how to allocate jobs to processors. The goals of job scheduling in a parallel environment are to minimize the parallel execution time of a job and try to balance the user’s desire with the system’s desire. The users always want their jobs be completed as quickly as possible, while the system wants to service as many jobs as possible. In this paper, a dynamic job scheduling algorithm was introduced. This algorithm tries to utilize the information of a practical system to allocate the jobs more evenly. The communication time between the processor and scheduler is overlapped with the computation time of the processor. So the communication overhead can be little. The principle of scheduling the job is based on the desirability of each processor. The scheduler would not allocate a new job to a processor that is already fully utilized. The execution efficiency of the system will be increased. This algorithm also can be reused in other complex algorithms.

A Heuristic for the Job Scheduling Problem with a Common Due Window on Parallel and Non-Identical Machines

In this paper, we give a mathematical model for earliness-tardiness job scheduling problem with a common due window on parallel and non-identical machines. Because the job scheduling problem discussed in the paper contains a problem of minimizing make-span, which is NP-complete on parallel and uniform machines, a heuristic algorithm is presented to find an approximate solution for the scheduling problem after proving an important theorem. Two numerical examples illustrate that the heuristic algorithm is very useful and effective in obtaining the near-optimal solution.

Energy-Saving Distributed Flexible Job Shop Scheduling Optimization with Dual Resource Constraints Based on Integrated Q-Learning Multi-Objective Grey Wolf Optimizer

The distributed flexible job shop scheduling problem(DFJSP)has attracted great attention with the growth of the global manufacturing industry.General DFJSP research only considers machine constraints and ignores worker constraints.As one critical factor of production,effective utilization of worker resources can increase productivity.Meanwhile,energy consumption is a growing concern due to the increasingly serious environmental issues.Therefore,the distributed flexible job shop scheduling problem with dual resource constraints(DFJSP-DRC)for minimizing makespan and total energy consumption is studied in this paper.To solve the problem,we present a multi-objective mathematical model for DFJSP-DRC and propose a Q-learning-based multi-objective grey wolf optimizer(Q-MOGWO).In Q-MOGWO,high-quality initial solutions are generated by a hybrid initialization strategy,and an improved active decoding strategy is designed to obtain the scheduling schemes.To further enhance the local search capability and expand the solution space,two wolf predation strategies and three critical factory neighborhood structures based on Q-learning are proposed.These strategies and structures enable Q-MOGWO to explore the solution space more efficiently and thus find better Pareto solutions.The effectiveness of Q-MOGWO in addressing DFJSP-DRC is verified through comparison with four algorithms using 45 instances.The results reveal that Q-MOGWO outperforms comparison algorithms in terms of solution quality.

Strengthened Dominance Relation NSGA-Ⅲ Algorithm Based on Differential Evolution to Solve Job Shop Scheduling Problem

The job shop scheduling problem is a classical combinatorial optimization challenge frequently encountered in manufacturing systems.It involves determining the optimal execution sequences for a set of jobs on various machines to maximize production efficiency and meet multiple objectives.The Non-dominated Sorting Genetic Algorithm Ⅲ(NSGA-Ⅲ)is an effective approach for solving the multi-objective job shop scheduling problem.Nevertheless,it has some limitations in solving scheduling problems,including inadequate global search capability,susceptibility to premature convergence,and challenges in balancing convergence and diversity.To enhance its performance,this paper introduces a strengthened dominance relation NSGA-Ⅲ algorithm based on differential evolution(NSGA-Ⅲ-SD).By incorporating constrained differential evolution and simulated binary crossover genetic operators,this algorithm effectively improves NSGA-Ⅲ’s global search capability while mitigating pre-mature convergence issues.Furthermore,it introduces a reinforced dominance relation to address the trade-off between convergence and diversity in NSGA-Ⅲ.Additionally,effective encoding and decoding methods for discrete job shop scheduling are proposed,which can improve the overall performance of the algorithm without complex computation.To validate the algorithm’s effectiveness,NSGA-Ⅲ-SD is extensively compared with other advanced multi-objective optimization algorithms using 20 job shop scheduling test instances.The experimental results demonstrate that NSGA-Ⅲ-SD achieves better solution quality and diversity,proving its effectiveness in solving the multi-objective job shop scheduling problem.

An Improved Harris Hawk Optimization Algorithm for Flexible Job Shop Scheduling Problem

Flexible job shop scheduling problem(FJSP)is the core decision-making problem of intelligent manufacturing production management.The Harris hawk optimization(HHO)algorithm,as a typical metaheuristic algorithm,has been widely employed to solve scheduling problems.However,HHO suffers from premature convergence when solving NP-hard problems.Therefore,this paper proposes an improved HHO algorithm(GNHHO)to solve the FJSP.GNHHO introduces an elitism strategy,a chaotic mechanism,a nonlinear escaping energy update strategy,and a Gaussian random walk strategy to prevent premature convergence.A flexible job shop scheduling model is constructed,and the static and dynamic FJSP is investigated to minimize the makespan.This paper chooses a two-segment encoding mode based on the job and the machine of the FJSP.To verify the effectiveness of GNHHO,this study tests it in 23 benchmark functions,10 standard job shop scheduling problems(JSPs),and 5 standard FJSPs.Besides,this study collects data from an agricultural company and uses the GNHHO algorithm to optimize the company’s FJSP.The optimized scheduling scheme demonstrates significant improvements in makespan,with an advancement of 28.16%for static scheduling and 35.63%for dynamic scheduling.Moreover,it achieves an average increase of 21.50%in the on-time order delivery rate.The results demonstrate that the performance of the GNHHO algorithm in solving FJSP is superior to some existing algorithms.

Deep Reinforcement Learning Solves Job-shop Scheduling Problems

To solve the sparse reward problem of job-shop scheduling by deep reinforcement learning,a deep reinforcement learning framework considering sparse reward problem is proposed.The job shop scheduling problem is transformed into Markov decision process,and six state features are designed to improve the state feature representation by using two-way scheduling method,including four state features that distinguish the optimal action and two state features that are related to the learning goal.An extended variant of graph isomorphic network GIN++is used to encode disjunction graphs to improve the performance and generalization ability of the model.Through iterative greedy algorithm,random strategy is generated as the initial strategy,and the action with the maximum information gain is selected to expand it to optimize the exploration ability of Actor-Critic algorithm.Through validation of the trained policy model on multiple public test data sets and comparison with other advanced DRL methods and scheduling rules,the proposed method reduces the minimum average gap by 3.49%,5.31%and 4.16%,respectively,compared with the priority rule-based method,and 5.34%compared with the learning-based method.11.97%and 5.02%,effectively improving the accuracy of DRL to solve the approximate solution of JSSP minimum completion time.

A Novel Collaborative Evolutionary Algorithm with Two-Population for Multi-Objective Flexible Job Shop Scheduling

Job shop scheduling(JS)is an important technology for modern manufacturing.Flexible job shop scheduling(FJS)is critical in JS,and it has been widely employed in many industries,including aerospace and energy.FJS enables any machine from a certain set to handle an operation,and this is an NP-hard problem.Furthermore,due to the requirements in real-world cases,multi-objective FJS is increasingly widespread,thus increasing the challenge of solving the FJS problems.As a result,it is necessary to develop a novel method to address this challenge.To achieve this goal,a novel collaborative evolutionary algorithmwith two-population based on Pareto optimality is proposed for FJS,which improves the solutions of FJS by interacting in each generation.In addition,several experimental results have demonstrated that the proposed method is promising and effective for multi-objective FJS,which has discovered some new Pareto solutions in the well-known benchmark problems,and some solutions can dominate the solutions of some other methods.

An Effective Neighborhood Solution Clipping Method for Large-Scale Job Shop Scheduling Problem

The job shop scheduling problem(JSSP)is a classical combinatorial optimization problem that exists widely in diverse scenarios of manufacturing systems.It is a well-known NP-hard problem,when the number of jobs increases,the difficulty of solving the problem exponentially increases.Therefore,a major challenge is to increase the solving efficiency of current algorithms.Modifying the neighborhood structure of the solutions can effectively improve the local search ability and efficiency.In this paper,a genetic Tabu search algorithm with neighborhood clipping(GTS_NC)is proposed for solving JSSP.A neighborhood solution clipping method is developed and embedded into Tabu search to improve the efficiency of the local search by clipping the search actions of unimproved neighborhood solutions.Moreover,a feasible neighborhood solution determination method is put forward,which can accurately distinguish feasible neighborhood solutions from infeasible ones.Both of the methods are based on the domain knowledge of JSSP.The proposed algorithmis compared with several competitive algorithms on benchmark instances.The experimental results show that the proposed algorithm can achieve superior results compared to other competitive algorithms.According to the numerical results of the experiments,it is verified that the neighborhood solution clippingmethod can accurately identify the unimproved solutions and reduces the computational time by at least 28%.

Applying Job Shop Scheduling to SMEs Manufacturing Platform to Revitalize B2B Relationship

A small and medium enterprises(SMEs)manufacturing platform aims to perform as a significant revenue to SMEs and vendors by providing scheduling and monitoring capabilities.The optimal job shop scheduling is generated by utilizing the scheduling system of the platform,and a minimum production time,i.e.,makespan decides whether the scheduling is optimal or not.This scheduling result allows manufacturers to achieve high productivity,energy savings,and customer satisfaction.Manufacturing in Industry 4.0 requires dynamic,uncertain,complex production environments,and customer-centered services.This paper proposes a novel method for solving the difficulties of the SMEs manufacturing by applying and implementing the job shop scheduling system on a SMEs manufacturing platform.The primary purpose of the SMEs manufacturing platform is to improve the B2B relationship between manufacturing companies and vendors.The platform also serves qualified and satisfactory production opportunities for buyers and producers by meeting two key factors:early delivery date and fulfillment of processing as many orders as possible.The genetic algorithm(GA)-based scheduling method results indicated that the proposed platform enables SME manufacturers to obtain optimized schedules by solving the job shop scheduling problem(JSSP)by comparing with the real-world data from a textile weaving factory in South Korea.The proposed platform will provide producers with an optimal production schedule,introduce new producers to buyers,and eventually foster relationships and mutual economic interests.

SOLVING FLEXIBLE JOB SHOP SCHEDULING PROBLEM BY GENETIC ALGORITHM

The job shop scheduli ng problem has been studied for decades and known as an NP-hard problem. The fl exible job shop scheduling problem is a generalization of the classical job sche duling problem that allows an operation to be processed on one machine out of a set of machines. The problem is to assign each operation to a machine and find a sequence for the operations on the machine in order that the maximal completion time of all operations is minimized. A genetic algorithm is used to solve the f lexible job shop scheduling problem. A novel gene coding method aiming at job sh op problem is introduced which is intuitive and does not need repairing process to validate the gene. Computer simulations are carried out and the results show the effectiveness of the proposed algorithm.

Improved Heuristic Job Scheduling Method to Enhance Throughput for Big Data Analytics

Data-parallel computing platforms,such as Hadoop and Spark,are deployed in computing clusters for big data analytics.There is a general tendency that multiple users share the same computing cluster.The schedule of multiple jobs becomes a serious challenge.Over a long period in the past,the Shortest-Job-First(SJF)method has been considered as the optimal solution to minimize the average job completion time.However,the SJF method leads to a low system throughput in the case where a small number of short jobs consume a large amount of resources.This factor prolongs the average job completion time.We propose an improved heuristic job scheduling method,called the Densest-Job-Set-First(DJSF)method.The DJSF method schedules jobs by maximizing the number of completed jobs per unit time,aiming to decrease the average Job Completion Time(JCT)and improve the system throughput.We perform extensive simulations based on Google cluster data.Compared with the SJF method,the DJSF method decreases the average JCT by 23.19% and enhances the system throughput by 42.19%.Compared with Tetris,the job packing method improves the job completion efficiency by 55.4%,so that the computing platforms complete more jobs in a short time span.

On/Off-Line Prediction Applied to Job Scheduling on Non-Dedicated NOWs

This paper proposes a prediction engine designed for non-dedicated clusters, which is able to estimate the turnaround time for parallel applications, even in the presence of serial workload of the workstation owner. The prediction engine can be configured to work with three different estimation kernels： a Historical kernel, a Simulation kernel based on analytical models and an integration of both, named Hybrid kernel. These estimation proposals were integrated into a scheduling system, named CISNE, which can be executed in an on-line or off-line mode. The accuracy of the proposed estimation methods was evaluated in relation to different job scheduling policies in a real and a simulated cluster environment. In both environments, we observed that the Hybrid system gives the best results because it combines the ability of a simulation engine to capture the dynamism of a non-dedicated environment together with the accuracy of the historical methods to estimate the application runtime considering the state of the resources.

Infeasibility test algorithm and fast repair algorithm of job shop scheduling problem

To diagnose the feasibility of the solution of a job-shop scheduling problem（JSSP）,a test algorithm based on diagraph and heuristic search is developed and verified through a case study.Meanwhile,a new repair algorithm for modifying an infeasible solution of the JSSP to become a feasible solution is proposed for the general JSSP.The computational complexity of the test algorithm and the repair algorithm is both O（n） under the worst-case scenario,and O（2J＋M） for the repair algorithm under the best-case scenario.The repair algorithm is not limited to specific optimization methods,such as local tabu search,genetic algorithms and shifting bottleneck procedures for job shop scheduling,but applicable to generic infeasible solutions for the JSSP to achieve feasibility.

Fuzzy Logic-Based Secure and Fault Tolerant Job Scheduling in Grid

The uncertainties of grid sites security are main hurdle to make the job scheduling secure, reliable and fault-tolerant. Most existing scheduling algorithms use fixed-number job replications to provide fault tolerant ability and high scheduling success rate, which consume excessive resources or can not provide sufficient fault tolerant functions when grid security conditions change. In this paper a fuzzy-logic-based self-adaptive replication scheduling (FSARS) algorithm is proposed to handle the fuzziness or uncertainties of job replication number which is highly related to trust factors behind grid sites and user jobs. Remote sensing-based soil moisture extraction (RSBSME) workload experiments in real grid environment are performed to evaluate the proposed approach and the results show that high scheduling success rate of up to 95% and less grid resource utilization can be achieved through FSARS. Extensive experiments show that FSARS scales well when user jobs and grid sites increase.

QoS-aware simulation job scheduling algorithm in virtualized cloud environment

Cloud computing is attracting an increasing number of simulation applications running in the virtualized cloud data center.These applications are submitted to the cloud in the form of simulation jobs.Meanwhile,the management and scheduling of simulation jobs are playing an essential role to offer efficient and high productivity computational service.In this paper,we design a management and scheduling service framework for simulation jobs in two-tier virtualization-based private cloud data center,named simulation execution as a service(SimEaaS).It aims at releasing users from complex simulation running settings,while guaranteeing the QoS requirements adaptively.Furthermore,a novel job scheduling algorithm named adaptive deadline-aware job size adjustment(ADaSA)algorithm is designed to realize high job responsiveness under QoS requirement for SimEaaS.ADaSA tries to make full use of the idle fragmentation resources by tuning the number of requested processes of submitted jobs in the queue adaptively,while guaranteeing that jobs’deadline requirements are not violated.Extensive experiments with trace-driven simulation are conducted to evaluate the performance of our ADaSA.The results show that ADaSA outperforms both cloud-based job scheduling algorithm KCEASY and traditional EASY in terms of response time(up to 90%)and bounded slow down(up to 95%),while obtains approximately equivalent deadline-missed rate.ADaSA also outperforms two representative moldable scheduling algorithms in terms of deadline-missed rate(up to 60%).

Modified bottleneck-based heuristic for large-scale job-shop scheduling problems with a single bottleneck

A modified bottleneck-based （MB） heuristic for large-scale job-shop scheduling problems with a welldefined bottleneck is suggested, which is simpler but more tailored than the shifting bottleneck （SB） procedure. In this algorithm, the bottleneck is first scheduled optimally while the non-bottleneck machines are subordinated around the solutions of the bottleneck schedule by some effective dispatching rules. Computational results indicate that the MB heuristic can achieve a better tradeoff between solution quality and computational time compared to SB procedure for medium-size problems. Furthermore, it can obtain a good solution in a short time for large-scale jobshop scheduling problems.

Job shop scheduling problem with alternative machines using genetic algorithms

The classical job shop scheduling problem(JSP) is the most popular machine scheduling model in practice and is known as NP-hard.The formulation of the JSP is based on the assumption that for each part type or job there is only one process plan that prescribes the sequence of operations and the machine on which each operation has to be performed.However,JSP with alternative machines for various operations is an extension of the classical JSP,which allows an operation to be processed by any machine from a given set of machines.Since this problem requires an additional decision of machine allocation during scheduling,it is much more complex than JSP.We present a domain independent genetic algorithm(GA) approach for the job shop scheduling problem with alternative machines.The GA is implemented in a spreadsheet environment.The performance of the proposed GA is analyzed by comparing with various problem instances taken from the literatures.The result shows that the proposed GA is competitive with the existing approaches.A simplified approach that would be beneficial to both practitioners and researchers is presented for solving scheduling problems with alternative machines.