Day-ahead scheduling based on reinforcement learning with hybrid action space

Driven by the improvement of the smart grid,the active distribution network(ADN)has attracted much attention due to its characteristic of active management.By making full use of electricity price signals for optimal scheduling,the total cost of the ADN can be reduced.However,the optimal dayahead scheduling problem is challenging since the future electricity price is unknown.Moreover,in ADN,some schedulable variables are continuous while some schedulable variables are discrete,which increases the difficulty of determining the optimal scheduling scheme.In this paper,the day-ahead scheduling problem of the ADN is formulated as a Markov decision process(MDP)with continuous-discrete hybrid action space.Then,an algorithm based on multi-agent hybrid reinforcement learning(HRL)is proposed to obtain the optimal scheduling scheme.The proposed algorithm adopts the structure of centralized training and decentralized execution,and different methods are applied to determine the selection policy of continuous scheduling variables and discrete scheduling variables.The simulation experiment results demonstrate the effectiveness of the algorithm.

Autonomous sortie scheduling for carrier aircraft fleet under towing mode

Safe and efficient sortie scheduling on the confined flight deck is crucial for maintaining high combat effectiveness of the aircraft carrier.The primary difficulty exactly lies in the spatiotemporal coordination,i.e.,allocation of limited supporting resources and collision-avoidance between heterogeneous dispatch entities.In this paper,the problem is investigated in the perspective of hybrid flow-shop scheduling problem by synthesizing the precedence,space and resource constraints.Specifically,eight processing procedures are abstracted,where tractors,preparing spots,catapults,and launching are virtualized as machines.By analyzing the constraints in sortie scheduling,a mixed-integer planning model is constructed.In particular,the constraint on preparing spot occupancy is improved to further enhance the sortie efficiency.The basic trajectory library for each dispatch entity is generated and a delayed strategy is integrated to address the collision-avoidance issue.To efficiently solve the formulated HFSP,which is essentially a combinatorial problem with tightly coupled constraints,a chaos-initialized genetic algorithm is developed.The solution framework is validated by the simulation environment referring to the Fort-class carrier,exhibiting higher sortie efficiency when compared to existing strategies.And animation of the simulation results is available at www.bilibili.com/video/BV14t421A7Tt/.The study presents a promising supporting technique for autonomous flight deck operation in the foreseeable future,and can be easily extended to other supporting scenarios,e.g.,ammunition delivery and aircraft maintenance.

Two-Stage Scheduling Model for Flexible Resources in Active Distribution Networks Based on Probabilistic Risk Perception

Aiming at the problems of increasing uncertainty of low-carbon generation energy in active distribution network(ADN)and the difficulty of security assessment of distribution network,this paper proposes a two-phase scheduling model for flexible resources in ADN based on probabilistic risk perception.First,a full-cycle probabilistic trend sequence is constructed based on the source-load historical data,and in the day-ahead scheduling phase,the response interval of the flexibility resources on the load and storage side is optimized based on the probabilistic trend,with the probability of the security boundary as the security constraint,and with the economy as the objective.Then in the intraday phase,the core security and economic operation boundary of theADNis screened in real time.Fromthere,it quantitatively senses the degree of threat to the core security and economic operation boundary under the current source-load prediction information,and identifies the strictly secure and low/high-risk time periods.Flexibility resources within the response interval are dynamically adjusted in real-time by focusing on high-risk periods to cope with future core risks of the distribution grid.Finally,the improved IEEE 33-node distribution system is simulated to obtain the flexibility resource scheduling scheme on the load and storage side.Thescheduling results are evaluated from the perspectives of risk probability and flexible resource utilization efficiency,and the analysis shows that the scheduling model in this paper can promote the consumption of low-carbon energy from wind and photovoltaic sourceswhile reducing the operational risk of the distribution network.

Robust coordination of interdependent electricity and natural gas systems in day-ahead scheduling for facilitating volatile renewable generations via power-to-gas technology

The increasing interdependency of electricity and natural gas systems promotes coordination of the two systems for ensuring operational security and economics.This paper proposes a robust day-ahead scheduling model for the optimal coordinated operation of integrated energy systems while considering key uncertainties of the power system and natural gas system operation cost. Energy hub,with collocated gas-fired units, power-to-gas(Pt G) facilities, and natural gas storages, is considered to store or convert one type of energy(i.e., electricity or natural gas)into the other form, which could analogously function as large-scale electrical energy storages. The column-andconstraint generation(C&CG) is adopted to solve the proposed integrated robust model, in which nonlinear natural gas network constraints are reformulated via a set of linear constraints. Numerical experiments signify the effectiveness of the proposed model for handling volatile electrical loads and renewable generations via the coordinated scheduling of electricity and natural gas systems.

Day-ahead Scheduling of Multi-carrier Energy Systems with Multi-type Energy Storages and Wind Power

The integration of large-scale wind power brings challenges to the operation of integrated energy systems(IES).In this paper,a day-ahead scheduling model for IES with wind power and multi-type energy storage is proposed in a scenario-based stochastic programming framework.The structure of the IES consists of electricity,natural gas,and heating networks which are all included in the model.Operational constraints for IES incorporating multi-type energy storage devices are also considered.The constraints of the electricity network,natural gas network and heating network are formulated,and non-linear constraints are linearized.The calculation method for the correlation of wind speed between wind farms based on historical data is proposed.Uncertainties of correlated wind power were represented by creating multiple representative scenarios with different probabilities,and this was done using the Latin hyper-cube sampling(LHS)method.The stochastic scheduling model is formulated as a mixed integer linear programming(MILP)problem with the objective function of minimizing the total expected operation cost.Numerical results on a modified PJM 5-bus electricity system with a seven-node natural gas system and a six-node heating system validate the proposed model.The results demonstrate that multi-type energy storage devices can help reduce wind power curtailments and improve the operational flexibility of IES.

Optimal Day-ahead Scheduling of Islanded Microgrid Considering Risk-based Reserve Decision

Due to the lack of support from the main grid,the intermittency of renewable energy sources(RESs)and the fluctuation of load will derive uncertainties to the operation of islanded microgrids(IMGs).It is crucial to allocate appropriate reserve capacity for the economic and reliable operation of IMGs.With the high penetration of RESs,it faces both economic and environmental challenges if we only use spinning reserve for reserve support.To solve these problems,a multi-type reserve scheme for IMGs is proposed according to different operation characteristics of generation,load,and storage.The operation risk due to reserve shortage is modeled by the conditional value-at-risk(CVaR)method.The correlation of input variables is considered for the forecasting error modeling of RES and load,and Latin hypercube sampling(LHS)is adopted to generate the random scenarios of the forecasting error,so as to avoid the dimension disaster caused by conventional large-scale scenario sampling approaches.Furthermore,an optimal day-ahead scheduling model of joint energy and reserve considering riskbased reserve decision is established to coordinate the security and economy of the operation of IMGs.Finally,the comparison of numerical results of different schemes demonstrate the rationality and effectiveness of the proposed scheme and model.

Day-ahead scheduling of large numbers of thermostatically controlled loads based on equivalent energy storage model

Due to their heat/cool storage characteristics, thermostatically controlled loads(TCLs) play an important role in demand response programmers. However, the modeling of the heat/cool storage characteristic of large numbers of TCLs is not simple. In this paper, the heat exchange power is adopted to calculate the power instead of the average power, and the relationship between the heat exchange power and energy storage is considered to develop an equivalent storage model, based on which the time-varying power constraints and the energy storage constraints are developed to establish the overall day-ahead schedulingmodel. Finally, the proposed scheduling method is verified using the simulation results of a six-bus system.

MCWOA Scheduler:Modified Chimp-Whale Optimization Algorithm for Task Scheduling in Cloud Computing

Cloud computing provides a diverse and adaptable resource pool over the internet,allowing users to tap into various resources as needed.It has been seen as a robust solution to relevant challenges.A significant delay can hamper the performance of IoT-enabled cloud platforms.However,efficient task scheduling can lower the cloud infrastructure’s energy consumption,thus maximizing the service provider’s revenue by decreasing user job processing times.The proposed Modified Chimp-Whale Optimization Algorithm called Modified Chimp-Whale Optimization Algorithm(MCWOA),combines elements of the Chimp Optimization Algorithm(COA)and the Whale Optimization Algorithm(WOA).To enhance MCWOA’s identification precision,the Sobol sequence is used in the population initialization phase,ensuring an even distribution of the population across the solution space.Moreover,the traditional MCWOA’s local search capabilities are augmented by incorporating the whale optimization algorithm’s bubble-net hunting and random search mechanisms into MCWOA’s position-updating process.This study demonstrates the effectiveness of the proposed approach using a two-story rigid frame and a simply supported beam model.Simulated outcomes reveal that the new method outperforms the original MCWOA,especially in multi-damage detection scenarios.MCWOA excels in avoiding false positives and enhancing computational speed,making it an optimal choice for structural damage detection.The efficiency of the proposed MCWOA is assessed against metrics such as energy usage,computational expense,task duration,and delay.The simulated data indicates that the new MCWOA outpaces other methods across all metrics.The study also references the Whale Optimization Algorithm(WOA),Chimp Algorithm(CA),Ant Lion Optimizer(ALO),Genetic Algorithm(GA)and Grey Wolf Optimizer(GWO).

DR-IS:Dynamic Response Incremental Scheduling in Time-Sensitive Network

Time-Sensitive Network(TSN)with deterministic transmission capability is increasingly used in many emerging fields.It mainly guarantees the Quality of Service(QoS)of applications with strict requirements on time and security.One of the core features of TSN is traffic scheduling with bounded low delay in the network.However,traffic scheduling schemes in TSN are usually synthesized offline and lack dynamism.To implement incremental scheduling of newly arrived traffic in TSN,we propose a Dynamic Response Incremental Scheduling(DR-IS)method for time-sensitive traffic and deploy it on a software-defined time-sensitive network architecture.Under the premise of meeting the traffic scheduling requirements,we adopt two modes,traffic shift and traffic exchange,to dynamically adjust the time slot injection position of the traffic in the original scheme,and determine the sending offset time of the new timesensitive traffic to minimize the global traffic transmission jitter.The evaluation results show that DRIS method can effectively control the large increase of traffic transmission jitter in incremental scheduling without affecting the transmission delay,thus realizing the dynamic incremental scheduling of time-sensitive traffic in TSN.

Energy-Saving Distributed Flexible Job Shop Scheduling Optimization with Dual Resource Constraints Based on Integrated Q-Learning Multi-Objective Grey Wolf Optimizer

The distributed flexible job shop scheduling problem(DFJSP)has attracted great attention with the growth of the global manufacturing industry.General DFJSP research only considers machine constraints and ignores worker constraints.As one critical factor of production,effective utilization of worker resources can increase productivity.Meanwhile,energy consumption is a growing concern due to the increasingly serious environmental issues.Therefore,the distributed flexible job shop scheduling problem with dual resource constraints(DFJSP-DRC)for minimizing makespan and total energy consumption is studied in this paper.To solve the problem,we present a multi-objective mathematical model for DFJSP-DRC and propose a Q-learning-based multi-objective grey wolf optimizer(Q-MOGWO).In Q-MOGWO,high-quality initial solutions are generated by a hybrid initialization strategy,and an improved active decoding strategy is designed to obtain the scheduling schemes.To further enhance the local search capability and expand the solution space,two wolf predation strategies and three critical factory neighborhood structures based on Q-learning are proposed.These strategies and structures enable Q-MOGWO to explore the solution space more efficiently and thus find better Pareto solutions.The effectiveness of Q-MOGWO in addressing DFJSP-DRC is verified through comparison with four algorithms using 45 instances.The results reveal that Q-MOGWO outperforms comparison algorithms in terms of solution quality.

A Layered Energy-Efficient Multi-Node Scheduling Mechanism for Large-Scale WSN

In recent years, target tracking has been considered one of the most important applications of wireless sensornetwork (WSN). Optimizing target tracking performance and prolonging network lifetime are two equally criticalobjectives in this scenario. The existing mechanisms still have weaknesses in balancing the two demands. Theproposed heuristic multi-node collaborative scheduling mechanism (HMNCS) comprises cluster head (CH)election, pre-selection, and task set selectionmechanisms, where the latter two kinds of selections forma two-layerselection mechanism. The CH election innovatively introduces the movement trend of the target and establishesa scoring mechanism to determine the optimal CH, which can delay the CH rotation and thus reduce energyconsumption. The pre-selection mechanism adaptively filters out suitable nodes as the candidate task set to applyfor tracking tasks, which can reduce the application consumption and the overhead of the following task setselection. Finally, the task node selection is mathematically transformed into an optimization problem and thegenetic algorithm is adopted to form a final task set in the task set selection mechanism. Simulation results showthat HMNCS outperforms other compared mechanisms in the tracking accuracy and the network lifetime.

A Novel Scheduling Framework for Multi-Programming Quantum Computing in Cloud Environment

As cloud quantum computing gains broader acceptance,a growing quantity of researchers are directing their focus towards this domain.Nevertheless,the rapid surge in demand for cloud-based quantum computing resources has led to a scarcity,which in turn hampers users from achieving optimal satisfaction.Therefore,cloud quantum computing service providers require a unified analysis and scheduling framework for their quantumresources and user jobs to meet the ever-growing usage demands.This paper introduces a new multi-programming scheduling framework for quantum computing in a cloud environment.The framework addresses the issue of limited quantum computing resources in cloud environments and ensures a satisfactory user experience.It introduces three innovative designs:1)Our framework automatically allocates tasks to different quantum backends while ensuring fairness among users by considering both the cloud-based quantum resources and the user-submitted tasks.2)Multi-programming mechanism is employed across different quantum backends to enhance the overall throughput of the quantum cloud.In comparison to conventional task schedulers,our proposed framework achieves a throughput improvement of more than two-fold in the quantum cloud.3)The framework can balance fidelity and user waiting time by adaptively adjusting scheduling parameters.

Dynamic Offloading and Scheduling Strategy for Telematics Tasks Based on Latency Minimization

In current research on task offloading and resource scheduling in vehicular networks,vehicles are commonly assumed to maintain constant speed or relatively stationary states,and the impact of speed variations on task offloading is often overlooked.It is frequently assumed that vehicles can be accurately modeled during actual motion processes.However,in vehicular dynamic environments,both the tasks generated by the vehicles and the vehicles’surroundings are constantly changing,making it difficult to achieve real-time modeling for actual dynamic vehicular network scenarios.Taking into account the actual dynamic vehicular scenarios,this paper considers the real-time non-uniform movement of vehicles and proposes a vehicular task dynamic offloading and scheduling algorithm for single-task multi-vehicle vehicular network scenarios,attempting to solve the dynamic decision-making problem in task offloading process.The optimization objective is to minimize the average task completion time,which is formulated as a multi-constrained non-linear programming problem.Due to the mobility of vehicles,a constraint model is applied in the decision-making process to dynamically determine whether the communication range is sufficient for task offloading and transmission.Finally,the proposed vehicular task dynamic offloading and scheduling algorithm based on muti-agent deep deterministic policy gradient(MADDPG)is applied to solve the optimal solution of the optimization problem.Simulation results show that the algorithm proposed in this paper is able to achieve lower latency task computation offloading.Meanwhile,the average task completion time of the proposed algorithm in this paper can be improved by 7.6%compared to the performance of the MADDPG scheme and 51.1%compared to the performance of deep deterministic policy gradient(DDPG).

A Novel Predictive Model for Edge Computing Resource Scheduling Based on Deep Neural Network

Currently,applications accessing remote computing resources through cloud data centers is the main mode of operation,but this mode of operation greatly increases communication latency and reduces overall quality of service(QoS)and quality of experience(QoE).Edge computing technology extends cloud service functionality to the edge of the mobile network,closer to the task execution end,and can effectivelymitigate the communication latency problem.However,the massive and heterogeneous nature of servers in edge computing systems brings new challenges to task scheduling and resource management,and the booming development of artificial neural networks provides us withmore powerfulmethods to alleviate this limitation.Therefore,in this paper,we proposed a time series forecasting model incorporating Conv1D,LSTM and GRU for edge computing device resource scheduling,trained and tested the forecasting model using a small self-built dataset,and achieved competitive experimental results.

Performance Prediction Based Workload Scheduling in Co-Located Cluster

Cloud service providers generally co-locate online services and batch jobs onto the same computer cluster,where the resources can be pooled in order to maximize data center resource utilization.Due to resource competition between batch jobs and online services,co-location frequently impairs the performance of online services.This study presents a quality of service(QoS)prediction-based schedulingmodel(QPSM)for co-locatedworkloads.The performance prediction of QPSM consists of two parts:the prediction of an online service’s QoS anomaly based on XGBoost and the prediction of the completion time of an offline batch job based on randomforest.On-line service QoS anomaly prediction is used to evaluate the influence of batch jobmix on on-line service performance,and batch job completion time prediction is utilized to reduce the total waiting time of batch jobs.When the same number of batch jobs are scheduled in experiments using typical test sets such as CloudSuite,the scheduling time required by QPSM is reduced by about 6 h on average compared with the first-come,first-served strategy and by about 11 h compared with the random scheduling strategy.Compared with the non-co-located situation,QPSM can improve CPU resource utilization by 12.15% and memory resource utilization by 5.7% on average.Experiments show that the QPSM scheduling strategy proposed in this study can effectively guarantee the quality of online services and further improve cluster resource utilization.

Q-Learning-Assisted Meta-Heuristics for Scheduling Distributed Hybrid Flow Shop Problems

The flow shop scheduling problem is important for the manufacturing industry.Effective flow shop scheduling can bring great benefits to the industry.However,there are few types of research on Distributed Hybrid Flow Shop Problems(DHFSP)by learning assisted meta-heuristics.This work addresses a DHFSP with minimizing the maximum completion time(Makespan).First,a mathematical model is developed for the concerned DHFSP.Second,four Q-learning-assisted meta-heuristics,e.g.,genetic algorithm(GA),artificial bee colony algorithm(ABC),particle swarm optimization(PSO),and differential evolution(DE),are proposed.According to the nature of DHFSP,six local search operations are designed for finding high-quality solutions in local space.Instead of randomselection,Q-learning assists meta-heuristics in choosing the appropriate local search operations during iterations.Finally,based on 60 cases,comprehensive numerical experiments are conducted to assess the effectiveness of the proposed algorithms.The experimental results and discussions prove that using Q-learning to select appropriate local search operations is more effective than the random strategy.To verify the competitiveness of the Q-learning assistedmeta-heuristics,they are compared with the improved iterated greedy algorithm(IIG),which is also for solving DHFSP.The Friedman test is executed on the results by five algorithms.It is concluded that the performance of four Q-learning-assisted meta-heuristics are better than IIG,and the Q-learning-assisted PSO shows the best competitiveness.

Joint computation offloading and parallel scheduling to maximize delay-guarantee in cooperative MEC systems

The growing development of the Internet of Things(IoT)is accelerating the emergence and growth of new IoT services and applications,which will result in massive amounts of data being generated,transmitted and pro-cessed in wireless communication networks.Mobile Edge Computing(MEC)is a desired paradigm to timely process the data from IoT for value maximization.In MEC,a number of computing-capable devices are deployed at the network edge near data sources to support edge computing,such that the long network transmission delay in cloud computing paradigm could be avoided.Since an edge device might not always have sufficient resources to process the massive amount of data,computation offloading is significantly important considering the coop-eration among edge devices.However,the dynamic traffic characteristics and heterogeneous computing capa-bilities of edge devices challenge the offloading.In addition,different scheduling schemes might provide different computation delays to the offloaded tasks.Thus,offloading in mobile nodes and scheduling in the MEC server are coupled to determine service delay.This paper seeks to guarantee low delay for computation intensive applica-tions by jointly optimizing the offloading and scheduling in such an MEC system.We propose a Delay-Greedy Computation Offloading(DGCO)algorithm to make offloading decisions for new tasks in distributed computing-enabled mobile devices.A Reinforcement Learning-based Parallel Scheduling(RLPS)algorithm is further designed to schedule offloaded tasks in the multi-core MEC server.With an offloading delay broadcast mechanism,the DGCO and RLPS cooperate to achieve the goal of delay-guarantee-ratio maximization.Finally,the simulation results show that our proposal can bound the end-to-end delay of various tasks.Even under slightly heavy task load,the delay-guarantee-ratio given by DGCO-RLPS can still approximate 95%,while that given by benchmarked algorithms is reduced to intolerable value.The simulation results are demonstrated the effective-ness of DGCO-RLPS for delay guarantee in MEC.

Improved STNModels and Heuristic Rules for Cooperative Scheduling in Automated Container Terminals

Improving the cooperative scheduling efficiency of equipment is the key for automated container terminals to copewith the development trend of large-scale ships. In order to improve the solution efficiency of the existing spacetimenetwork (STN) model for the cooperative scheduling problem of yard cranes (YCs) and automated guidedvehicles (AGVs) and extend its application scenarios, two improved STN models are proposed. The flow balanceconstraints in the original model are decomposed, and the trajectory constraints of YCs and AGVs are added toacquire the model STN_A. The coupling constraint in STN_A is updated, and buffer constraints are added toSTN_A so that themodel STN_B is built.As the size of the problem increases, the solution speed of CPLEX becomesthe bottleneck. So a heuristic method containing three groups of heuristic rules is designed to obtain a near-optimalsolution quickly. Experimental results showthat the computation time of STN_A is shortened by 49.47% on averageand the gap is reduced by 1.69% on average compared with the original model. The gap between the solution ofthe heuristic rules and the solution of CPLEX is less than 3.50%, and the solution time of the heuristic rules is onaverage 99.85% less than the solution time of CPLEX. Compared with STN_A, the computation time for solvingSTN_B increases by 58.93% on average.

An Elite-Class Teaching-Learning-Based Optimization for Reentrant Hybrid Flow Shop Scheduling with Bottleneck Stage

Bottleneck stage and reentrance often exist in real-life manufacturing processes;however,the previous research rarely addresses these two processing conditions in a scheduling problem.In this study,a reentrant hybrid flow shop scheduling problem(RHFSP)with a bottleneck stage is considered,and an elite-class teaching-learning-based optimization(ETLBO)algorithm is proposed to minimize maximum completion time.To produce high-quality solutions,teachers are divided into formal ones and substitute ones,and multiple classes are formed.The teacher phase is composed of teacher competition and teacher teaching.The learner phase is replaced with a reinforcement search of the elite class.Adaptive adjustment on teachers and classes is established based on class quality,which is determined by the number of elite solutions in class.Numerous experimental results demonstrate the effectiveness of new strategies,and ETLBO has a significant advantage in solving the considered RHFSP.

Data-driven Wasserstein distributionally robust chance-constrained optimization for crude oil scheduling under uncertainty

Crude oil scheduling optimization is an effective method to enhance the economic benefits of oil refining.But uncertainties,including uncertain demands of crude distillation units(CDUs),might make the production plans made by the traditional deterministic optimization models infeasible.A data-driven Wasserstein distributionally robust chance-constrained(WDRCC)optimization approach is proposed in this paper to deal with demand uncertainty in crude oil scheduling.First,a new deterministic crude oil scheduling optimization model is developed as the basis of this approach.The Wasserstein distance is then used to build ambiguity sets from historical data to describe the possible realizations of probability distributions of uncertain demands.A cross-validation method is advanced to choose suitable radii for these ambiguity sets.The deterministic model is reformulated as a WDRCC optimization model for crude oil scheduling to guarantee the demand constraints hold with a desired high probability even in the worst situation in ambiguity sets.The proposed WDRCC model is transferred into an equivalent conditional value-at-risk representation and further derived as a mixed-integer nonlinear programming counterpart.Industrial case studies from a real-world refinery are conducted to show the effectiveness of the proposed method.Out-of-sample tests demonstrate that the solution of the WDRCC model is more robust than those of the deterministic model and the chance-constrained model.