Reinforcement Learning-Based Energy Management for Hybrid Power Systems:State-of-the-Art Survey,Review,and Perspectives

The new energy vehicle plays a crucial role in green transportation,and the energy management strategy of hybrid power systems is essential for ensuring energy-efficient driving.This paper presents a state-of-the-art survey and review of reinforcement learning-based energy management strategies for hybrid power systems.Additionally,it envisions the outlook for autonomous intelligent hybrid electric vehicles,with reinforcement learning as the foundational technology.First of all,to provide a macro view of historical development,the brief history of deep learning,reinforcement learning,and deep reinforcement learning is presented in the form of a timeline.Then,the comprehensive survey and review are conducted by collecting papers from mainstream academic databases.Enumerating most of the contributions based on three main directions—algorithm innovation,powertrain innovation,and environment innovation—provides an objective review of the research status.Finally,to advance the application of reinforcement learning in autonomous intelligent hybrid electric vehicles,future research plans positioned as“Alpha HEV”are envisioned,integrating Autopilot and energy-saving control.

Smart Energy Management System Using Machine Learning

Energy management is an inspiring domain in developing of renewable energy sources.However,the growth of decentralized energy production is revealing an increased complexity for power grid managers,inferring more quality and reliability to regulate electricity flows and less imbalance between electricity production and demand.The major objective of an energy management system is to achieve optimum energy procurement and utilization throughout the organization,minimize energy costs without affecting production,and minimize environmental effects.Modern energy management is an essential and complex subject because of the excessive consumption in residential buildings,which necessitates energy optimization and increased user comfort.To address the issue of energy management,many researchers have developed various frameworks;while the objective of each framework was to sustain a balance between user comfort and energy consumption,this problem hasn’t been fully solved because of how difficult it is to solve it.An inclusive and Intelligent Energy Management System(IEMS)aims to provide overall energy efficiency regarding increased power generation,increase flexibility,increase renewable generation systems,improve energy consumption,reduce carbon dioxide emissions,improve stability,and reduce energy costs.Machine Learning(ML)is an emerging approach that may be beneficial to predict energy efficiency in a better way with the assistance of the Internet of Energy(IoE)network.The IoE network is playing a vital role in the energy sector for collecting effective data and usage,resulting in smart resource management.In this research work,an IEMS is proposed for Smart Cities(SC)using the ML technique to better resolve the energy management problem.The proposed system minimized the energy consumption with its intelligent nature and provided better outcomes than the previous approaches in terms of 92.11% accuracy,and 7.89% miss-rate.

AI-Driven Learning Management Systems:Modern Developments, Challenges and Future Trends during theAge of ChatGPT

COVID-19 pandemic restrictions limited all social activities to curtail the spread of the virus.The foremost and most prime sector among those affected were schools,colleges,and universities.The education system of entire nations had shifted to online education during this time.Many shortcomings of Learning Management Systems(LMSs)were detected to support education in an online mode that spawned the research in Artificial Intelligence(AI)based tools that are being developed by the research community to improve the effectiveness of LMSs.This paper presents a detailed survey of the different enhancements to LMSs,which are led by key advances in the area of AI to enhance the real-time and non-real-time user experience.The AI-based enhancements proposed to the LMSs start from the Application layer and Presentation layer in the form of flipped classroom models for the efficient learning environment and appropriately designed UI/UX for efficient utilization of LMS utilities and resources,including AI-based chatbots.Session layer enhancements are also required,such as AI-based online proctoring and user authentication using Biometrics.These extend to the Transport layer to support real-time and rate adaptive encrypted video transmission for user security/privacy and satisfactory working of AI-algorithms.It also needs the support of the Networking layer for IP-based geolocation features,the Virtual Private Network(VPN)feature,and the support of Software-Defined Networks(SDN)for optimum Quality of Service(QoS).Finally,in addition to these,non-real-time user experience is enhanced by other AI-based enhancements such as Plagiarism detection algorithms and Data Analytics.

Towards Automated Assessment of Learning Management Systems in Higher Education Institutions in Zambia

Zambia like any other country in most African regions is still grappling with the dynamics of harnessing technology for the betterment of Higher Education. The onset of the Covid 19 pandemic brought a test for the preparedness of the Zambian Higher Education Institutions (HEIs) in harnessing technology for pedagogical activities. As countries worldwide switched to electronic learning during the pandemic, the same could not be said for Zambian HEIs. Zambian HEIs struggled to conduct pedagogical activities on learning management platforms. This study investigated the factors affecting the implementation and assessment of learning Management systems in Zambia’s HEIs. With its focus on assessing: 1) system features, 2) compliance with regulatory standards, 3) quality of service and 4) technology acceptance as the four key assessment areas of an LMS, this article proposed a model for assessing learning management systems in Zambian HEIs. To test the proposed model, a software tool was also developed.

Enhancing Secure Development in Globally Distributed Software Product Lines: A Machine Learning-Powered Framework for Cyber-Resilient Ecosystems

Embracing software product lines(SPLs)is pivotal in the dynamic landscape of contemporary software devel-opment.However,the flexibility and global distribution inherent in modern systems pose significant challenges to managing SPL variability,underscoring the critical importance of robust cybersecurity measures.This paper advocates for leveraging machine learning(ML)to address variability management issues and fortify the security of SPL.In the context of the broader special issue theme on innovative cybersecurity approaches,our proposed ML-based framework offers an interdisciplinary perspective,blending insights from computing,social sciences,and business.Specifically,it employs ML for demand analysis,dynamic feature extraction,and enhanced feature selection in distributed settings,contributing to cyber-resilient ecosystems.Our experiments demonstrate the framework’s superiority,emphasizing its potential to boost productivity and security in SPLs.As digital threats evolve,this research catalyzes interdisciplinary collaborations,aligning with the special issue’s goal of breaking down academic barriers to strengthen digital ecosystems against sophisticated attacks while upholding ethics,privacy,and human values.

Survey on AI and Machine Learning Techniques for Microgrid Energy Management Systems

In the era of an energy revolution,grid decentralization has emerged as a viable solution to meet the increasing global energy demand by incorporating renewables at the distributed level.Microgrids are considered a driving component for accelerating grid decentralization.To optimally utilize the available resources and address potential challenges,there is a need to have an intelligent and reliable energy management system(EMS)for the microgrid.The artificial intelligence field has the potential to address the problems in EMS and can provide resilient,efficient,reliable,and scalable solutions.This paper presents an overview of existing conventional and AI-based techniques for energy management systems in microgrids.We analyze EMS methods for centralized,decentralized,and distributed microgrids separately.Then,we summarize machine learning techniques such as ANNs,federated learning,LSTMs,RNNs,and reinforcement learning for EMS objectives such as economic dispatch,optimal power flow,and scheduling.With the incorporation of AI,microgrids can achieve greater performance efficiency and more reliability for managing a large number of energy resources.However,challenges such as data privacy,security,scalability,explainability,etc.,need to be addressed.To conclude,the authors state the possible future research directions to explore AI-based EMS's potential in real-world applications.

A New Solution to Intrusion Detection Systems Based on Improved Federated-Learning Chain

In the context of enterprise systems,intrusion detection(ID)emerges as a critical element driving the digital transformation of enterprises.With systems spanning various sectors of enterprises geographically dispersed,the necessity for seamless information exchange has surged significantly.The existing cross-domain solutions are challenged by such issues as insufficient security,high communication overhead,and a lack of effective update mechanisms,rendering them less feasible for prolonged application on resource-limited devices.This study proposes a new cross-domain collaboration scheme based on federated chains to streamline the server-side workload.Within this framework,individual nodes solely engage in training local data and subsequently amalgamate the final model employing a federated learning algorithm to uphold enterprise systems with efficiency and security.To curtail the resource utilization of blockchains and deter malicious nodes,a node administration module predicated on the workload paradigm is introduced,enabling the release of surplus resources in response to variations in a node’s contribution metric.Upon encountering an intrusion,the system triggers an alert and logs the characteristics of the breach,facilitating a comprehensive global update across all nodes for collective defense.Experimental results across multiple scenarios have verified the security and effectiveness of the proposed solution,with no loss of its recognition accuracy.

Security Monitoring and Management for the Network Services in the Orchestration of SDN-NFV Environment Using Machine Learning Techniques

Software Defined Network(SDN)and Network Function Virtualization(NFV)technology promote several benefits to network operators,including reduced maintenance costs,increased network operational performance,simplified network lifecycle,and policies management.Network vulnerabilities try to modify services provided by Network Function Virtualization MANagement and Orchestration(NFV MANO),and malicious attacks in different scenarios disrupt the NFV Orchestrator(NFVO)and Virtualized Infrastructure Manager(VIM)lifecycle management related to network services or individual Virtualized Network Function(VNF).This paper proposes an anomaly detection mechanism that monitors threats in NFV MANO and manages promptly and adaptively to implement and handle security functions in order to enhance the quality of experience for end users.An anomaly detector investigates these identified risks and provides secure network services.It enables virtual network security functions and identifies anomalies in Kubernetes(a cloud-based platform).For training and testing purpose of the proposed approach,an intrusion-containing dataset is used that hold multiple malicious activities like a Smurf,Neptune,Teardrop,Pod,Land,IPsweep,etc.,categorized as Probing(Prob),Denial of Service(DoS),User to Root(U2R),and Remote to User(R2L)attacks.An anomaly detector is anticipated with the capabilities of a Machine Learning(ML)technique,making use of supervised learning techniques like Logistic Regression(LR),Support Vector Machine(SVM),Random Forest(RF),Naïve Bayes(NB),and Extreme Gradient Boosting(XGBoost).The proposed framework has been evaluated by deploying the identified ML algorithm on a Jupyter notebook in Kubeflow to simulate Kubernetes for validation purposes.RF classifier has shown better outcomes(99.90%accuracy)than other classifiers in detecting anomalies/intrusions in the containerized environment.

Optimal Machine Learning Enabled Performance Monitoring for Learning Management Systems

Learning Management System(LMS)is an application software that is used in automation,delivery,administration,tracking,and reporting of courses and programs in educational sector.The LMS which exploits machine learning(ML)has the ability of accessing user data and exploit it for improving the learning experience.The recently developed artificial intelligence(AI)and ML models helps to accomplish effective performance monitoring for LMS.Among the different processes involved in ML based LMS,feature selection and classification processesfind beneficial.In this motivation,this study introduces Glowworm-based Feature Selection with Machine Learning Enabled Performance Monitoring(GSO-MFWELM)technique for LMS.The key objective of the proposed GSO-MFWELM technique is to effectually monitor the performance in LMS.The pro-posed GSO-MFWELM technique involves GSO-based feature selection techni-que to select the optimal features.Besides,Weighted Extreme Learning Machine(WELM)model is applied for classification process whereas the parameters involved in WELM model are optimallyfine-tuned with the help of May-fly Optimization(MFO)algorithm.The design of GSO and MFO techniques result in reduced computation complexity and improved classification performance.The presented GSO-MFWELM technique was validated for its performance against benchmark dataset and the results were inspected under several aspects.The simulation results established the supremacy of GSO-MFWELM technique over recent approaches with the maximum classification accuracy of 0.9589.

Intelligent Student Mental Health Assessment Model on Learning Management System

A learning management system(LMS)is a software or web based application,commonly utilized for planning,designing,and assessing a particular learning procedure.Generally,the LMS offers a method of creating and delivering content to the instructor,monitoring students’involvement,and validating their outcomes.Since mental health issues become common among studies in higher education globally,it is needed to properly determine it to improve mental stabi-lity.This article develops a new seven spot lady bird feature selection with opti-mal sparse autoencoder(SSLBFS-OSAE)model to assess students’mental health on LMS.The major aim of the SSLBFS-OSAE model is to determine the proper health status of the students with respect to depression,anxiety,and stress(DAS).The SSLBFS-OSAE model involves a new SSLBFS model to elect a useful set of features.In addition,OSAE model is applied for the classification of mental health conditions and the performance can be improved by the use of cuckoo search optimization(CSO)based parameter tuning process.The design of CSO algorithm for optimally tuning the SAE parameters results in enhanced classifica-tion outcomes.For examining the improved classifier results of the SSLBFS-OSAE model,a comprehensive results analysis is done and the obtained values highlighted the supremacy of the SSLBFS model over its recent methods interms of different measures.

Optimization of Interactive Videos Empowered the Experience of Learning Management System

The Learning management system(LMS)is now being used for uploading educational content in both distance and blended setups.LMS platform has two types of users:the educators who upload the content,and the students who have to access the content.The students,usually rely on text notes or books and video tutorials while their exams are conducted with formal methods.Formal assessments and examination criteria are ineffective with restricted learning space which makes the student tend only to read the educational contents and videos instead of interactive mode.The aim is to design an interactive LMS and examination video-based interface to cater the issues of educators and students.It is designed according to Human-computer interaction(HCI)principles to make the interactive User interface(UI)through User experience(UX).The interactive lectures in the form of annotated videos increase user engagement and improve the self-study context of users involved in LMS.The interface design defines how the design will interact with users and how the interface exchanges information.The findings show that interactive videos for LMS allow the users to have a more personalized learning experience by engaging in the educational content.The result shows a highly personalized learning experience due to the interactive video and quiz within the video.

Exploring Students Engagement Towards the Learning Management System (LMS) Using Learning Analytics

Learning analytics is a rapidly evolving research discipline that uses theinsights generated from data analysis to support learners as well as optimize boththe learning process and environment. This paper studied students’ engagementlevel of the Learning Management System (LMS) via a learning analytics tool,student’s approach in managing their studies and possible learning analytic methods to analyze student data. Moreover, extensive systematic literature review(SLR) was employed for the selection, sorting and exclusion of articles fromdiverse renowned sources. The findings show that most of the engagement inLMS are driven by educators. Additionally, we have discussed the factors inLMS, causes of low engagement and ways of increasing engagement factorsvia the Learning Analytics approach. Nevertheless, apart from recognizing theLearning Analytics approach as being a successful method and technique for analyzing the LMS data, this research further highlighted the possibility of mergingthe learning analytics technique with the LMS engagement in every institution asbeing a direction for future research.

Characterization of Memory Access in Deep Learning and Its Implications in Memory Management

Due to the recent trend of software intelligence in the Fourth Industrial Revolution,deep learning has become a mainstream workload for modern computer systems.Since the data size of deep learning increasingly grows,managing the limited memory capacity efficiently for deep learning workloads becomes important.In this paper,we analyze memory accesses in deep learning workloads and find out some unique characteristics differentiated from traditional workloads.First,when comparing instruction and data accesses,data access accounts for 96%–99%of total memory accesses in deep learning workloads,which is quite different from traditional workloads.Second,when comparing read and write accesses,write access dominates,accounting for 64%–80%of total memory accesses.Third,although write access makes up the majority of memory accesses,it shows a low access bias of 0.3 in the Zipf parameter.Fourth,in predicting re-access,recency is important in read access,but frequency provides more accurate information in write access.Based on these observations,we introduce a Non-Volatile Random Access Memory(NVRAM)-accelerated memory architecture for deep learning workloads,and present a new memory management policy for this architecture.By considering the memory access characteristics of deep learning workloads,the proposed policy improves memory performance by 64.3%on average compared to the CLOCK policy.

Privacy Data Management Mechanism Based on Blockchain and Federated Learning

Due to the extensive use of various intelligent terminals and the popularity of network social tools,a large amount of data in the field of medical emerged.How to manage these massive data safely and reliably has become an important challenge for the medical network community.This paper proposes a data management framework of medical network community based on Consortium Blockchain(CB)and Federated learning(FL),which realizes the data security sharing between medical institutions and research institutions.Under this framework,the data security sharing mechanism of medical network community based on smart contract and the data privacy protection mechanism based on FL and alliance chain are designed to ensure the security of data and the privacy of important data in medical network community,respectively.An intelligent contract system based on Keyed-Homomorphic Public Key(KH-PKE)Encryption scheme is designed,so that medical data can be saved in the CB in the form of ciphertext,and the automatic sharing of data is realized.Zero knowledge mechanism is used to ensure the correctness of shared data.Moreover,the zero-knowledge mechanism introduces the dynamic group signature mechanism of chosen ciphertext attack(CCA)anonymity,which makes the scheme more efficient in computing and communication cost.In the end of this paper,the performance of the scheme is analyzed fromboth asymptotic and practical aspects.Through experimental comparative analysis,the scheme proposed in this paper is more effective and feasible.

Reinforcement Learning-Based Electric Vehicles Energy Management Strategy with Battery Thermal Model

The promotion of electric vehicles(EVs)is restricted due to their short cruising range.It is desirable to design an effective energy management strategy to improve their energy efficiency.Most existing work concerning energy management strategies focused on hybrids rather than the EVs.The work focusing on the energy management strategy for EVs mainly uses the traditional optimization strategies,thereby limiting the advantages of energy economy.To this end,a novel energy management strategy that considered the impact of battery thermal effects was proposed with the help of reinforcement learning.The main idea was to first analyze the energy flow path of EVs,further formulize the energy management as an optimization problem,and finally propose an online strategy based on reinforcement learning to obtain the optimal strategy.Additionally,extensive simulation results have demonstrated that our strategy reduces energy consumption by at least 27.4%compared to the existing methods.

Deep Learning-Based Swot Analysis in Construction and Demolition Waste Management

Researchers worldwide have employed a varied array of sources to cal-culate the successful management of Construction and Demolition(C&DW).Limited research has been undertaken in the domain of Construction and Demoli-tion Waste Management(C&DWM)and consequently leaving a large gap in the availability of effective management techniques.Due to the limited time available for building removal and materials collection,preparing for building materials reuse at the end of life is frequently a challenging task.In this research work Con-volutional Neural Network-Long Short-Term Memory(CNN-LSTM)is proposed to predict the number of waste materials that are obtained from a building at the end of its useful life.As a result,an effective Waste Management(WM)plan has been established through SWOT analysis.The results of the study reveal that,given fundamental building characteristics,it is possible to predict the number of materials that would be collected with high precision from a building after demolition.The proposed deep learning models achieved an average R-squared value of 0.98 and a Mean Absolute Error of 18.1 and 20.14 better than existing methods such as random forest,CNN,and DBN(Data Bus Network).

Privacy Preserving Demand Side Management Method via Multi-Agent Reinforcement Learning

The smart grid utilizes the demand side management technology to motivate energy users towards cutting demand during peak power consumption periods, which greatly improves the operation efficiency of the power grid. However, as the number of energy users participating in the smart grid continues to increase, the demand side management strategy of individual agent is greatly affected by the dynamic strategies of other agents. In addition, the existing demand side management methods, which need to obtain users’ power consumption information,seriously threaten the users’ privacy. To address the dynamic issue in the multi-microgrid demand side management model, a novel multi-agent reinforcement learning method based on centralized training and decentralized execution paradigm is presented to mitigate the damage of training performance caused by the instability of training experience. In order to protect users’ privacy, we design a neural network with fixed parameters as the encryptor to transform the users’ energy consumption information from low-dimensional to high-dimensional and theoretically prove that the proposed encryptor-based privacy preserving method will not affect the convergence property of the reinforcement learning algorithm. We verify the effectiveness of the proposed demand side management scheme with the real-world energy consumption data of Xi’an, Shaanxi, China. Simulation results show that the proposed method can effectively improve users’ satisfaction while reducing the bill payment compared with traditional reinforcement learning(RL) methods(i.e., deep Q learning(DQN), deep deterministic policy gradient(DDPG),QMIX and multi-agent deep deterministic policy gradient(MADDPG)). The results also demonstrate that the proposed privacy protection scheme can effectively protect users’ privacy while ensuring the performance of the algorithm.

Multi-Agent Hierarchical Graph Attention Reinforcement Learning for Grid-Aware Energy Management

The increasing adoption of renewable energy has posed challenges for voltage regulation in power distribution networks.Gridaware energy management,which includes the control of smart inverters and energy management systems,is a trending way to mitigate this problem.However,existing multi-agent reinforcement learning methods for grid-aware energy management have not sufficiently considered the importance of agent cooperation and the unique characteristics of the grid,which leads to limited performance.In this study,we propose a new approach named multi-agent hierarchical graph attention reinforcement learning framework(MAHGA)to stabilize the voltage.Specifically,under the paradigm of centralized training and decentralized execution,we model the power distribution network as a novel hierarchical graph containing the agent-level topology and the bus-level topology.Then a hierarchical graph attention model is devised to capture the complex correlation between agents.Moreover,we incorporate graph contrastive learning as an auxiliary task in the reinforcement learning process to improve representation learning from graphs.Experiments on several real-world scenarios reveal that our approach achieves the best performance and can reduce the number of voltage violations remarkably.

基于Q-Learning的分簇无线传感网信任管理机制

针对无线传感器网络中存在的安全问题,提出了基于Q-Learning的分簇无线传感网信任管理机制(Q-learning based trust management mechanism for clustered wireless sensor networks,QLTMM-CWSN).该机制主要考虑通信信任、数据信任和能量信任3个方面.在网络运行过程中,基于节点的通信行为、数据分布和能量消耗,使用Q-Learning算法更新节点信任值,并选择簇内信任值最高的节点作为可信簇头节点.当簇中主簇头节点的信任值低于阈值时,可信簇头节点代替主簇头节点管理簇内成员节点,维护正常的数据传输.研究结果表明,QLTMM-CWSN机制能有效抵御通信攻击、伪造本地数据攻击、能量攻击和混合攻击.

Learning from Japan for Possible Improvement in Existing Disaster Risk Management System of Nepal

Nepal and Japan, both are multi-hazard prone countries having experience of devastating disasters. It is difficult, if not impossible, to stop natural hazard events at source. However, the impact can be reduced significantly by preventing them from turning into disasters. The impact of disasters can vary depending on the capacity to handle the situation;and the capacity depends on the level of preparedness and mitigation measures taken in advance. Japan has set example for the rest of the world when it comes to Disaster Risk Management (DRM). Recovery and reconstruction after disasters are not just to develop the area as it was earlier, but it has to be taken as an opportunity for developing better than earlier, which is called as “Build Back Better”. This concept was raised by Japanese Government in UN World conference, Sendai in 2015 [1]. Dynamic, evolutionary and proactive DRM policy and plans with innovation, and the use of science and technology to find solutions, and effective implementation of the policy and plans, coupled with the culture of safety among the citizens, and the spirit of never give up “Nana KarobiYa Oki” (Seven times fall down, Eight times get up), are the unique features that every country should learn from Japan’s DRM mechanism. This paper is an effort to buy-in the good practices from Japan to improve DRM system in Nepal. It is a product of three-month intensive research in the University of Tokyo under a PhD research that consisted of reviewing existing DRM documents and several interactive meetings with stakeholders in Japan.