Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Combining reinforcement learning with mathematical programming:An approach for optimal design of heat exchanger networks

Heat integration is important for energy-saving in the process industry.It is linked to the persistently challenging task of optimal design of heat exchanger networks(HEN).Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem,it is not easy to find solutions of high quality for large-scale problems.The reinforcement learning(RL)method,which learns strategies through ongoing exploration and exploitation,reveals advantages in such area.However,due to the complexity of the HEN design problem,the RL method for HEN should be dedicated and designed.A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods.An insightful state representation of the HEN structure as well as a customized reward function is introduced.A Q-learning algorithm is applied to update the HEN structure using theε-greedy strategy.Better results are obtained from three literature cases of different scales.

Short-term displacement prediction for newly established monitoring slopes based on transfer learning

This study makes a significant progress in addressing the challenges of short-term slope displacement prediction in the Universal Landslide Monitoring Program,an unprecedented disaster mitigation program in China,where lots of newly established monitoring slopes lack sufficient historical deformation data,making it difficult to extract deformation patterns and provide effective predictions which plays a crucial role in the early warning and forecasting of landslide hazards.A slope displacement prediction method based on transfer learning is therefore proposed.Initially,the method transfers the deformation patterns learned from slopes with relatively rich deformation data by a pre-trained model based on a multi-slope integrated dataset to newly established monitoring slopes with limited or even no useful data,thus enabling rapid and efficient predictions for these slopes.Subsequently,as time goes on and monitoring data accumulates,fine-tuning of the pre-trained model for individual slopes can further improve prediction accuracy,enabling continuous optimization of prediction results.A case study indicates that,after being trained on a multi-slope integrated dataset,the TCN-Transformer model can efficiently serve as a pretrained model for displacement prediction at newly established monitoring slopes.The three-day average RMSE is significantly reduced by 34.6%compared to models trained only on individual slope data,and it also successfully predicts the majority of deformation peaks.The fine-tuned model based on accumulated data on the target newly established monitoring slope further reduced the three-day RMSE by 37.2%,demonstrating a considerable predictive accuracy.In conclusion,taking advantage of transfer learning,the proposed slope displacement prediction method effectively utilizes the available data,which enables the rapid deployment and continual refinement of displacement predictions on newly established monitoring slopes.

Statistical Analysis of Students’Learning Effectiveness in Online Courses Offered by British Partners in China-UK Joint Education Program

In the context of internationalization,China-UK Joint Education Programs are receiving increasing attention from universities.Based on the difficulties faced in China-UK Joint Education Program,this paper adopts a questionnaire survey method to study the learning effectiveness of students majoring in digital media technology in the China-UK Joint Education Program at Guangxi University of Finance and Economics,focusing on four aspects:learning materials,learning content,teacher conditions,and student learning outcomes.The research analysis in this paper not only provides strong support for the construction of China-UK Joint Education Program but also offers references for other China-UK Joint Education Programs.

Approximate Dynamic Programming for Self-Learning Control

This paper introduces a self-learning control approach based on approximate dynamic programming. Dynamic programming was introduced by Bellman in the 1950's for solving optimal control problems of nonlinear dynamical systems. Due to its high computational complexity, the applications of dynamic programming have been limited to simple and small problems. The key step in finding approximate solutions to dynamic programming is to estimate the performance index in dynamic programming. The optimal control signal can then be determined by minimizing (or maximizing) the performance index. Artificial neural networks are very efficient tools in representing the performance index in dynamic programming. This paper assumes the use of neural networks for estimating the performance index in dynamic programming and for generating optimal control signals, thus to achieve optimal control through self-learning.

Adaptive Multi-Step Evaluation Design With Stability Guarantee for Discrete-Time Optimal Learning Control

This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.

Improving Sensor-free Detection of Programming Difficulties Using Deep Learning

Programming difficulties are one of the common problems faced by software engineering students,which can lead to a rapid decline in motivation and even drop out.Probing students’programming difficulties is a crucial step in understanding their current programming situation and implementing appropriate instructional interventions.However,how to detect students’programming difficulties accurately without students’awareness remains a big challenge.Address the issues above;this paper adopts a sensor-free difficulties detecting method based on a deep neural network which employs a recurrent neural network(RNN)model and uses the sequential timing data from programming behaviour.The method can detect students’programming difficulties in real-time with 93%accuracy without interference in the programming process.In the long term,this method is the first step for establishing an automated intelligent programming environment.At the same time,it can assist teachers in noticing the difficulties that students encounter.Then,teachers can adjust their teaching plans and provide manual tutoring intervention more quickly.

An Empirical Study of the Optimum Team Size Requirement in a Collaborative Computer Programming/Learning Environment

Pair programming has been widely acclaimed the best way to go in computer programming. Recently, collaboration involving more subjects has been shown to produce better results in programming environments. However, the optimum group size needed for the collaboration has not been adequately addressed. This paper seeks to inculcate and acquaint the students involved in the study with the spirit of team work in software projects and to empirically determine the effective (optimum) team size that may be desirable in programming/learning real life environments. Two different experiments were organized and conducted. Parameters for determining the optimal team size were formulated. Volunteered participants of different genders were randomly grouped into five parallel teams of different sizes ranging from 1 to 5 in the first experiment. Each team size was replicated six times. The second experiment involved teams of same gender compositions (males or females) in different sizes. The times (efforts) for problem analysis and coding as well as compile-time errors (bugs) were recorded for each team size. The effectiveness was finally analyzed for the teams. The study shows that collaboration is highly beneficial to new learners of computer programming. They easily grasp the programming concepts when the learning is done in the company of others. The study also demonstrates that the optimum team size that may be adopted in a collaborative learning of computer programming is four.

Diagnosing Student Learning Problems in Object Oriented Programming

Students often face difficulties while taking basic programming courses due to several factors. In response, research has presented subjective assessments for diagnosing learning problems to improve the teaching of programming in higher education. In this paper, the authors propose an Object Oriented conceptual map model and organize this approach into three levels： constructing a Concept Effect Propagation Table, constructing Test Item-Concept Relationships and diagnosing Student Learning Problems with Matrix Composition. The authors＇ work is a modification of the approaches of Chert and Bai as well as Chu et al., as the authors use statistical methods, rather than fuzzy sets, for the authors＇ analysis. This paper includes a statistical summary, which has been tested on a small sample of students in King Abdulaziz University, Jeddah, Saudi Arabia, illustrating the learning problems in an Object Oriented course. The experimental results have demonstrated that this approach might aid learning and teaching in an effective way.

Call for papers Journal of Control Theory and Applications Special issue on Approximate dynamic programming and reinforcement learning

Approximate dynamic programming (ADP) is a general and effective approach for solving optimal control and estimation problems by adapting to uncertain and nonconvex environments over time.

Current Trends in Online Programming Languages Learning Tools: A Systematic Literature Review

Students face difficulties in programming languages learning (PLL) which encourages many scholars to investigate the factors behind that. Although there a number of positive and negative factors found to be effective in PLL procedure, utilising online tools in PLL were recognized as a positive recommended means. This motivates many researchers to provide solutions and proposals which result in a number of choices and options. However, categorising those efforts and showing what has been done, would provide a better and clear image for future studies. Therefore, this paper aims to conduct a systematic literature review to show what studies have been done and then categorise them based on the type of online tools and the aims of the research. The study follows Kitchenham and Charters guidelines for writing SLR (Systematic Literature Review). The search result reached 1390 publications between 2013-09/2018. After the filtration which has been done through selected criteria, 160 publications were found to be adequate to answer the review questions. The main results of this systematic review are categorizing the aims of the studies in online PLL tools, classifying the tools and finding the current trends of the online PLL tools.

Research on the Transformation of Teaching and Research Form of Professional Teachers in Blended Learning at Colleges and Universities - Taking the Java Programming Course as an Example

In view of the current situation that offline teaching is the main mode of teaching Java Programming in higher vocational schools,this paper introduces the online and offline hybrid teaching method and expounds it from the aspects of blended learning design,teaching organization,and implementation.At the same time,combined with the characteristics of blended learning,this paper proposes that under the new mode,teachers should actively change the form of teaching and research,the teaching mode,and the role of teachers,take students as the center,and build an independent and effective classroom.

基于生成式人工智能的大学生编程学习行为分析研究

生成式人工智能可以为教育提供高效且个性化的智能化服务和技术支持。作为典型的生成式人工智能语言模型,ChatGPT在编程中的应用已然获得业界的广泛关注。然而,鲜有学者从实证研究的层面探究学习者如何利用ChatGPT来进行编程学习。研究通过细粒度地采集学习者的编程行为和知识探究问题,对36位学习者的编程过程进行分析。研究结果表明:(1)学习者将ChatGPT视为有用的编程学习资源,依赖其指导学习过程,并倾向于将代码或调试错误信息拷贝至ChatGPT,进而复制其反馈信息;(2)高绩效组主要在前期使用ChatGPT辅助编程,低绩效组在整个编程过程中更频繁地使用ChatGPT进行编程;(3)学习者在使用ChatGPT时主要关注浅层和中层知识的探究,其中,高绩效组通过自主提问获得ChatGPT的反馈,中低绩效组则依赖对ChatGPT反馈内容的追问获得问题解决方案。研究针对如何利用ChatGPT辅助大学生开展编程学习提出了相关的建议,以期为提高编程学习效率提供参考。

基于ChatGPT的课程学习助手系统的设计与实现

计算机编程语言类课程在学生学习和教师教学过程中遇到诸多困难,聊天机器人程序ChatGPT的出现,为该类课程的学习带来新的尝试。通过设计一款基于ChatGPT的计算机编程语言类课程学习助手系统,帮助学生解决程序代码输入量少、程序代码输出困难的问题。从ChatGPT接入、数据集构建到关键字配置三个方面阐述系统的设计和实现,并通过实际运行测试,效果良好。

从管段走向管网:管道泄漏诊断技术研究进展

管道泄漏诊断技术在保障管道系统安全运行中起着至关重要的作用。首先,介绍了管道泄漏诊断系统的结构,并指出由单一管段向复杂管网泄漏诊断的发展趋势。进一步从基于数据驱动的传统泄漏检测方法、管道泄漏信号源定位技术和基于深度学习的复杂管网泄漏检测方法3个方面进行综述,分析了不同方法的优势、局限性和适用范围。最后指出,随着管网系统的复杂度增加,传统方法的局限性逐渐显现,基于深度学习技术的复杂管网微弱泄漏诊断、多源信号融合和管网智能化的研究将成为未来的研究趋势。

“教—学—评”一致性何以可能

“教—学—评”一致性是近年来课程与教学改革领域的一个重要议题。针对实践中时有发生的“教—学—评”不一致问题,《义务教育课程方案(2022年版)》倡导“教—学—评”一致性原则,强调中小学教师明确“为什么教”“教什么”“教到什么程度”以及“怎么教”,促进教学、学习与评价之间的一致性,使教、学、评三者协同发力,有效提高教学实效。达成“教—学—评”一致性的三个条件可以归结为:基于操作化目标设计教学、让指向目标的学习真正发生、有效评价学生达成目标的程度。只有教师的教、学生的学以及评价活动统一在同一个目标之下,“教—学—评”一致性才能得以实现。

结合数据驱动与物理模型的主动配电网双时间尺度电压协调优化控制

高比例电动汽车、分布式风电、光伏接入配电网,导致电压频繁地剧烈波动。传统调压设备与逆变器动作速度差异巨大,如何协调是难点问题。该文结合数据驱动与物理建模方法,提出一种配电网双时间尺度电压协调优化控制策略。针对短时间尺度(min级)电压波动,以静止无功补偿器、分布式电源无功功率为决策变量,以电压二次方偏差最小为目标函数,针对平衡与不平衡配电网,基于支路潮流方程,计及物理约束构建了二次规划模型。针对长时间尺度(h级)电压波动,以电压调节器匝比、可投切电容电抗器挡位、储能系统充放电功率为动作,当前时段配电网节点功率为状态,节点电压二次方偏差为代价,构建了马尔可夫决策过程。为克服连续-离散动作空间维数灾,提出了一种基于松弛-预报-校正的深度确定性策略梯度强化学习求解算法。最后,采用IEEE 33节点平衡与123节点不平衡配电网验证了所提出方法的有效性。

基于Q-learning算法的配电网储能装置控制策略研究

通过在配电网末端接入用于系统调压等辅助服务的储能装置,能有效解决可再生能源的高度间歇性和负荷需求波动导致的配变过载问题。基于强化学习的Q-learning算法,针对储能电池运行情况进行建模仿真,通过单时段优化内嵌的Q值得到各时段储能电池荷电状态的最优调度方案。实例试验分析表明,当迭代次数达到一定数量时,利用Q-learning算法能够达到理论上的最优解。最后,通过将Q-learning算法与动态规划算法生成的标准最优调度方案进行对比,证明了Q-learning算法能够与动态规划算法达成一致最优解。

传统文化类小程序界面设计要素对学习兴趣的影响研究

目的本研究旨在探讨传统文化类微信小程序界面设计要素对用户学习兴趣的影响。通过分析界面设计因素,旨在揭示界面设计要素如何促进用户对传统文化学习的兴趣。方法采用作者独立设计的小程序界面方案作为研究样本,结合主观调查问卷和眼动追踪实验,研究不同界面设计要素对用户在使用过程中学习兴趣的影响模式。结果眼动追踪数据与问卷调查结果均显示,在传统文化类微信小程序的界面设计中,适宜的字体样式与色彩搭配、引人入胜的交互设计,以及合理的功能布局对增强用户学习兴趣具有显著作用。结论本研究的发现为传统文化类微信小程序的界面设计提供了提升用户体验和学习兴趣的理论依据、实践指导。这些结论对设计者在开发类似应用程序时,如何通过界面设计激发和维持用户学习兴趣具有重要的参考价值。

英语学习App的开发与应用

移动应用程序的开发和应用不仅给大学生的学习带来了便捷性,也给教学改革带来了新的思路。本文利用安卓平台和技术自主设计开发了一款英语学习App。该应用功能简洁,具备教师英语教学和学生英语学习的各种功能。实践表明,教师根据教学需求自主开发程序不仅是可行的,而且对教学也是十分有益的。