Recent Progress in Reinforcement Learning and Adaptive Dynamic Programming for Advanced Control Applications

Reinforcement learning(RL) has roots in dynamic programming and it is called adaptive/approximate dynamic programming(ADP) within the control community. This paper reviews recent developments in ADP along with RL and its applications to various advanced control fields. First, the background of the development of ADP is described, emphasizing the significance of regulation and tracking control problems. Some effective offline and online algorithms for ADP/adaptive critic control are displayed, where the main results towards discrete-time systems and continuous-time systems are surveyed, respectively.Then, the research progress on adaptive critic control based on the event-triggered framework and under uncertain environment is discussed, respectively, where event-based design, robust stabilization, and game design are reviewed. Moreover, the extensions of ADP for addressing control problems under complex environment attract enormous attention. The ADP architecture is revisited under the perspective of data-driven and RL frameworks,showing how they promote ADP formulation significantly.Finally, several typical control applications with respect to RL and ADP are summarized, particularly in the fields of wastewater treatment processes and power systems, followed by some general prospects for future research. Overall, the comprehensive survey on ADP and RL for advanced control applications has d emonstrated its remarkable potential within the artificial intelligence era. In addition, it also plays a vital role in promoting environmental protection and industrial intelligence.

Combining reinforcement learning with mathematical programming:An approach for optimal design of heat exchanger networks

Heat integration is important for energy-saving in the process industry.It is linked to the persistently challenging task of optimal design of heat exchanger networks(HEN).Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem,it is not easy to find solutions of high quality for large-scale problems.The reinforcement learning(RL)method,which learns strategies through ongoing exploration and exploitation,reveals advantages in such area.However,due to the complexity of the HEN design problem,the RL method for HEN should be dedicated and designed.A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods.An insightful state representation of the HEN structure as well as a customized reward function is introduced.A Q-learning algorithm is applied to update the HEN structure using theε-greedy strategy.Better results are obtained from three literature cases of different scales.

Statistical Analysis of Students’Learning Effectiveness in Online Courses Offered by British Partners in China-UK Joint Education Program

In the context of internationalization,China-UK Joint Education Programs are receiving increasing attention from universities.Based on the difficulties faced in China-UK Joint Education Program,this paper adopts a questionnaire survey method to study the learning effectiveness of students majoring in digital media technology in the China-UK Joint Education Program at Guangxi University of Finance and Economics,focusing on four aspects:learning materials,learning content,teacher conditions,and student learning outcomes.The research analysis in this paper not only provides strong support for the construction of China-UK Joint Education Program but also offers references for other China-UK Joint Education Programs.

Short-term displacement prediction for newly established monitoring slopes based on transfer learning

This study makes a significant progress in addressing the challenges of short-term slope displacement prediction in the Universal Landslide Monitoring Program,an unprecedented disaster mitigation program in China,where lots of newly established monitoring slopes lack sufficient historical deformation data,making it difficult to extract deformation patterns and provide effective predictions which plays a crucial role in the early warning and forecasting of landslide hazards.A slope displacement prediction method based on transfer learning is therefore proposed.Initially,the method transfers the deformation patterns learned from slopes with relatively rich deformation data by a pre-trained model based on a multi-slope integrated dataset to newly established monitoring slopes with limited or even no useful data,thus enabling rapid and efficient predictions for these slopes.Subsequently,as time goes on and monitoring data accumulates,fine-tuning of the pre-trained model for individual slopes can further improve prediction accuracy,enabling continuous optimization of prediction results.A case study indicates that,after being trained on a multi-slope integrated dataset,the TCN-Transformer model can efficiently serve as a pretrained model for displacement prediction at newly established monitoring slopes.The three-day average RMSE is significantly reduced by 34.6%compared to models trained only on individual slope data,and it also successfully predicts the majority of deformation peaks.The fine-tuned model based on accumulated data on the target newly established monitoring slope further reduced the three-day RMSE by 37.2%,demonstrating a considerable predictive accuracy.In conclusion,taking advantage of transfer learning,the proposed slope displacement prediction method effectively utilizes the available data,which enables the rapid deployment and continual refinement of displacement predictions on newly established monitoring slopes.

Adaptive Multi-Step Evaluation Design With Stability Guarantee for Discrete-Time Optimal Learning Control

This paper is concerned with a novel integrated multi-step heuristic dynamic programming(MsHDP)algorithm for solving optimal control problems.It is shown that,initialized by the zero cost function,MsHDP can converge to the optimal solution of the Hamilton-Jacobi-Bellman(HJB)equation.Then,the stability of the system is analyzed using control policies generated by MsHDP.Also,a general stability criterion is designed to determine the admissibility of the current control policy.That is,the criterion is applicable not only to traditional value iteration and policy iteration but also to MsHDP.Further,based on the convergence and the stability criterion,the integrated MsHDP algorithm using immature control policies is developed to accelerate learning efficiency greatly.Besides,actor-critic is utilized to implement the integrated MsHDP scheme,where neural networks are used to evaluate and improve the iterative policy as the parameter architecture.Finally,two simulation examples are given to demonstrate that the learning effectiveness of the integrated MsHDP scheme surpasses those of other fixed or integrated methods.

Detecting Design Patterns in Object-Oriented Program Source Code by Using Metrics and Machine Learning

Detecting well-known design patterns in object-oriented program source code can help maintainers understand the design of a program. Through the detection, the understandability, maintainability, and reusability of object-oriented programs can be improved. There are automated detection techniques;however, many existing techniques are based on static analysis and use strict conditions composed on class structure data. Hence, it is difficult for them to detect and distinguish design patterns in which the class structures are similar. Moreover, it is difficult for them to deal with diversity in design pattern applications. To solve these problems in existing techniques, we propose a design pattern detection technique using source code metrics and machine learning. Our technique judges candidates for the roles that compose design patterns by using machine learning and measurements of several metrics, and it detects design patterns by analyzing the relations between candidates. It suppresses false negatives and distinguishes patterns in which the class structures are similar. As a result of experimental evaluations with a set of programs, we confirmed that our technique is more accurate than two conventional techniques.

Prediction of Outcomes in Mini-Basketball Training Program for Preschool Children with Autism Using Machine Learning Models

In recent years evidence has emerged suggesting that Mini-basketball training program(MBTP)can be an effec-tive intervention method to improve social communication(SC)impairments and restricted and repetitive beha-viors(RRBs)in preschool children suffering from autism spectrum disorder(ASD).However,there is a considerable degree if interindividual variability concerning these social outcomes and thus not all preschool chil-dren with ASD profit from a MBTP intervention to the same extent.In order to make more accurate predictions which preschool children with ASD can benefit from an MBTP intervention or which preschool children with ASD need additional interventions to achieve behavioral improvements,further research is required.This study aimed to investigate which individual factors of preschool children with ASD can predict MBTP intervention out-comes concerning SC impairments and RRBs.Then,test the performance of machine learning models in predict-ing intervention outcomes based on these factors.Participants were 26 preschool children with ASD who enrolled in a quasi-experiment and received MBTP intervention.Baseline demographic variables(e.g.,age,body,mass index[BMI]),indicators of physicalfitness(e.g.,handgrip strength,balance performance),performance in execu-tive function,severity of ASD symptoms,level of SC impairments,and severity of RRBs were obtained to predict treatment outcomes after MBTP intervention.Machine learning models were established based on support vector machine algorithm were implemented.For comparison,we also employed multiple linear regression models in statistics.Ourfindings suggest that in preschool children with ASD symptomatic severity(r=0.712,p<0.001)and baseline SC impairments(r=0.713,p<0.001)are predictors for intervention outcomes of SC impair-ments.Furthermore,BMI(r=-0.430,p=0.028),symptomatic severity(r=0.656,p<0.001),baseline SC impair-ments(r=0.504,p=0.009)and baseline RRBs(r=0.647,p<0.001)can predict intervention outcomes of RRBs.Statistical models predicted 59.6%of variance in post-treatment SC impairments(MSE=0.455,RMSE=0.675,R2=0.596)and 58.9%of variance in post-treatment RRBs(MSE=0.464,RMSE=0.681,R2=0.589).Machine learning models predicted 83%of variance in post-treatment SC impairments(MSE=0.188,RMSE=0.434,R2=0.83)and 85.9%of variance in post-treatment RRBs(MSE=0.051,RMSE=0.226,R2=0.859),which were better than statistical models.Ourfindings suggest that baseline characteristics such as symptomatic severity of 144 IJMHP,2022,vol.24,no.2 ASD symptoms and SC impairments are important predictors determining MBTP intervention-induced improvements concerning SC impairments and RBBs.Furthermore,the current study revealed that machine learning models can successfully be applied to predict the MBTP intervention-related outcomes in preschool chil-dren with ASD,and performed better than statistical models.Ourfindings can help to inform which preschool children with ASD are most likely to benefit from an MBTP intervention,and they might provide a reference for the development of personalized intervention programs for preschool children with ASD.

Approximate Dynamic Programming for Self-Learning Control

This paper introduces a self-learning control approach based on approximate dynamic programming. Dynamic programming was introduced by Bellman in the 1950's for solving optimal control problems of nonlinear dynamical systems. Due to its high computational complexity, the applications of dynamic programming have been limited to simple and small problems. The key step in finding approximate solutions to dynamic programming is to estimate the performance index in dynamic programming. The optimal control signal can then be determined by minimizing (or maximizing) the performance index. Artificial neural networks are very efficient tools in representing the performance index in dynamic programming. This paper assumes the use of neural networks for estimating the performance index in dynamic programming and for generating optimal control signals, thus to achieve optimal control through self-learning.

Studying Design and Use of Healthcare Technologies in Interaction: The Social Learning Perspective in a Dutch Quality Improvement Collaborative Program

Designing technologies is a process that relies on multiple interactions between design and use contexts. These interactions are essential to the development and establishment of technologies. This article seeks to understand the attempts of healthcare organisations to integrate use contexts into the design of healthcare technologies following insights of the theoretical approaches of social learning and user representations. We present a multiple case study of three healthcare technologies involved in improving elderly care practice. These cases were part of a Dutch quality improvement collaborative program, which urged that development of these technologies was not “just” development, but should occur in close collaboration with other parts of the collaborative program, which were more focused on implementation. These cases illustrate different ways to develop technologies in interaction with use contexts and users. Despite the infrastructure of the collaborative program, interactions were not without problems. We conclude by arguing that interactions between design and use are not naturally occurring phenomena, but must be actively organised in order to create effects.

Improving Sensor-free Detection of Programming Difficulties Using Deep Learning

Programming difficulties are one of the common problems faced by software engineering students,which can lead to a rapid decline in motivation and even drop out.Probing students’programming difficulties is a crucial step in understanding their current programming situation and implementing appropriate instructional interventions.However,how to detect students’programming difficulties accurately without students’awareness remains a big challenge.Address the issues above;this paper adopts a sensor-free difficulties detecting method based on a deep neural network which employs a recurrent neural network(RNN)model and uses the sequential timing data from programming behaviour.The method can detect students’programming difficulties in real-time with 93%accuracy without interference in the programming process.In the long term,this method is the first step for establishing an automated intelligent programming environment.At the same time,it can assist teachers in noticing the difficulties that students encounter.Then,teachers can adjust their teaching plans and provide manual tutoring intervention more quickly.

An Empirical Study of the Optimum Team Size Requirement in a Collaborative Computer Programming/Learning Environment

Pair programming has been widely acclaimed the best way to go in computer programming. Recently, collaboration involving more subjects has been shown to produce better results in programming environments. However, the optimum group size needed for the collaboration has not been adequately addressed. This paper seeks to inculcate and acquaint the students involved in the study with the spirit of team work in software projects and to empirically determine the effective (optimum) team size that may be desirable in programming/learning real life environments. Two different experiments were organized and conducted. Parameters for determining the optimal team size were formulated. Volunteered participants of different genders were randomly grouped into five parallel teams of different sizes ranging from 1 to 5 in the first experiment. Each team size was replicated six times. The second experiment involved teams of same gender compositions (males or females) in different sizes. The times (efforts) for problem analysis and coding as well as compile-time errors (bugs) were recorded for each team size. The effectiveness was finally analyzed for the teams. The study shows that collaboration is highly beneficial to new learners of computer programming. They easily grasp the programming concepts when the learning is done in the company of others. The study also demonstrates that the optimum team size that may be adopted in a collaborative learning of computer programming is four.

Diagnosing Student Learning Problems in Object Oriented Programming

Students often face difficulties while taking basic programming courses due to several factors. In response, research has presented subjective assessments for diagnosing learning problems to improve the teaching of programming in higher education. In this paper, the authors propose an Object Oriented conceptual map model and organize this approach into three levels： constructing a Concept Effect Propagation Table, constructing Test Item-Concept Relationships and diagnosing Student Learning Problems with Matrix Composition. The authors＇ work is a modification of the approaches of Chert and Bai as well as Chu et al., as the authors use statistical methods, rather than fuzzy sets, for the authors＇ analysis. This paper includes a statistical summary, which has been tested on a small sample of students in King Abdulaziz University, Jeddah, Saudi Arabia, illustrating the learning problems in an Object Oriented course. The experimental results have demonstrated that this approach might aid learning and teaching in an effective way.

Using Deep Learning to Improve Automated Program Static Assessment

Evaluation is an essential part of the teaching process,especially in the programming course.Both students and teachers can benefit significantly from automatic program evaluation.It shortens the time required for assessment so that students can get immediate feedback.At the same time,it can also significantly reduce the workload of teachers.Currently,the automated program assessment system mainly uses a combination of static and dynamic analysis methods.The system is faced with two crucial problems of the unfinished code evaluation and the template code construction.This paper proposes a method of combining deep learning with static analysis.The syntax tree repair is used to solve the problem that the code with compiling errors cannot generate the correct syntax tree.Moreover,the target code is converted to a subset of solution space through the syntax tree standardization,which reduces the number of template code needed.Based on deep learning,the embedded token vector keeps the code’s context all the time,which ensures that the lexical-semantic remains unchanged as much as possible after the syntax tree changes.Finally,the standardized tree is represented as a vector by the recursive neural network.Cosine similarity between target and template code vectors is used as an evaluation score.The experiment shows that the similarity scores obtained by this method are consistent with the expert scores.This method can provide support for future research,such as difficult feedback and has great significance.

Impact of Intercultural Learning Program on Language Learners＇ Intercultural Competence

As institutions of higher education continue to internationalize, the number of exchanged students climbs ever higher in China. It is important to know what they have gained in the intercultural learning experiences in terms of attitudes, awareness, and knowledge, which are the three constructed dimensions of Intercultural Competence in this study. A qualitative research is conducted by using interview and seven English language learners are investigated to explore their learning outcomes in the intercultural learning program. From the thematic analysis of the interviews, it can be found that these English language learners＇ intercultural competence has been improved during their stay in USA. They successfully overcome cultural shock, hold positive attitudes toward target culture and people, become more confident in English study, and acquire more cultural knowledge. The research outcomes contribute to promoting intercultural learning effectiveness and improving learners＇ intercultural competence.

基于Q-learning算法的配电网储能装置控制策略研究

通过在配电网末端接入用于系统调压等辅助服务的储能装置,能有效解决可再生能源的高度间歇性和负荷需求波动导致的配变过载问题。基于强化学习的Q-learning算法,针对储能电池运行情况进行建模仿真,通过单时段优化内嵌的Q值得到各时段储能电池荷电状态的最优调度方案。实例试验分析表明,当迭代次数达到一定数量时,利用Q-learning算法能够达到理论上的最优解。最后,通过将Q-learning算法与动态规划算法生成的标准最优调度方案进行对比,证明了Q-learning算法能够与动态规划算法达成一致最优解。

A new strategy of integrating American culture with English teaching ＆ learning by using VOA Special English feature programs

The paper mainly studies a new way of combining American culture with English language teaching and learning by means of the VOA Special English feature programs. It is based on a 5-year period of teaching practice, experience and reflection. This interactive, innovative, productive and student-centered approach is a far cry from the rigid, old, unproductive and teacher-centered approach of teaching British and American culture without combining language learning and practice as is commonly used here in Chinese colleges. The author also throws some light on how to use on-line teaching resources and how to cultivate independent learning on the part of the students he has worked with.

Innovation in Adult Language Learning Programs: A Case Study of "Livemocha" Online Language Learning Community

Due to the increasing broadband Internet access, VOIP adoption as well as the Web 2.0 concepts, online language learning communities have gained great popularity among adult language learners. Based upon Malcolm Knowles' adult learning theory, this paper analyzes the innovation that online language learning communities have brought to adult language learning programs. At the same time, the limitations of such programs have been pointed out.

Call for papers Journal of Control Theory and Applications Special issue on Approximate dynamic programming and reinforcement learning

Approximate dynamic programming (ADP) is a general and effective approach for solving optimal control and estimation problems by adapting to uncertain and nonconvex environments over time.

Current Trends in Online Programming Languages Learning Tools: A Systematic Literature Review

Students face difficulties in programming languages learning (PLL) which encourages many scholars to investigate the factors behind that. Although there a number of positive and negative factors found to be effective in PLL procedure, utilising online tools in PLL were recognized as a positive recommended means. This motivates many researchers to provide solutions and proposals which result in a number of choices and options. However, categorising those efforts and showing what has been done, would provide a better and clear image for future studies. Therefore, this paper aims to conduct a systematic literature review to show what studies have been done and then categorise them based on the type of online tools and the aims of the research. The study follows Kitchenham and Charters guidelines for writing SLR (Systematic Literature Review). The search result reached 1390 publications between 2013-09/2018. After the filtration which has been done through selected criteria, 160 publications were found to be adequate to answer the review questions. The main results of this systematic review are categorizing the aims of the studies in online PLL tools, classifying the tools and finding the current trends of the online PLL tools.

MET as Program Resources for College English Teaching and Learning：Application and Implications

This paper investigates the current situation of Modern Educational Technology （MET） as program resources for college English teaching, exploring the effectiveness of the integration of MET and language teaching and learning. The case study in this paper illustrates that MET does benefit college English teaching but there are many problems of its application. Pedagogical implications are then provided to enhance the effective integration of MET and language teaching and learning： （1） it is wise for the teachers and learners to reconcile what is desirable with what is acceptable and possible in MET application; （2） teachers should optimize other existing resources to solve the problems caused by inequality and unavailability of MET; （3） teachers should have a clear conception of the relationship of MET and pedagogy to design and apply courseware appropriately; （4） students have beliefs in the benefits of MET, resulting in enhancing their individual language learning and efficiency.