Deep reinforcement learning using least-squares truncated temporal-difference

Policy evaluation(PE)is a critical sub-problem in reinforcement learning,which estimates the value function for a given policy and can be used for policy improvement.However,there still exist some limitations in current PE methods,such as low sample efficiency and local convergence,especially on complex tasks.In this study,a novel PE algorithm called Least-Squares Truncated Temporal-Difference learning(LST2D)is proposed.In LST2D,an adaptive truncation mechanism is designed,which effectively takes advantage of the fast convergence property of Least-Squares Temporal Difference learning and the asymptotic convergence property of Temporal Difference learning(TD).Then,two feature pre-training methods are utilised to improve the approximation ability of LST2D.Furthermore,an Actor-Critic algorithm based on LST2D and pre-trained feature representations(ACLPF)is proposed,where LST2D is integrated into the critic network to improve learning-prediction efficiency.Comprehensive simulation studies were conducted on four robotic tasks,and the corresponding results illustrate the effectiveness of LST2D.The proposed ACLPF algorithm outperformed DQN,ACER and PPO in terms of sample efficiency and stability,which demonstrated that LST2D can be applied to online learning control problems by incorporating it into the actor-critic architecture.

Statistical Analysis and Evaluation of Macroeconomic Policies: A Selective Review

In this paper,we highlight some recent developments of a new route to evaluate macroeconomic policy effects,which are investigated under the framework with potential outcomes.First,this paper begins with a brief introduction of the basic model setup in modern econometric analysis of program evaluation.Secondly,primary attention goes to the focus on causal effect estimation of macroeconomic policy with single time series data together with some extensions to multiple time series data.Furthermore,we examine the connection of this new approach to traditional macroeconomic models for policy analysis and evaluation.Finally,we conclude by addressing some possible future research directions in statistics and econometrics.

Implementing Open Access Policy:First case studies

When implementing open access, policy pioneers and flagship institutions alike have faced considerable challenges in meeting their own aims and achieving a recognized success. Legitimate authority, sufficient resources and the right timing are crucial, but the professionals charged with implementing policy still need several years to accomplish significant progress. This study defines a methodological standard for evaluating the first generation of open access policies. Evaluating implementation establishes evidence, enables reflection, and may foster the emergence of a second generation of open access policies.While the study is based on a small number of cases, these case studies cover most of the pioneer institutions, present the most significant issues and offer an international overview.Each case is reconstructed individually on the basis of public documents and background information, and supported by interviews with professionals responsible for open access implementation. This article presents the highlights from each case study. The results are utilized to indicate how a second generation of policies might define open access as a key component of digital research infrastructures that provide inputs and outputs for research,teaching and learning in real time.

Evaluation of the Effect of China’s Health Care Reform Based on Prospect Theory

Objective To evaluate the effect of health care reform policy in China comprehensively and provide suggestions for its further implementation.Methods Data on the effect of health care reform were obtained from“China Health Statistics Yearbook”and National Bureau of Statistics of China and the indicators were selected by corrected item total correlation(CITC)and Cronbach’sαreliability coefficient.Then,the selected indicators were calculated through the prospect theory model.Meanwhile,the gray relation analysis method was introduced to enlarge the differences between the advantages and disadvantages to make the comprehensive evaluation result more obvious.Results and Conclusion The implementation of China’s health care reform has a significant impact on China’s medical and health system.However,the effect of the policy will become less with the increase of the total amount.An effective management can ensure that the policy continues to play its role.

人工关节国家集中带量采购的政策效果分析

目的利用中选结果数据评估人工关节集中带量采购的政策效果,以完善后续高值医用耗材集中带量采购政策。方法利用韦唐政策评估模式中的目标模式,依据人工关节集中带量采购的政策目标,构建效果评估体系,以评估人工关节集中带量采购政策效果。结果根据韦唐政策评估模式中的目标模式评估,人工关节集中带量采购平均降幅66.60%,带量比例超90.00%,价差普遍控制在2倍左右,关联中选情况少,企业分量增幅与报价及需求量的相关性符合预期,91.67%企业直接或复活中选,88.32%的分量具有延续性,至少56.87%的跨类别分量较为合理,因此基本实现了政策目标,但也出现了少数组别中选价差较大、伴随服务报价价差较大以及对医疗机构高价偏好的行为缺乏应对措施等问题。结论人工关节集中带量采购基本实现了降低耗材价格、挤压灰色空间、稳定价差、保证企业参与积极性、提高临床选择性、稳定临床供给和使用、优质产品临床替代的政策目标。建议:优化价差控制机制,减少异常中选价的出现;细化伴随服务报价规则,优化价格形成机制;避免医疗机构选择高价产品的倾向对集采产生负面影响。

Distributed policy evaluation via inexact ADMM in multi-agent reinforcement learning

This paper studies a distributed policy evaluation in multi-agent reinforcement learning.Under cooperative settings,each agent only obtains a local reward,while all agents share a common environmental state.To optimize the global return as the sum of local return,the agents exchange information with their neighbors through a communication network.The mean squared projected Bellman error minimization problem is reformulated as a constrained convex optimization problem with a consensus constraint;then,a distributed alternating directions method of multipliers(ADMM)algorithm is proposed to solve it.Furthermore,an inexact step for ADMM is used to achieve efficient computation at each iteration.The convergence of the proposed algorithm is established.yipeng@tongji.edu.cn;LilLi received the B.Sc.and M.Se.degrees from Shengyang Agri-culture University,China in 1996 and 1999.respectivly.and the Ph.D.degree from Shenyang Institute of Automation,Chinese Academy of Science,in 2003.She joined Tongji Universitry,Shanghai,China,in 2003,and is now a professor at the Depart-ment of Control Science and Engineering.Her research inter-ests are in data-driven modeling and opimization,computaional intelligence.

Effects of the Northeast China Revitalization Strategy on Regional Economic Growth and Social Development

Measuring the economic and social effects of the Northeast China Revitalization Strategy is critical to addressing regional sustainable development in China. To shed light on this issue, an integrated perspective was adopted that is combined with the difference-in-differences method to measure the effects of the strategy on economic growth and social development in Northeast China. The findings suggest that the strategy has significantly improved regional economic growth and per-capita income by increasing its gross domestic product(GDP) and GDP per capita by 25.70% and 46.00%, respectively. However, the strategy has significantly worsened the regional employment in the secondary industry of the region. In addition, the strategy has not significantly improved regional infrastructural road, education investment or social security, and has had no significant effect on mitigating regional disparity. In addition, the policy effects are highly heterogeneous across cities based on city size and characteristics. Therefore, there is no simple answer regarding whether the Northeast China Revitalization Strategy has reached its original goals from an integrated perspective. The next phase of the strategy should emphasize improving research and development(R&D) and human capital investments based on urban heterogeneity to prevent conservative path-dependency and the lock-in of outdated technologies.

Efficient policy evaluation by matrix sketching

In the reinforcement learning,policy evaluation aims to predict long-term values of a state under a certain policy.Since high-dimensional representations become more and more common in the reinforcement learning,how to reduce the computational cost becomes a significant problem to the policy evaluation.Many recent works focus on adopting matrix sketching methods to accelerate least-square temporal difference(TD)algorithms and quasi-Newton temporal difference algorithms.Among these sketching methods,the truncated incremental SVD shows better performance because it is stable and efficient.However,the convergence properties of the incremental SVD is still open.In this paper,we first show that the conventional incremental SVD algorithms could have enormous approximation errors in the worst case.Then we propose a variant of incremental SVD with better theoretical guarantees by shrinking the singular values periodically.Moreover,we employ our improved incremental SVD to accelerate least-square TD and quasi-Newton TD algorithms.The experimental results verify the correctness and effectiveness of our methods.

Current Situation of E-commerce Entering Rural Areas in Datong County of Qinghai Province

Datong County has been developed for nearly 5 years since it was selected as a comprehensive demonstration county of e-commerce entering rural areas in 2017.In this context,this paper analyzes the development status of e-commerce in rural areas of Datong County,and provides feasible suggestions for the development of e-commerce entering rural areas in demonstration counties.

Evaluating the Impacts of Food Safety Policies on Japan's Rice Imports Based on the Simultaneous Buy and Sell System

This study evaluated the impacts of food safety policies on Japan's Simultaneous Buy and Sell rice imports through measuring tariff equivalents of food safety policies.In order to construct an estimated model,a Japanese consumer's utility function is introduced and developed with consumer's preference parameters and elasticity of substitution.In the empirical study part,Japan's positive list system and rice traceability were analyzed and assessed as critical food safety policies.Results showed that after the implementation of the positive list system,consumers'preference for foreign rice and the substitution elasticity diminished.This decreasing tendency was quite similar to the results after the enforcement of rice traceability.The tariff equivalents of food safety policies on imported rice fluctuated around ¥50 yen/kg from fiscal year 2000 to 2005 and decreased because of the global grain price hike after 2006.The tariff equivalents soared in 2010,which was induced by the traceability regulation,and then dulled during Japan's earthquake and tsunami in 2011.Subsequently,after the recovery from natural disasters,the tariff equivalents of food safety policies became higher.Therefore,food safety policies had made imported rice less attractive,weakened the competitive power of rice exporting countries,and had statistically significant impacts on Japan's rice importation.

New trends in China's science and education evaluation research under the background of“Breaking the Five- only”

Based on the background of the special clean-up action of"Breaking the Five-only",this paper combs the relevant policies of domestic science and education evaluation.Using CiteSpace and VOSviewer scientific measurement software,this paper makes a visual analysis on the related domestic research of the"Five-only"and"science and education evaluation",and expounds the frontier hot spots and trends of science and education evaluation research in China.Based on this,this paper summarizes the countermeasures and suggestions on how to"break"the"Five-only"and how to"establish"the"new system of science and education evaluation",in order to provide a reference for the sustainable and healthy development of science and education evaluation in China.

A framework for the extended evaluation of ABAC policies

A main challenge of attribute-based access control(ABAC)is the handling of missing information.Several studies have shown that the way standard ABAC mechanisms,e.g.based on XACML,handle missing information is flawed,making ABAC policies vulnerable to attribute-hiding attacks.Recent work has addressed the problem of missing information in ABAC by introducing the notion of extended evaluation,where the evaluation of a query considers all queries that can be obtained by extending the initial query.This method counters attribute-hiding attacks,but a na飗e implementation is intractable,as it requires an evaluation of the whole query space.In this paper,we present a framework for the extended evaluation of ABAC policies.The framework relies on Binary Decision Diagram(BDDs)data structures for the efficient computation of the extended evaluation of ABAC policies.We also introduce the notion of query constraints and attribute value power to avoid evaluating queries that do not represent a valid state of the system and to identify which attribute values should be considered in the computation of the extended evaluation,respectively.We illustrate our framework using three real-world policies,which would be intractable with the original method but which are analyzed in seconds using our framework.

A framework for the extended evaluation of ABAC policies

A main challenge of attribute-based access control(ABAC)is the handling of missing information.Several studies have shown that the way standard ABAC mechanisms,e.g.based on XACML,handle missing information is flawed,making ABAC policies vulnerable to attribute-hiding attacks.Recent work has addressed the problem of missing information in ABAC by introducing the notion of extended evaluation,where the evaluation of a query considers all queries that can be obtained by extending the initial query.This method counters attribute-hiding attacks,but a naïve implementation is intractable,as it requires an evaluation of the whole query space.In this paper,we present a framework for the extended evaluation of ABAC policies.The framework relies on Binary Decision Diagram(BDDs)data structures for the efficient computation of the extended evaluation of ABAC policies.We also introduce the notion of query constraints and attribute value power to avoid evaluating queries that do not represent a valid state of the system and to identify which attribute values should be considered in the computation of the extended evaluation,respectively.We illustrate our framework using three real-world policies,which would be intractable with the original method but which are analyzed in seconds using our framework.

Quantifying the impacts of emissions and meteorology on the interannual variations of air pollutants in major Chinese cities from 2015 to 2021

Air pollutant concentration is a function of emission rates and meteorology.To accurately evaluate the effect of control measures,the meteorological covariate must be corrected from the observations.This study quantified the impacts of emission abatement and meteorological condition on the interannual variations of SO_(2),NO_(2),CO,O_(3),PM_(10) and PM_(2.5) concentrations in 31 major Chinese cities using an optimized machine learning-based meteorological normalization technique.Overall,the annual average concentrations of SO_(2),NO_(2),CO,PM_(10)and PM_(2.5)were reduced by 86%,51%,99%,86% and 88%from 2015 to 2020,respectively,in the studied cities,attributable to their emission reductions.However,the concentration of O_(3) was found with no significant decrease with the reduction of precursors.Emission abatement notably improved air quality between 2015 and 2018.Such a decline in emissions tended to progressively slow down since 2018.Overall,the meteorological conditions in 2016–2017 and 2018–2019 were unfavorable for a better air quality,while it became favorable in 2020–2021.Specifically,emission abatement in 2021 further lowered the concentrations of SO_(2),NO_(2),CO,and PM_(2.5),while the emission of PM_(10) increased.And changes in precursors emissions worsened O_(3) air quality.To meet the demand of improving air quality,more aggressive abatement measures need to be formulated to synergistically reduce NOx,volatile organic compounds,and coarse particles.

Do Free Trade Zones Improve Air Quality?

As a new window of opening up to the outside world in the new era,the establishment of Free Trade Zones(FTZs)in China is an important national strategy for promoting high-quality economic development wherein the prevention and control of pollution is an important hurdle to be surmounted throughout the process.Based on data taken from model cities for environmental protection in China from 2008 to 2017,given the effect of policy spillover,this study considers the establishment of FTZs to be a"quasinatural experiment."It uses the general analysis paradigm of spatial difference-indifference(DID)to systematically examine the impact of FTZs on air pollution as well as to conduct an in-depth analysis of their spatial heterogeneity and mechanism of action.The study shows that the establishmentofFTZs significantlyreduces the concentration of air pollutants in cities.If the spatial DID method is adopted to measure the policy spillover effect of the establishment of free trade zones,urban air pollutants declines by 12%to 17%,while the estimated result using the traditional DID method is only 7%.The establishment of FTZs significantly alleviates air pollution in neighboring non-pilot cities as well.Based on the range of the spillover effect from the center of the FTZ,it is found that the average spatial spillover effect presents as an inverted"U"curve as the research radius increases,with an optimal policy spillover effect radius of about 200km,while the policy spillover effect between pilot cities remains poor.The establishment of an FTZ not only improves the local atmospheric environment by promoting industrial structure optimization and green technology innovation in pilot cities but also generates a spillover effect on neighboring non-pilot cities through the same mechanism,thus contributing to improving the atmospheric environment in those non-pilot cities.These findings remain valid following a series of robustness tests such as the spatial parallel trend test and the placebo test.This study offers an answer to the key question of whether free trade zone policy can truly promote high-quality economic development in the new era.It provides useful policy insights for further expanding opening up,winning the battle for the prevention and control of pollution,and promoting the high-quality development of China's economy.

Empirical methods in studying the effect of pharmaceutical policies

Healthcare policies are often evaluated using empirical analysis methods. In this study, we summarized the hypothesis test, segmented regression analysis of interrupted time series, autoregressive integrated moving average model, difference in difference model, pooled regression model, fixed effect model, random effect model, propensity score matching method, cost-benefit analysis, and cost-effectiveness analysis. We also established a flow chart to demonstrate the process of selection of empirical analysis methods based on the purpose of policy evaluation, whether there is a control group, the type of data, and whether the data is linear.

Are Confucius Institutes Building Blocks or Stumbling Blocks for Foreign Students in China: An Empirical Study of 190 Countries (1999–2015)

Purpose:This study investigates the impact of Confucius Institutes on China’s international education initiatives.First,it examines whether the establishment of Confucius Institutes has a gravitational effect on the scale of foreign students coming to China.Second,it discerns whether there are any association between the establishment of Confucius Institutes and the attributes of foreign students based on the type of program they select.Third,it identifies whether there are any differences in the impact of Confucius Institutes based on constraints such as the number of foreign students,income levels,cultural distance,and geographical endowment.Design/Approach/Methods:Based on the trade gravity model and the push–pull theory,we estimated a two-way fixed-effects model using panel data of all source countries from 1999 to 2015.Findings:Results show that Confucius Institutes have helped China attract more foreign students on Chinese government scholarships,with each Institute increasing such enrolment by 1.3%.However,Confucius Institutes have resulted in a drop in the total number of foreign students studying in China,especially nondegree students.This substitution effect means that the Institutes actually help China improve the composition of its foreign students.Further heterogeneity analysis shows that the substitution effect is primarily driven by the main source countries.Accordingly,this study suggests that China should establish more Confucius Institutes in Africa and Eastern Europe to maximize the catalyzing effect of Confucius Institutes.Originality/Value:In contrast to previous studies on foreign scholarship in China,this study examines all 190 source countries from 1999 to 2015.This comprehensive study also explores the heterogeneous effects of foreign students on trade across economic,cultural,and geographical domains.