Stock Market Trading Based on Market Sentiments and Reinforcement Learning

Stock market is a place,where shares of different companies are traded.It is a collection of buyers’and sellers’stocks.In this digital era,analysis and prediction in the stock market have gained an essential role in shaping today’s economy.Stock market analysis can be either fundamental or technical.Technical analysis can be performed either with technical indicators or through machine learning techniques.In this paper,we report a system that uses a Reinforcement Learning(RL)network and market sentiments to make decisions about stock market trading.The system uses sentiment analysis on daily market news to spot trends in stock prices.The sentiment analysis module generates a unified score as a measure of the daily news about sentiments.This score is then fed into the RL module as one of its inputs.The RL section gives decisions in the form of three actions:buy,sell,or hold.The objective is to maximize long-term future profits.We have used stock data of Apple from 2006 to 2016 to interpret how sentiments affect trading.The stock price of any company rises,when significant positive news become available in the public domain.Our results reveal the influence of market sentiments on forecasting of stock prices.

Multi-agent Simulation for Strategic Bidding in Electricity Markets Using Reinforcement Learning

In this paper,a theoretical framework of Multiagent Simulation(MAS)is proposed for strategic bidding in electricity markets using reinforcement learning,which consists of two parts:one is a MAS system used to simulate the competitive bidding of the actual electricity market;the other is an adaptive learning strategy bidding system used to provide agents with more intelligent bidding strategies.An ExperienceWeighted Attraction(EWA)reinforcement learning algorithm(RLA)is applied to the MAS model and a new MAS method is presented for strategic bidding in electricity markets using a new Improved EWA(IEWA).From both qualitative and quantitative perspectives,it is compared with three other MAS methods using the Roth-Erev(RE),Q-learning and EWA.The results show that the performance of the MAS method using IEWA is proved to be better than the others.The four MAS models using four RLAs are built for strategic bidding in electricity markets.Through running the four MAS models,the rationality and correctness of the four MAS methods are verified for strategic bidding in electricity markets using reinforcement learning.

Practical Meta-Reinforcement Learning of Evolutionary Strategy with Quantum Neural Networks for Stock Trading

We show the practicality of two existing meta-learning algorithms Model- Agnostic Meta-Learning and Fast Context Adaptation Via Meta-learning using an evolutionary strategy for parameter optimization, as well as propose two novel quantum adaptations of those algorithms using continuous quantum neural networks, for learning to trade portfolios of stocks on the stock market. The goal of meta-learning is to train a model on a variety of tasks, such that it can solve new learning tasks using only a small number of training samples. In our classical approach, we trained our meta-learning models on a variety of portfolios that contained 5 randomly sampled Consumer Cyclical stocks from a pool of 60. In our quantum approach, we trained our quantum meta-learning models on a simulated quantum computer with portfolios containing 2 randomly sampled Consumer Cyclical stocks. Our findings suggest that both classical models could learn a new portfolio with 0.01% of the number of training samples to learn the original portfolios and can achieve a comparable performance within 0.1% Return on Investment of the Buy and Hold strategy. We also show that our much smaller quantum meta-learned models with only 60 model parameters and 25 training epochs have a similar learning pattern to our much larger classical meta-learned models that have over 250,000 model parameters and 2500 training epochs. Given these findings, we also discuss the benefits of scaling up our experiments from a simulated quantum computer to a real quantum computer. To the best of our knowledge, we are the first to apply the ideas of both classical meta-learning as well as quantum meta-learning to enhance stock trading.

Agent-based Modeling and Simulation for the Electricity Market with Residential Demand Response

Currently,critical peak load caused by residential customers has attracted utility companies and policymakers to pay more attention to residential demand response(RDR)programs.In typical RDR programs,residential customers react to the price or incentive-based signals,but the actions can fall behind flexible market situations.For those residential customers equipped with smart meters,they may contribute more DR loads if they can participate in DR events in a proactive way.In this paper,we propose a comprehensive market framework in which residential customers can provide proactive RDR actions in a day-ahead market(DAM).We model and evaluate the interactions between generation companies(GenCos),retailers,residential customers,and the independent system operator(ISO)via an agent-based modeling and simulation(ABMS)approach.The simulation framework contains two main procedures—the bottom-up modeling procedure and the reinforcement learning(RL)procedure.The bottom-up modeling procedure models the residential load profiles separately by household types to capture the RDR potential differences in advance so that residential customers may rationally provide automatic DR actions.Retailers and GenCos optimize their bidding strategies via the RL procedure.The modified optimization approach in this procedure can prevent the training results from falling into local optimum solutions.The ISO clears the DAM to maximize social welfare via Karush-Kuhn-Tucker(KKT)conditions.Based on realistic residential data in China,the proposed models and methods are verified and compared in a large multi-scenario test case with 30,000 residential households.Results show that proactive RDR programs and interactions between market entities may yield significant benefits for both the supply and demand sides.The models and methods in this paper may be used by utility companies,electricity retailers,market operators,and policy makers to evaluate the consequences of a proactive RDR and the interactions among multi-entities.

一个基于Agent的股票市场仿真模型的Swarm实现

Swarm是一个实现基于Agent的仿真模型的软件工具。首先介绍了使用Swarm仿真平台实现基于Agent的仿真模型的一般方法和应考虑的关键问题,然后详细地阐述了使用Swarm来实现一个基于Agent的连续竞价股票市场仿真模型中的Agent和Agent交互的方法与过程。为该领域的研究者在模型的实现这一问题上提供了一定的参考和借鉴。

基于Agent的股票交易模拟及应用

股票市场是市场经济的重要组成部分。但是现有的基于演绎推理的理论分析方法在处理股市这类复杂性系统时遇到了很多困难,因此基于归纳推理的实验经济学方法成为一种可行的选择。论文基于多Agent系统,采用再励学习算法模拟交易者行为特征,实现了一个股票市场的模拟系统,并且应用这一系统研究了涨跌停板交易机制对于股市的影响。

基于Agent的股市随机过程方法预测

利用Swarm仿真平台和Agent建模技术 ,结合马尔可夫过程对证券市场的股指进行了研究 ,提出多Agent变区间马尔可夫预测方法 ,并对模型进行了仿真 。

基于多Agent的股票市场流动性仿真研究

基于多Agent的金融市场仿真是随着金融市场微观经济理论和行为金融学的发展而兴起的一种金融复杂系统研究方法。本文用多Agent仿真方法对连续竞价股票市场进行了模拟,分析了交易者之间的学习与交互行为对整个市场流动性造成的影响。

基于深度学习的智能交易Agent的交易行为研究

基于Agent的模型（ABM）在许多领域取得了显著研究成果,在Agent设计等方面的改进也层出不穷。由于在真实市场中Agent之间拥有不同的学习能力、不同的学习速度、不同的社交网络,因此不同的Agent设定使得模型结果也不一致。为了得到更一般的结论,文章将在深度学习的基础上融合行为金融学,设定各种类型的Agent模拟股票市场。实证结果表明文章中的模型能够很好地反映真实股票市场的运行情况,表明智能Agent交易行为的变化和股市动态具有较强的相关性。通过规范市场Agent相关行为可以起到规范市场的作用。

基于多Agent仿真技术的虚拟股票市场分析

准确地对股票市场进行预测,是股票投资者和政府宏观调控的必然需求。多Agent仿真技术利用复杂系统理论,能够对复杂的社会、经济以及生态等系统进行仿真模拟。本文利用多Agent技术对股市进行建模,并利用MASON工具对股票市场进行仿真。实验结果表明,多Agent仿真技术能够较好地反映股票市场的基本特性,从而为股票投资和宏观调控提供良好的决策支持。

Development of deep-learning-based autonomous agents for low-speed maneuvering in Unity

This study provides a systematic analysis of the resource-consuming training of deep reinforcement-learning (DRL) agents for simulated low-speed automated driving (AD). In Unity, this study established two case studies: garage parking and navigating an obstacle-dense area. Our analysis involves training a path-planning agent with real-time-only sensor information. This study addresses research questions insufficiently covered in the literature, exploring curriculum learning (CL), agent generalization (knowledge transfer), computation distribution (CPU vs. GPU), and mapless navigation. CL proved necessary for the garage scenario and beneficial for obstacle avoidance. It involved adjustments at different stages, including terminal conditions, environment complexity, and reward function hyperparameters, guided by their evolution in multiple training attempts. Fine-tuning the simulation tick and decision period parameters was crucial for effective training. The abstraction of high-level concepts (e.g., obstacle avoidance) necessitates training the agent in sufficiently complex environments in terms of the number of obstacles. While blogs and forums discuss training machine learning models in Unity, a lack of scientific articles on DRL agents for AD persists. However, since agent development requires considerable training time and difficult procedures, there is a growing need to support such research through scientific means. In addition to our findings, we contribute to the R&D community by providing our environment with open sources.

基于演员-评论家框架的层次化多智能体协同决策方法

针对复杂作战环境下多智能体协同决策中出现的任务分配不合理、决策一致性较差等问题,提出一种基于演员-评论家(Actor-Critic,AC)框架的层次化多智能体协同决策方法。通过将决策过程分为不同层次,并使用AC框架来实现智能体之间的信息交流和决策协同,以提高决策效率和战斗力。在高层次,顶层智能体制定任务决策,将总任务分解并分配给底层智能体。在低层次,底层智能体根据子任务进行动作决策,并将结果反馈给高层次。实验结果表明,所提方法在多种作战仿真场景下均取得了较好的性能,展现了其在提升军事作战协同决策能力方面的潜力。

基于多代理强化学习的多新型市场主体虚拟电厂博弈竞价及效益分配策略

目前新型市场主体规模较小但数量众多,为提高竞争力可以使其组成联盟以多新型市场主体虚拟电厂的形式参与市场博弈,而公平的效益分配方法是维持联盟稳定的基础。为此,该文提出了一种多新型市场主体虚拟电厂博弈竞价及效益分配策略。首先,考虑多新型市场主体虚拟电厂和传统机组均作为价格影响者,构建包含电能量和备用辅助服务的主辅联合市场交易模型,并在不完全信息市场环境下采用多代理强化学习(multi-agentreinforcementlearning,MADDPG)算法求解。其次,采用分布式联盟构造方法得到最优多新型市场主体联盟结构。为解决效益分配方法中的维数灾问题,引入蒙特卡洛近似夏普利值,对虚拟电厂内各新型市场主体的超额收益进行合理分配。最后,算例分析表明所提方法给出了多新型主体虚拟电厂参与主辅联合市场的最优联盟结构和竞价策略,在保证精度的前提下提高了超额收益分配的计算速度,与单独参与市场相比提高了所有新型市场主体的收益。

基于智能体建模的新型电力系统下火电企业市场交易策略

高比例新能源渗透情景下火电企业竞价策略研究对保障火电企业运营和推进新型电力系统建设具有重要意义。基于智能体建模框架,建立电力现货市场仿真模型和机组自学习决策模型。其中,环境模块建立了考虑源荷双侧不确定性的风光火储多方参与的电力现货市场出清模型;智能体模块将火电机组投标决策过程刻画为部分观测马尔科夫决策过程,采用深度确定性策略梯度算法求解。以HRP-38节点系统为例进行仿真分析,明晰高比例新能源下火电企业市场交易策略。结果表明:在不考虑火电机组提供辅助服务的前提下,随着新能源渗透率的提高,仍有部分位置独特且具有成本优势的火电机组拥有竞争力;预测误差增大将使大容量火电机组投标策略趋于保守,而小容量机组投标策略相反;火电机组在各类场景下均具有隐性共谋倾向,即彼此隐藏信息时仍同时提高报价。

市场环境下智能配用电系统分层协同优化运行:研究挑战、进展与展望

随着分布式资源在配电网中的比例不断提高,如何在市场化交易机制下实现配用电系统安全经济运行成为当下的研究热点。在市场环境下,配用电系统各层的运行管理面临着不确定性逐层加剧、市场规模快速扩展、市场交易与系统安全运行难以有效衔接等多重挑战。该文首先梳理市场环境下配用电系统运行优化的关键问题;其次,对传统解析优化方法的研究成果与研究中仍待解决的问题进行总结;然后,针对配用电系统市场交易、运行优化等问题特点,系统性地介绍深度强化学习技术,分析归纳深度强化学习在配用电系统中的研究现状。最后,提炼出贯穿配用电系统多层多主体协同优化问题中的三重研究需求,并对深度强化学习技术未来的应用路径与发展趋势进行展望。

基于两阶段深度强化学习算法的多智能体自由合谋竞价机理研究

电力市场建设初期,不完善的监管机制为发电商提供了暗中交流,联合竞价的机会。然而,如何找到潜在的发电商合谋组合是相对困难的事情。针对这一问题,该文建立一种允许发电商自由联合的竞价模型,并提出全新的两阶段深度强化学习算法,来求解由离散的合谋对象选择和连续的报价系数确定组合形成的离散、连续动作混合决策问题。在不同阻塞情况下,对发电商联合策略形成过程进行分析,并在大算例中验证了算法的有效性。仿真结果表明,所提出的方法可以对市场主体的自由联合行为进行有效模拟,发现潜在的合谋组合。

计及风电的发电商报价多智能体模型

新型电力系统背景下,新能源发电商的报价问题一直是电力现货市场中的研究热点。相比传统能源,风电出力受外界不确定性因素的影响较大,给风力发电商求解最优报价带来了挑战。为此,基于多智能体强化学习算法WoLF-PHC构建了计及风电的发电商报价策略模型。模型中,考虑了风电、火电和水电3种能源参与的现货市场,每一个发电商抽象为一个智能体,且基于随机约束规划算法建模风电智能体的收益函数;对于智能体的报价策略模型,将D3QN与WoLF-PHC算法结合,使模型能够满足报价时智能体状态空间复杂的情况;此外,对于交互环境的建模,提出利用DDPM扩散模型生成风电出力数据,优化风电出清场景的仿真。最后,基于3节点的电力仿真系统开展模拟实验,实验结果表明,提出的风电收益函数建模、WoLF-PHC改进、风电出力生成等技术是可行的,能有效解决风电参与竞价的现货市场报价问题,并且能够在较少的迭代次数后学习到较优的策略。

稀疏异质多智能体环境下基于强化学习的课程学习框架

现代战争的战场较大且兵种较多,利用多智能体强化学习(MARL)进行战场推演可以加强作战单位之间的协同决策能力,从而提升战斗力。当前MARL在兵棋推演研究和对抗演练中的应用普遍存在两个简化:各个智能体的同质化以及作战单位分布稠密。实际战争场景中并不总是满足这两个设定,可能包含多种异质的智能体以及作战单位分布稀疏。为了探索强化学习在更多场景中的应用,分别就这两方面进行改进研究。首先,设计并实现了多尺度多智能体抢滩登陆环境M2ALE,M2ALE针对上述两个简化设定做了针对性的复杂化,添加了多种异质智能体和作战单位分布稀疏的场景,这两种复杂化设定加剧了多智能体环境的探索困难问题和非平稳性,使用常用的多智能体算法通常难以训练。其次,提出了一种异质多智能体课程学习框架HMACL,用于应对M2ALE环境的难点。HMACL包括3个模块:1)任务生成模块(STG),用于生成源任务以引导智能体训练;2)种类策略提升模块(CPI),针对多智能体系统本身的非平稳性,提出了一种基于智能体种类的参数共享(Class Based Parameter Sharing)策略,实现了异质智能体系统中的参数共享;3)训练模块(Trainer),通过从STG获取源任务,从CPI获取最新的策略,使用任意MARL算法训练当前的最新策略。HMACL可以缓解常用MARL算法在M2ALE环境中的探索难问题和非平稳性问题,引导多智能体系统在M2ALE环境中的学习过程。实验结果表明,使用HMACL使得MARL算法在M2ALE环境下的采样效率和最终性能得到大幅度的提升。

基于强化学习的综合能源系统智能体设计

为满足综合能源系统现货市场的交易需求,文中构建了申报策略强化学习的智能体。该智能体可以完成市场环境与申报策略间的动态匹配,进而灵活选取最适宜的申报策略。同时还根据综合能源系统申报策略的制定需求,设计了智能体的环境变量、动作空间和奖励函数。基于某省电网实际数据构造了智能体的仿真数据集,并从主流技术路径下筛选了8种申报方法构建智能体的动作策略库来进行实验对比。仿真结果表明,动态申报智能体策略的理想度相比其他申报方法高6%以上,且该方法在策略上的准确性良好,决策时间也能够满足现货市场的决策要求。

基于动态选择预测器的深度强化学习投资组合模型

近年来,投资组合管理问题在人工智能领域得到了广泛的研究,但现有的基于深度学习的量化交易方法还存在一些问题。首先,对股票的预测模式单一,通常一个模型只能训练出一个交易专家,交易决策也仅根据模型预测结果作出;其次,模型使用的数据源相对单一,只考虑了股票自身数据,忽略了整个市场风险对股票的影响。针对上述问题,提出了基于动态选择预测器的强化学习模型(DSDRL)。该模型分为3部分,首先提取股票数据的特征并传入多个预测器中,针对不同的投资策略训练多个预测模型,用动态选择器得到当前最优预测结果;其次,利用市场环境评价模块对当前市场风险进行量化,得到合适的投资金额比例;最后,在前两个模块的基础上建立了一种深度强化学习模型模拟真实的交易环境,基于预测的结果和投资金额比例得到实际投资组合策略。文中使用中证500和标普500的日k线数据进行测试验证,结果表明,此模型在夏普率等指标上均优于其他参照模型。