基于深度强化学习的高频量化交易策略研究

Research on high-frequency quantitative trading strategy based on deep reinforcement learning

当前国内金融市场的投资交易已从基于传统技术分析等方法的主观交易逐渐转向基于程序化的量化策略交易。股票市场已有大量量化策略的研究工作,但针对期货市场的量化交易策略的研究还不足,已有策略在日内高频交易中的投资回报和风险控制还有待优化。为提升期货高频量化策略的盈利和风控能力,文中设计一种期货交易环境,将1 min时间粒度的高频K线作为环境状态,针对期货交易中持仓状态和交易操作构建相应的动作空间及算法;采用基于LSTM的深度强化学习模型LSTM-Dueling DQN,使其更适用于处理序列输入的状态空间,并显著提升模型的学习速度。对DQN、Double DQN、基于全连接神经网络的Dueling DQN(FF-Dueling DQN)三个基准模型进行实验对比,得到文中构建的交易策略在四个黑色系商品期货交易中累计收益率最高达到43%,年化收益率达到153%,最大回撤控制在10.7%以内。实验结果表明,所提策略在震荡行情和趋势行情中都能实现超出业绩基准的超额收益。

Investment trading in the domestic financial market has gradually shifted from subjective trading based on traditional technical analysis to quantitative trading based on programmability. There has been a considerable amount of research work on quantitative strategies in the current stock market,but there is not enough research on quantitative trading strategies for the futures market,and the investment returns and risk control of existing strategies in intraday high-frequency trading need to be further optimized. In order to improve the profitability and risk control of high-frequency quantitative strategies for futures,a futures trading environment is designed,which takes the 1-min time granularity high-frequency K-bar as the environment state,and corresponding action spaces and algorithms are constructed for position states and trading operations in futures trading. The deep reinforcement learning model LSTM-Dueling DQN based on LSTM is used to make it more suitable for processing the state space of sequence input,and significantly improve the learning speed of the model. Three benchmark models of DQN(Deep QNetwork),Double DQN,and Dueling DQN based on fully connected neural network(FF-Dueling DQN)are compared in this experiments. It is show that the trading strategy constructed in this paper has a maximum cumulative yield of 43%,an annualized yield of 153% and a maximum pullback of 10.7% in the four black commodity futures transactions. The experimental results show that the proposed strategy can achieve excess returns beyond the performance benchmark in both volatile and trend markets.