基于LSTM和多头注意力机制的企业违约预测模型

Enterprise default prediction model based on LSTM and multi-head attention

违约预测是指用企业过去时刻的数据和违约状态预测企业未来的违约概率。违约预测对股票投资、债券投资和银行贷款等具有极为重要的意义。本研究涉及两个科学问题:一是如何使用连续多年的企业数据预测企业违约概率;二是研究输入模型的每个时间窗口对违约预测状态的影响程度。用LSTM网络建立违约预测模型,用连续多年的企业数据预测违约概率,改变了违约预测建模时只用一个时间窗口预测违约概率的现状,并首次将多头注意力机制应用于违约预测模型,探索每个时间窗口对违约预测值的影响程度,避免了现有模型只做预测不揭示时间窗口对违约预测影响程度的弊端。研究表明:一是在违约预测建模时考虑企业数据的时序性更合理且会提升模型预测精度;二是违约预测的最佳时间窗口个数可以是5到10之间的数,总体上时间窗口越多违约预测精度越高;三是本文搭建的违约预测模型框架有效减少了违约预测结果的第2类错误,降低了坏客户被预测为好客户的风险。

Default prediction refers to the use of the data and default state of the company in the past to predict the future probability of default of the company.Default prediction is extremely important for stock investment,bond investment and bank loans.This research involves two scientific issues:one is how to use continuous years of corporate data to predict the default probability,and the other is to study the impact of each time window of the input default prediction model on the default prediction state.In this paper,the default prediction model based on the LSTM network uses continuous years of corporate data to predict the probability of default,which has changed the current situation that only one year of data is used for default prediction modeling.In order to explore the impact of each time window on the default prediction value,this paper first applies the multi head attention mechanism to the default prediction model.This study selects the data of listed companies from 2000 to 2019 as an empirical sample.Each sample of listed companies has 542 indicators,including financial indicators,non-financial indicators and macroeconomic indicators.In order to obtain the most suitable default prediction model for Chinese listed companies based on LSTM and multi-head attention mechanism,this paper has carried out multiple verifications on the key hyper parameters involved in the modeling.Further,in order to better analyze the impact of each structure in the model on the accuracy of default prediction,this paper conducts ablation analysis on the default prediction model built,that is,starting from the structure corresponding to the best performance of the model,and gradually removing the neural network where these structures are located Layer,observe the changes in the accuracy of the algorithm.Finally,in order to study the degree of influence of each time window on the default prediction value,this paper visualizes the output results of the LSTM layer,the attention matrix and the weights of the fully connected layer.The results are as follows:1)It is more reasonable to consider the time series of enterprise data when modeling default prediction,and the use of time series data modeling can help improve the accuracy of the default prediction.2)Through the study of the multi-head attention matrix,it is found that the data of different time windows have different effects on the default prediction results,the same time window has different effects on the default prediction results of different samples,and the sample information captured by different attention heads is different.3)The optimal number of time windows for default prediction can be a number between 5 and 10.Generally speaking,the more time windows,the higher the accuracy of default prediction.4)Model ablation experiments show that the default prediction model built in this paper effectively reduces the second type of error in the default prediction results and reduces the risk of bad customers being predicted as good customers.This paper introduces the LSTM model and the multi-head attention mechanism into the system of the research theory of default prediction of listed companies,to achieve the purpose of predicting the default probability of companies with data for many years,and to a certain extent improve the theory of the research of default prediction of listed companies.The default prediction model established in this paper can provide risk warning information for the market and enterprises,promote the healthy development of the capital market,and prompt enterprises to solve existing problems in a timely manner.The prediction results of the establishment of the default prediction model in the article can also provide a basis for decision-making in the investment activities of institutions or individuals.