建立内存机制保证海量数据实时性系统的运行

Establish Memory Mechanism to Guarantee the Operation of the Real-time System with Massive Data

在实时性很强且数据量很大的系统中提高处理数据效率,是近年来很多人一直在探讨的一个话题。把研究对象建立在一个大型证券交易系统的数据库中,像这类普遍使用关系数据库的系统中,诸如实时数据处理和更新等事务往往通过硬盘操作耗费很多时间,我们的目的就是运用内存和实时数据库的一些主要思想和原理,研究一套合适和高效的内存(主存)机制,使得在不破坏原有数据库结构和不利用大成本建立实时数据库的基础上,大大提高数据存储的速度,进而提高交易过程的效率。结合一个实际项目,遵循软件工程的基本流程,在需求分析阶段获取了业务上需要达到的数据处理要求并进行了进一步的可行性分析,同时参阅了内存机制相关的资料,在设计阶段根据交易系统数据库的结构提出了适合交易数据库的内存表框架,在全表中选出了需要放在内存空间中的表,设置了优化的调用参数,并且利用改进的红黑树设计了一套索引算法,以及结合实时数据库的做法建立了一套事务处理机制和通信机制。在随后进行的一些效率和强度测试中,证明了实际应用中能使存储的效率提高一个数量级,并使整体的交易处理效率提高3倍以上。最后我们还考虑了维护方面的问题,保证了系统能够长时间正确运行。

It＇s really became a hot topic to increase the efficiency in a database system with large amount of data and high intensity of incoming requests. This study is also based in a large stock trade system using relational database which may be stuck by some disk writing operation like updates or commitment in some transactions. The goal is to pick up some principles of memory and real-time database building up a memory （ main memory） based mechanism for this system to increase the efficiency of data processing and operations for customers, without corrupting the construction and throwing out bulks of money to buy a real-time database. The study is worked with an real project, and confirm ed to a basic flow of software engineering. In requirements analysis step, the request and the rate of the process of data are gotton, analyzed the possibility and read lots of materials. In design step, the table which to be put into main memory referenced to the current system is selected out, and optimized configuration parameters. Meanwhile some indexes using RBTrees, and a rule for transaction and transformation are created. In the rate and intensity test afterward, time times＇ upgrade in writing processes, and at least 3 times for the whole trade processes are trusted. At last had thought about the problem in maintenance steo in order to draw a long live to this system.