基于内存映射文件的复杂对象快速读取方法

Fast Reading Method of Complex Objects Based on Memory Mapping Files

当前内存数据库(NoSQL)、嵌入式数据库技术在高并发高性能系统中得到了广泛的应用,但对于复杂对象数据的读取效率仍然低下,研究发现主要性能瓶颈有两个:一是内核态与用户态间的内存拷贝,拷贝消耗时间与复杂对象的数据量成线性增长;二是从数据库数据到运行时数据对象的格式转化操作,不但需要开辟新的内存空间存储运行时数据对象,而且还需要解析原始数据并拷贝至新对象之中。为此,提出了一种基于内存映射(memory mapping)文件的复杂对象共享读取方法。借助内存映射文件与自定义内存分配器,实现了结构复杂的C++标准模板库容器对象跨进程无拷贝、无格式转化的共享,有效降低了数据读取延时。通过性能的分析比较表明,与NoSQL内存数据库、嵌入式数据库比,读取性能效率提升10倍以上。再加上底层技术成熟稳定,复用了标准模板库,具有开发成本低、可维护性强、实用性高等优点,因此,适用于高并发高性能的高可用后台服务系统。

The current NoSQL database and embedded database technology have been widely used in high concurrent and high performance systems,but the efficiency of reading complex object data is still low.It is found that there are two main performance bottlenecks.One is the memory copy between kernel mode and user mode,which consumes time linearly with the amount of data of complex objects.The other is format conversion operation from database data to runtime data object,which not only needs to allocate new memory space to store runtime data objects,but also needs to parse the original data and copy them to new objects.Therefore,we propose a method of shared reading of complex objects based on memory mapping file.By means of memory mapping file and custom memory allocator,the sharing of container objects of C++standard template library with complex structure without copy and format conversion across processes is realized to effectively reduce the data reading delay.The performance analysis and comparison show that the efficiency of data reading is more than 10 times higher than that of NoSQL memory database and embedded database.In addition,the bottom technology is mature and stable and the standard template library is reused,with the advantages of low development cost,strong maintainability and high practicability.Therefore,it is suitable for high concurrent and high performance high-availability back-end service system.