HDFS访问中间件的事务设计与实现

Transaction Design and Implementation for HDFS Access Middleware

针对Hadoop分布式文件系统(HDFS)不支持事务导致其故障恢复比较复杂的问题,提出一种支持事务的HDFS访问中间件的设计方案。该方案在对HDFS进行数据访问时还能提供事务支持。首先,分析访问HDFS的应用系统的事务需求,从HDFS事务流程、事务日志和事务恢复3个方面进行描述。其次,设计HDFS访问中间件的事务流程和体系架构。再次,根据设计方案,采用J2EE技术进行系统实现。最后,设计一个实例来测试中间件的设计和实现,通过这个实例说明,该中间件能有效地简化HDFS的数据访问和事务故障恢复的过程。

To settle the problem of Hadoop distributed file system（ HDFS） not supporting transaction and difficult to recover from failure,a HDFS access middleware to support transaction for its application systems was proposed. Accessing to HDFS data,the solution can provide transaction supporting. Firstly,the transaction requirements for those applications systems accessing to HDFS were analyzed. Transaction process,transaction log and transaction recovery were described. Secondly,the transaction process and architecture of the middleware were designed. Thirdly,J2 EE was used to implement the design of the proposed middleware.Finally,a test case was designed to test the proposed middleware design and implementation. It illustrates the proposed middleware can simplify both the process of accessing to HDFS and the process of recovering from failure.