期刊文献+
任意字段
题名或关键词
题名
关键词
文摘
作者
第一作者
机构
刊名
分类号
参考文献
作者简介
基金资助
栏目信息

基于纠删码的HDFS存储方案

HDFS Storage Solutions Based on Erasure Codes
下载PDF
导出
摘要 HDFS文件系统通过多副本备份的方式解决数据损坏或丢失的问题,但是随着存储系统内容增多,在数据量级很大的时候,这种容灾方案消耗的额外存储空间是实际存储内容的数倍,不利于系统资源长期积累.文章提出使用纠删码编/解码文件代替HDFS的副本备份容灾策略,在保证数据安全性的前提下大大提高了存储空间利用率,降低存储额外消耗. Through the multiple-backup strategy HDFS can restore data easily when data is damaged or missed. However, the data stored in system increases all the time. When the data scale has become very big, the strategy will need several times of storage space to store the backup data. This article proposes to use erasure codes to replace the multiple-backup strategy, which can greatly improve the storage efficiency and reduce extra storage expend.
作者 卞艺杰 马瑞敏 李亚冰 吴慧
机构地区 河海大学商学院
出处 《计算机系统应用》 2014年第11期208-213,共6页 Computer Systems & Applications
关键词 纠删码 副本备份容灾 HDFS HDFS erasure code multiple-backup
分类号 TP333 [自动化与计算机技术—计算机系统结构]
  • 相关文献

参考文献6

  • 1Facebook研发闪存解决照片存出问题http://tech.sina.com.cn/digi/dc/2013-02-25/09278086522.shtml. 2013.
  • 2HadoopHDFS.http://hadoop.apache.org/docs/rl.0.4/hdfs_ design,html.
  • 3罗象宏,舒继武.存储系统中的纠删码研究综述[J].计算机研究与发展,2012,49(1):1-11. 被引量：92
  • 4郭春梅,毕学尧.纠删码的分析与研究[J].信息安全与技术,2010,1(7):38-42. 被引量：3
  • 5Lin WK, Chiu DM, Lee YB. Erasure code replicationrevisited. Proc. of the Fourth IEEE International Conferenceon Peer-to-Peer Computing. 2004.
  • 6Bhagwan R, Moore D, Savage S, Voelker GM. Replicationstrategies for highly available peer-to-peer storage. Proc. ofFuDiCo: Future directions in Distributed Computing. 2002.

二级参考文献40

  • 1孟庆春,王晓京.Raptor Code预编码技术研究[J].计算机工程,2007,33(1):1-3. 被引量：11
  • 2Layman P, Varian H R. How much information 2003? [EB/OL]. [2010 10-18]. http://www2, sims. berkeley. edu/research/proiects/how-mueh-info-2003.
  • 3Pinheiro E, Weber W D, Barroso L A. Failure trends in a large disk drive population [C] //Proc of the 5th USENIX Conf on File and Storage Technologies. Berkeley, CA: USENIX Association, 2007 : 17-28.
  • 4Schroeder B, Gibson G A. Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? [C] //Proc of the 5th USENIX Conf on File and Storage Technologies. Berkeley, CA: USENIX Association, 2007: 1-16.
  • 5Bairavasundaram L N, Goodson G R, Pasupathy S, et al. An analysis of latent sector errors in disk drives [C]//Proc of 2007 ACM SIGMETRICS Int Conf on Measurement and Modeling of Computer Systems. New York: ACM, 200: 289-300.
  • 6Hafner J M, Deenadhayalan V, Rao K, et al. Matrix methods for lost data reconstruction in erasure codes [C] // Proc of the 4th USENIX Conf on File and Storage Technologies. Berkeley, CA: USENIX Association, 2005: 183-196.
  • 7Hafner J M, Deenadhayalan V, Kanungo T, et al. Performance metrics for erasure codes in storage systems, RJ 10321 [R]. San Jose, [A] IBM Research, 2004.
  • 8Li M, Shu J, Zheng W. GRID Codes: Strip based erasure codes with high fault tolerance for storage systems [J].ACM Transon Storage, 2009, 4(4): 1-22.
  • 9Blaum M, Brady J, Bruek J, et al. EVENODD: An efficient scheme for tolerating double disk failures in RAID architectures [J].IEEE Trans on Computer, 1995, 44 (2) 192-202.
  • 10Corbett P, English B, Goel A, et al. Row-diagonal redundant for double disk failure correction [C] //Proc of the 3rd USENIX Conf on File and Storage Technologies. Berkeley, CA: USENIX Association, 2004:2-15.

共引文献93

计算机系统应用
2014年 第11期

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部