In this paper, we explored a load-balancing algorithm in a cluster file system contains two levels of metadata-server, primary-level server quickly distributestasks to second-level servers depending on the closest loa...In this paper, we explored a load-balancing algorithm in a cluster file system contains two levels of metadata-server, primary-level server quickly distributestasks to second-level servers depending on the closest load-balancing information. At the same time, we explored a method which accurately reflect I/O traffic and storage of storage-node： computing the heat-value of file, according to which we realized a more logical storage allocation. According to the experiment result, we conclude that this new algorithm shortens the executing time of tasks and improves the system performance compared with other load algorithm.展开更多
Distributed metadata consistency is one of the critical issues of metadata clusters in distributed file systems. Existing methods to maintain metadata consistency generally need several log forced write operations. Si...Distributed metadata consistency is one of the critical issues of metadata clusters in distributed file systems. Existing methods to maintain metadata consistency generally need several log forced write operations. Since synchronous disk IO is very inefficient, the average response time of metadata operations is greatly increased. In this paper, an asynchronous atomic commit protocol （ACP） named Dual-Log （DL） is presented. It does not need any log forced write operations. Optimizing for distributed metadata operations involving only two metadata servers, DL mutually records the redo log in counterpart metadata servers by transferring through the low latency network. A crashed metadata server can redo the metadata operation with the redundant redo log. Since the latency of the network is much lower than the latency of disk IO, DL can improve the performance of distributed metadata service significantly. The prototype of DL is implemented based on local journal. The performance is tested by comparing with two widely used protocols, EP and S2PC-MP, and the results show that the average response time of distributed metadata operations is reduced by about 40%-60%, and the recovery time is only I second under 10 thousands uncompleted distributed metadata operations.展开更多
Existing in-kernel distributed file systems cannot cope with the higher requirements in well- equipped cluster environments, especially when the system becomes larger and inevitably heterogeneous. TH-CluFS is a clus...Existing in-kernel distributed file systems cannot cope with the higher requirements in well- equipped cluster environments, especially when the system becomes larger and inevitably heterogeneous. TH-CluFS is a cluster file system designed for large heterogeneous systems. TH-CluFS is implemented completely in the user space by emulating the network file system (NFS) V2 server, and is easily portable to other portable operating system interface (POSIX)-compliant platforms with application programming/binary interface API/ABI compliance. In addition, TH-CluFS uses a serverless architecture which flexibly distributes data at file granularity and achieves a consistent file system view from distributed metadata. The global cache makes full use of the aggregated memories and disks in the cluster to optimize system performance. Experimental results suggest that although TH-CluFS is implemented as user-level components, it functions as a portable, single system image, and scalable cluster file system with acceptable performance sacrifices.展开更多
面向企业网或校园网的移动办公与存储的网盘系统有着广泛的市场需求,传统的网盘技术在性能、用户共享、安全性、可扩展性等方面存在诸多缺陷。针对这些不足,本文提出了一种基于云存储的高性能网盘系统架构:采用分布式文件系统MooseFS实...面向企业网或校园网的移动办公与存储的网盘系统有着广泛的市场需求,传统的网盘技术在性能、用户共享、安全性、可扩展性等方面存在诸多缺陷。针对这些不足,本文提出了一种基于云存储的高性能网盘系统架构:采用分布式文件系统MooseFS实现用户数据存储与访问的集群架构;在安全性方面,结合SAMBA实现用户权限管理,用户数据存储支持128 bit AES加密,SSH保证了传输链路的安全;最后,结合用户的实际需求,提供基于Web的访问方式以及客户端的同步盘模式。结果表明,系统在性能、安全性、可扩展性等多方面具有显著优势。展开更多
基金Supported by the Industrialized Foundation ofHebei Province(020501) the Natural Science Foundation of HebeiUniversity(2005Q04)
文摘In this paper, we explored a load-balancing algorithm in a cluster file system contains two levels of metadata-server, primary-level server quickly distributestasks to second-level servers depending on the closest load-balancing information. At the same time, we explored a method which accurately reflect I/O traffic and storage of storage-node： computing the heat-value of file, according to which we realized a more logical storage allocation. According to the experiment result, we conclude that this new algorithm shortens the executing time of tasks and improves the system performance compared with other load algorithm.
基金supported by the National Basic Research 973 Program of China under Grant No.2011CB302304the NationalHigh Technology Research and Development 863 Program of China under Grant Nos.2011AA01A102 and 2013AA013205+1 种基金the StrategicPriority Research Program of the Chinese Academy of Sciences under Grant No.XDA06010401the Chinese Academy of SciencesKey Deployment Project under Grant No.KGZD-EW-103-5(7)
文摘Distributed metadata consistency is one of the critical issues of metadata clusters in distributed file systems. Existing methods to maintain metadata consistency generally need several log forced write operations. Since synchronous disk IO is very inefficient, the average response time of metadata operations is greatly increased. In this paper, an asynchronous atomic commit protocol （ACP） named Dual-Log （DL） is presented. It does not need any log forced write operations. Optimizing for distributed metadata operations involving only two metadata servers, DL mutually records the redo log in counterpart metadata servers by transferring through the low latency network. A crashed metadata server can redo the metadata operation with the redundant redo log. Since the latency of the network is much lower than the latency of disk IO, DL can improve the performance of distributed metadata service significantly. The prototype of DL is implemented based on local journal. The performance is tested by comparing with two widely used protocols, EP and S2PC-MP, and the results show that the average response time of distributed metadata operations is reduced by about 40%-60%, and the recovery time is only I second under 10 thousands uncompleted distributed metadata operations.
基金Supported by the National Natural Science Foundation of China(No. 60073010) and China Grid Project
文摘Existing in-kernel distributed file systems cannot cope with the higher requirements in well- equipped cluster environments, especially when the system becomes larger and inevitably heterogeneous. TH-CluFS is a cluster file system designed for large heterogeneous systems. TH-CluFS is implemented completely in the user space by emulating the network file system (NFS) V2 server, and is easily portable to other portable operating system interface (POSIX)-compliant platforms with application programming/binary interface API/ABI compliance. In addition, TH-CluFS uses a serverless architecture which flexibly distributes data at file granularity and achieves a consistent file system view from distributed metadata. The global cache makes full use of the aggregated memories and disks in the cluster to optimize system performance. Experimental results suggest that although TH-CluFS is implemented as user-level components, it functions as a portable, single system image, and scalable cluster file system with acceptable performance sacrifices.
文摘面向企业网或校园网的移动办公与存储的网盘系统有着广泛的市场需求,传统的网盘技术在性能、用户共享、安全性、可扩展性等方面存在诸多缺陷。针对这些不足,本文提出了一种基于云存储的高性能网盘系统架构:采用分布式文件系统MooseFS实现用户数据存储与访问的集群架构;在安全性方面,结合SAMBA实现用户权限管理,用户数据存储支持128 bit AES加密,SSH保证了传输链路的安全;最后,结合用户的实际需求,提供基于Web的访问方式以及客户端的同步盘模式。结果表明,系统在性能、安全性、可扩展性等多方面具有显著优势。
