In consultative committee for space data systems（CCSDS） file delivery protocol（CFDP） recommendation of reliable transmission,there are no detail transmission procedure and delay calculation of prompted negative ac...In consultative committee for space data systems（CCSDS） file delivery protocol（CFDP） recommendation of reliable transmission,there are no detail transmission procedure and delay calculation of prompted negative acknowledge and asynchronous negative acknowledge models.CFDP is designed to provide data and storage management,story and forward,custody transfer and reliable end-to-end delivery over deep space characterized by huge latency,intermittent link,asymmetric bandwidth and big bit error rate（BER）.Four reliable transmission models are analyzed and an expected file-delivery time is calculated with different trans-mission rates,numbers and sizes of packet data units,BERs and frequencies of external events,etc.By comparison of four CFDP models,the requirement of BER for typical missions in deep space is obtained and rules of choosing CFDP models under different uplink state informations are given,which provides references for protocol models selection,utilization and modification.展开更多
Satellite networking communications in navigation satellite system and spacebased deep space exploration have the features of a long delay and high bit error rate （BER）. Through analyzing the advantages and disadvan...Satellite networking communications in navigation satellite system and spacebased deep space exploration have the features of a long delay and high bit error rate （BER）. Through analyzing the advantages and disadvantages of the Consulta tive Committee for the Space Data System （CCSDS） file delivery protocol （CFDP）, a new improved repeated sending file delivery protocol （RSFDP） based on the adaptive repeated sending is put forward to build an efficient and reliable file transmission. According to the estimation of the BER of the transmission link, RSFDP repeatedly sends the lost protocol data units （PDUs） at the stage of the retransmission to improve the success rate and reduce time of the retransmission. Theoretical analyses and results of the Opnet simulation indicate that the performance of RSFDP has significant improvement gains over CFDP in the link with a long delay and high BER. The realizing results based on the space borne filed programmable gate array （FPGA） platform show the applicability of the proposed algorithm.展开更多
以深度学习为代表的数据分析应用越来越多依赖分布式文件系统存储管理大规模数据集.为了增强数据访问的兼容性,现有分布式文件存储系统通常需提供标准POSIX接口,以支持深度学习等应用的无缝对接.然而,以内核模块形态开发提供POSIX接口...以深度学习为代表的数据分析应用越来越多依赖分布式文件系统存储管理大规模数据集.为了增强数据访问的兼容性,现有分布式文件存储系统通常需提供标准POSIX接口,以支持深度学习等应用的无缝对接.然而,以内核模块形态开发提供POSIX接口的文件系统非常复杂耗时.近年来,用户态文件系统(Filesystem in Userspace,FUSE)框架大幅简化了文件系统的开发工作,已被Alluxio和Ceph等诸多知名分布式文件系统使用.目前常用的用户态FUSE库libfuse仅提供C语言编程接口,但现有大数据分布式文件系统基本都是基于Java语言开发的(例如HDFS和Alluxio等),为了使基于Java语言开发的分布式文件系统可以对接C语言开发的FUSE库,需采用跨语言FUSE框架作为中介.跨语言FUSE框架利用跨编程语言的函数回调机制,使操作系统FUSE库的C语言函数可以跨语言的调用分布式文件系统提供的Java语言编程接口,从而为大数据分布式文件系统提供标准POSIX接口的访问能力.但在数据密集型应用中,现有跨语言FUSE框架的执行效率低,导致数据密集型作业(深度学习、大数据分析等)中数据I/O耗时占据了显著的性能开销,成为新的潜在性能瓶颈.针对此问题,本文首先评估分析了重要且广为使用的跨语言FUSE框架JNR-FUSE的性能,发现并定位其在高并发和小文件场景下存在的性能瓶颈;接着从多方面剖析性能瓶颈根因,进而总结出高效跨语言FUSE框架的性能优化方向,并面向Java语言设计实现了跨语言FUSE框架JNI-FUSE.JNI-FUSE利用延迟分离和元信息缓存等优化技术降低跨语言函数回调开销,从而提升跨语言FUSE框架的性能.实验结果表明,对比当前性能最好的Java FUSE框架JNR-FUSE,本文提出的JNI-FUSE带来了1.15~6.04倍的FUSE框架性能提升和1.90~2.71倍的文件系统端到端性能提升,并为上层深度学习训练任务带来了1.06~1.73倍的训练加速.本文设计提出的JNI-FUSE(Java Native Interface-Filesystem in User SpacE)因性能优势,已被知名开源分布式文件系统Alluxio官方接受集成.展开更多
Deep space communication has its own features such as long propagation delays,heavy noise,asymmetric link rates,and intermittent connectivity in space,therefore TCP/IP protocol cannot perform as well as it does in ter...Deep space communication has its own features such as long propagation delays,heavy noise,asymmetric link rates,and intermittent connectivity in space,therefore TCP/IP protocol cannot perform as well as it does in terrestrial communications.Accordingly,the Consultative Committee for Space Data Systems(CCSDS) developed CCSDS File Delivery Protocol(CFDP),which sets standards of efficient file delivery service capable of transferring files to and from mass memory located in the space segment.In CFDP,four optional acknowledge modes are supported to make the communication more reliable.In this paper,we gave a general introduction of typical communication process in CFDP and analysis of its four Negative Acknowledgement(NAK) modes on the respect of file delivery delay and times of retransmission.We found out that despite the shortest file delivery delay,immediate NAK mode suffers from the problem that frequent retransmission may probably lead to network congestion.Thus,we proposed a new mode,the error counter-based NAK mode.By simulation of the case focused on the link between a deep space probe on Mars and a ter-restrial station on Earth,we concluded that error counter-based NAK mode has successfully reduced the retransmission times at negligible cost of certain amount of file delivery delay.展开更多
面向企业网或校园网的移动办公与存储的网盘系统有着广泛的市场需求,传统的网盘技术在性能、用户共享、安全性、可扩展性等方面存在诸多缺陷。针对这些不足,本文提出了一种基于云存储的高性能网盘系统架构:采用分布式文件系统MooseFS实...面向企业网或校园网的移动办公与存储的网盘系统有着广泛的市场需求,传统的网盘技术在性能、用户共享、安全性、可扩展性等方面存在诸多缺陷。针对这些不足,本文提出了一种基于云存储的高性能网盘系统架构:采用分布式文件系统MooseFS实现用户数据存储与访问的集群架构;在安全性方面,结合SAMBA实现用户权限管理,用户数据存储支持128 bit AES加密,SSH保证了传输链路的安全;最后,结合用户的实际需求,提供基于Web的访问方式以及客户端的同步盘模式。结果表明,系统在性能、安全性、可扩展性等多方面具有显著优势。展开更多
基金supported by the National Natural Science Fandation of China (6067208960772075)
文摘In consultative committee for space data systems（CCSDS） file delivery protocol（CFDP） recommendation of reliable transmission,there are no detail transmission procedure and delay calculation of prompted negative acknowledge and asynchronous negative acknowledge models.CFDP is designed to provide data and storage management,story and forward,custody transfer and reliable end-to-end delivery over deep space characterized by huge latency,intermittent link,asymmetric bandwidth and big bit error rate（BER）.Four reliable transmission models are analyzed and an expected file-delivery time is calculated with different trans-mission rates,numbers and sizes of packet data units,BERs and frequencies of external events,etc.By comparison of four CFDP models,the requirement of BER for typical missions in deep space is obtained and rules of choosing CFDP models under different uplink state informations are given,which provides references for protocol models selection,utilization and modification.
基金supported by the National High Technology Research and Development Program of China (863 Program) (2011AA1569)
文摘Satellite networking communications in navigation satellite system and spacebased deep space exploration have the features of a long delay and high bit error rate （BER）. Through analyzing the advantages and disadvantages of the Consulta tive Committee for the Space Data System （CCSDS） file delivery protocol （CFDP）, a new improved repeated sending file delivery protocol （RSFDP） based on the adaptive repeated sending is put forward to build an efficient and reliable file transmission. According to the estimation of the BER of the transmission link, RSFDP repeatedly sends the lost protocol data units （PDUs） at the stage of the retransmission to improve the success rate and reduce time of the retransmission. Theoretical analyses and results of the Opnet simulation indicate that the performance of RSFDP has significant improvement gains over CFDP in the link with a long delay and high BER. The realizing results based on the space borne filed programmable gate array （FPGA） platform show the applicability of the proposed algorithm.
文摘以深度学习为代表的数据分析应用越来越多依赖分布式文件系统存储管理大规模数据集.为了增强数据访问的兼容性,现有分布式文件存储系统通常需提供标准POSIX接口,以支持深度学习等应用的无缝对接.然而,以内核模块形态开发提供POSIX接口的文件系统非常复杂耗时.近年来,用户态文件系统(Filesystem in Userspace,FUSE)框架大幅简化了文件系统的开发工作,已被Alluxio和Ceph等诸多知名分布式文件系统使用.目前常用的用户态FUSE库libfuse仅提供C语言编程接口,但现有大数据分布式文件系统基本都是基于Java语言开发的(例如HDFS和Alluxio等),为了使基于Java语言开发的分布式文件系统可以对接C语言开发的FUSE库,需采用跨语言FUSE框架作为中介.跨语言FUSE框架利用跨编程语言的函数回调机制,使操作系统FUSE库的C语言函数可以跨语言的调用分布式文件系统提供的Java语言编程接口,从而为大数据分布式文件系统提供标准POSIX接口的访问能力.但在数据密集型应用中,现有跨语言FUSE框架的执行效率低,导致数据密集型作业(深度学习、大数据分析等)中数据I/O耗时占据了显著的性能开销,成为新的潜在性能瓶颈.针对此问题,本文首先评估分析了重要且广为使用的跨语言FUSE框架JNR-FUSE的性能,发现并定位其在高并发和小文件场景下存在的性能瓶颈;接着从多方面剖析性能瓶颈根因,进而总结出高效跨语言FUSE框架的性能优化方向,并面向Java语言设计实现了跨语言FUSE框架JNI-FUSE.JNI-FUSE利用延迟分离和元信息缓存等优化技术降低跨语言函数回调开销,从而提升跨语言FUSE框架的性能.实验结果表明,对比当前性能最好的Java FUSE框架JNR-FUSE,本文提出的JNI-FUSE带来了1.15~6.04倍的FUSE框架性能提升和1.90~2.71倍的文件系统端到端性能提升,并为上层深度学习训练任务带来了1.06~1.73倍的训练加速.本文设计提出的JNI-FUSE(Java Native Interface-Filesystem in User SpacE)因性能优势,已被知名开源分布式文件系统Alluxio官方接受集成.
文摘Deep space communication has its own features such as long propagation delays,heavy noise,asymmetric link rates,and intermittent connectivity in space,therefore TCP/IP protocol cannot perform as well as it does in terrestrial communications.Accordingly,the Consultative Committee for Space Data Systems(CCSDS) developed CCSDS File Delivery Protocol(CFDP),which sets standards of efficient file delivery service capable of transferring files to and from mass memory located in the space segment.In CFDP,four optional acknowledge modes are supported to make the communication more reliable.In this paper,we gave a general introduction of typical communication process in CFDP and analysis of its four Negative Acknowledgement(NAK) modes on the respect of file delivery delay and times of retransmission.We found out that despite the shortest file delivery delay,immediate NAK mode suffers from the problem that frequent retransmission may probably lead to network congestion.Thus,we proposed a new mode,the error counter-based NAK mode.By simulation of the case focused on the link between a deep space probe on Mars and a ter-restrial station on Earth,we concluded that error counter-based NAK mode has successfully reduced the retransmission times at negligible cost of certain amount of file delivery delay.
文摘面向企业网或校园网的移动办公与存储的网盘系统有着广泛的市场需求,传统的网盘技术在性能、用户共享、安全性、可扩展性等方面存在诸多缺陷。针对这些不足,本文提出了一种基于云存储的高性能网盘系统架构:采用分布式文件系统MooseFS实现用户数据存储与访问的集群架构;在安全性方面,结合SAMBA实现用户权限管理,用户数据存储支持128 bit AES加密,SSH保证了传输链路的安全;最后,结合用户的实际需求,提供基于Web的访问方式以及客户端的同步盘模式。结果表明,系统在性能、安全性、可扩展性等多方面具有显著优势。
