The bane of achieving a scalable distributed file sharing system is the centralized data system and single server oriented file [sharing] system. In this paper, the solution to the problems in a distributed environmen...The bane of achieving a scalable distributed file sharing system is the centralized data system and single server oriented file [sharing] system. In this paper, the solution to the problems in a distributed environment is presented. Thus, inability to upload sizeable files, slow transportation of files, weak security and lack of fault tolerance are the major problems in the existing system. Hence, the utmost need is to build a client-server network that runs on two or more server systems in order to implement scalability, such that when one server is down, the other(s) would still hold up the activities within the network. And to achieve this reliable network environment, LINUX network operating system is recommended among others as a preferred platform for the synchronization of the server systems, such that every user activity like sending of internal memos/mails, publication of academic articles, is replicated;thereby, achieving the proposed result. Hence, Scalable Distributed File Sharing System provides the robustness required to having a reliable intranet.展开更多
Many enterprises and personals are inclining to outsource their data to public clouds, but security and privacy are two critical problems cannot be ignored. The door of cloud provider may be broken, and the data may a...Many enterprises and personals are inclining to outsource their data to public clouds, but security and privacy are two critical problems cannot be ignored. The door of cloud provider may be broken, and the data may also be dug into by providers to find valuable information. In this paper, a secure and efficient storage file (SES FS) system is proposed to distribute files in several clouds and allows users to search the files securely and efficiently. In the proposed system, keywords were transformed into integers and secretly shared in a defined finite field, then the shares were mapped to random numbers in specified random domain in each cloud. Files were encrypted with distinct secret key and scattered within different clouds. Information about keyword/file was secretly shared among cloud providers. Legal users can search in the clouds to find correct encrypted files and reconstruct corresponding secret key. No adversary can find or detect the real file information even they can collude all the servers. Manipulation on shares by one or more clouds can be detected with high probability. The system can also detect malicious servers through introduced virtual points. One interesting property for the scheme is that new keywords can be added easily, which is difficult and usually not efficient for many searchable symmetric encryption systems. Detailed experimental result shows, with tolerable uploading delay, the scheme exhibits excellent performance on data retrieving aspect.展开更多
When examining the file transfer performance in a peer-to-peer file sharing system, a fundamental problem is how to describe the service rate for a file transfer. In this paper, the problem is examined by analyzing th...When examining the file transfer performance in a peer-to-peer file sharing system, a fundamental problem is how to describe the service rate for a file transfer. In this paper, the problem is examined by analyzing the distribution of server-like nodes' upstream-bandwidth among their concurrent transfers. A sufficient condition for the service rate, what a receiver obtains for downloading a file, to asymptotically be uniform is presented. On the aggregate service rate for transferring a file in a system, a sufficient condition for it to asymptotically follow a Zipf distribution is presented. These asymptotic equalities are both in the mean square sense. These analyses and the sufficient conditions provide a mathematic base for modeling file transfer processes in peer-to-peer file sharing systems.展开更多
为了在数据密集型工作流下有效降低缓存碎片整理开销并提高缓存命中率,提出一种持久性分布式文件系统客户端缓存DFS-Cache(Distributed File System Cache)。DFS-Cache基于非易失性内存(NVM)设计实现,能够保证数据的持久性和崩溃一致性...为了在数据密集型工作流下有效降低缓存碎片整理开销并提高缓存命中率,提出一种持久性分布式文件系统客户端缓存DFS-Cache(Distributed File System Cache)。DFS-Cache基于非易失性内存(NVM)设计实现,能够保证数据的持久性和崩溃一致性,并大幅减少冷启动时间。DFS-Cache包括基于虚拟内存重映射的缓存碎片整理机制和基于生存时间(TTL)的缓存空间管理策略。前者基于NVM可被内存控制器直接寻址的特性,动态修改虚拟地址和物理地址之间的映射关系,实现零拷贝的内存碎片整理;后者是一种冷热分离的分组管理策略,借助重映射的缓存碎片整理机制,提升缓存空间的管理效率。实验采用真实的Intel傲腾持久性内存设备,对比商用的分布式文件系统MooseFS和GlusterFS,采用Fio和Filebench等标准测试程序,DFS-Cache最高能提升5.73倍和1.89倍的系统吞吐量。展开更多
视点合成失真算法(Synthesized View Distortion Change,SVDC)作为三维高效视频编码(3D High Efficiency Video Coding,3D-HEVC)中改善深度图编码效率的有效途径,已成为当下三维视频领域的研究前沿之一。基于阵列处理器,利用分布式共享...视点合成失真算法(Synthesized View Distortion Change,SVDC)作为三维高效视频编码(3D High Efficiency Video Coding,3D-HEVC)中改善深度图编码效率的有效途径,已成为当下三维视频领域的研究前沿之一。基于阵列处理器,利用分布式共享存储结构设计并实现一种SVDC算法的并行映射方式,并根据访存特性提出失真值计算优化方案,以像素级误差平方和(Sum of Squared Differences,SSD)计算替代单元级SSD计算。实验表明,相比于HTM平台,算法的平均性能可以提升19.03%,所设计的失真值计算并行方案串/并加速比为2.36,使用像素级SSD计算后相比于优化前平均性能可以提升39.3%。展开更多
The multicore evolution has stimulated renewed interests in scaling up applications on shared-memory multiprocessors,significantly improving the scalability of many applications.But the scalability is limited within a...The multicore evolution has stimulated renewed interests in scaling up applications on shared-memory multiprocessors,significantly improving the scalability of many applications.But the scalability is limited within a single node;therefore programmers still have to redesign applications to scale out over multiple nodes.This paper revisits the design and implementation of distributed shared memory (DSM)as a way to scale out applications optimized for non-uniform memory access (NUMA)architecture over a well-connected cluster.This paper presents MAGI,an efficient DSM system that provides a transparent shared address space with scalable performance on a cluster with fast network interfaces.MAGI is unique in that it presents a NUMA abstraction to fully harness the multicore resources in each node through hierarchical synchronization and memory management.MAGI also exploits the memory access patterns of big-data applications and leverages a set of optimizations for remote direct memory access (RDMA)to reduce the number of page faults and the cost of the coherence protocol.MAGI has been implemented as a user-space library with pthread-compatible interfaces and can run existing multithreaded applications with minimized modifications.We deployed MAGI over an 8-node RDMA-enabled cluster.Experimental evaluation shows that MAGI achieves up to 9.25:4 speedup compared with an unoptimized implementation,leading to a sealable performance for large-scale data-intensive applications.展开更多
文摘The bane of achieving a scalable distributed file sharing system is the centralized data system and single server oriented file [sharing] system. In this paper, the solution to the problems in a distributed environment is presented. Thus, inability to upload sizeable files, slow transportation of files, weak security and lack of fault tolerance are the major problems in the existing system. Hence, the utmost need is to build a client-server network that runs on two or more server systems in order to implement scalability, such that when one server is down, the other(s) would still hold up the activities within the network. And to achieve this reliable network environment, LINUX network operating system is recommended among others as a preferred platform for the synchronization of the server systems, such that every user activity like sending of internal memos/mails, publication of academic articles, is replicated;thereby, achieving the proposed result. Hence, Scalable Distributed File Sharing System provides the robustness required to having a reliable intranet.
基金Demonstration on the Construction of Guangdong Survey and Geomatics Industry Technology Innovation Alliance (2017B090907030)The Demonstration of Big Data Application for Land Resource Management and Service (2015B010110006)+3 种基金Qiong Huang is supported by Guangdong Natural Science Funds for Distinguished Young Scholar (No. 2014A030306021)Guangdong Program for Special Support of Top-notch Young Professionals (No. 2015TQ01X796)Pearl River Nova Program of Guangzhou (No. 201610010037)and the National Natural Science Foundation of China (Nos. 61472146, 61672242).
文摘Many enterprises and personals are inclining to outsource their data to public clouds, but security and privacy are two critical problems cannot be ignored. The door of cloud provider may be broken, and the data may also be dug into by providers to find valuable information. In this paper, a secure and efficient storage file (SES FS) system is proposed to distribute files in several clouds and allows users to search the files securely and efficiently. In the proposed system, keywords were transformed into integers and secretly shared in a defined finite field, then the shares were mapped to random numbers in specified random domain in each cloud. Files were encrypted with distinct secret key and scattered within different clouds. Information about keyword/file was secretly shared among cloud providers. Legal users can search in the clouds to find correct encrypted files and reconstruct corresponding secret key. No adversary can find or detect the real file information even they can collude all the servers. Manipulation on shares by one or more clouds can be detected with high probability. The system can also detect malicious servers through introduced virtual points. One interesting property for the scheme is that new keywords can be added easily, which is difficult and usually not efficient for many searchable symmetric encryption systems. Detailed experimental result shows, with tolerable uploading delay, the scheme exhibits excellent performance on data retrieving aspect.
基金National High Technology Research and Development Program of China (No.2007AA01Z457)Shanghai Science and Technology Development Fundation,China(No.07QA14033)
文摘When examining the file transfer performance in a peer-to-peer file sharing system, a fundamental problem is how to describe the service rate for a file transfer. In this paper, the problem is examined by analyzing the distribution of server-like nodes' upstream-bandwidth among their concurrent transfers. A sufficient condition for the service rate, what a receiver obtains for downloading a file, to asymptotically be uniform is presented. On the aggregate service rate for transferring a file in a system, a sufficient condition for it to asymptotically follow a Zipf distribution is presented. These asymptotic equalities are both in the mean square sense. These analyses and the sufficient conditions provide a mathematic base for modeling file transfer processes in peer-to-peer file sharing systems.
文摘为了在数据密集型工作流下有效降低缓存碎片整理开销并提高缓存命中率,提出一种持久性分布式文件系统客户端缓存DFS-Cache(Distributed File System Cache)。DFS-Cache基于非易失性内存(NVM)设计实现,能够保证数据的持久性和崩溃一致性,并大幅减少冷启动时间。DFS-Cache包括基于虚拟内存重映射的缓存碎片整理机制和基于生存时间(TTL)的缓存空间管理策略。前者基于NVM可被内存控制器直接寻址的特性,动态修改虚拟地址和物理地址之间的映射关系,实现零拷贝的内存碎片整理;后者是一种冷热分离的分组管理策略,借助重映射的缓存碎片整理机制,提升缓存空间的管理效率。实验采用真实的Intel傲腾持久性内存设备,对比商用的分布式文件系统MooseFS和GlusterFS,采用Fio和Filebench等标准测试程序,DFS-Cache最高能提升5.73倍和1.89倍的系统吞吐量。
文摘视点合成失真算法(Synthesized View Distortion Change,SVDC)作为三维高效视频编码(3D High Efficiency Video Coding,3D-HEVC)中改善深度图编码效率的有效途径,已成为当下三维视频领域的研究前沿之一。基于阵列处理器,利用分布式共享存储结构设计并实现一种SVDC算法的并行映射方式,并根据访存特性提出失真值计算优化方案,以像素级误差平方和(Sum of Squared Differences,SSD)计算替代单元级SSD计算。实验表明,相比于HTM平台,算法的平均性能可以提升19.03%,所设计的失真值计算并行方案串/并加速比为2.36,使用像素级SSD计算后相比于优化前平均性能可以提升39.3%。
基金the National Key Research and Development Program of China under Grant No.2016YFBI000500the National Natural Science Foundation of China under Grant No.61572314the National Youth Top-Notch Talent Support Program of China.
文摘The multicore evolution has stimulated renewed interests in scaling up applications on shared-memory multiprocessors,significantly improving the scalability of many applications.But the scalability is limited within a single node;therefore programmers still have to redesign applications to scale out over multiple nodes.This paper revisits the design and implementation of distributed shared memory (DSM)as a way to scale out applications optimized for non-uniform memory access (NUMA)architecture over a well-connected cluster.This paper presents MAGI,an efficient DSM system that provides a transparent shared address space with scalable performance on a cluster with fast network interfaces.MAGI is unique in that it presents a NUMA abstraction to fully harness the multicore resources in each node through hierarchical synchronization and memory management.MAGI also exploits the memory access patterns of big-data applications and leverages a set of optimizations for remote direct memory access (RDMA)to reduce the number of page faults and the cost of the coherence protocol.MAGI has been implemented as a user-space library with pthread-compatible interfaces and can run existing multithreaded applications with minimized modifications.We deployed MAGI over an 8-node RDMA-enabled cluster.Experimental evaluation shows that MAGI achieves up to 9.25:4 speedup compared with an unoptimized implementation,leading to a sealable performance for large-scale data-intensive applications.
