In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of ...In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.展开更多
As known to all that Hénon chaotic system is not appropriate for generating the key-streams because it has non-uniformly distributed output signal, a new key-stream generation scheme based on Hénon chaotic s...As known to all that Hénon chaotic system is not appropriate for generating the key-streams because it has non-uniformly distributed output signal, a new key-stream generation scheme based on Hénon chaotic system is presented. In order to get the key-streams with good statistics and long enough cycle length, a specific method for dividing the Hénon attractor into numerous non-overlapping sub-regions, and a new one-to-one mapping strategy between the divided sub-regions and elements of dynamically generated matrix consisting of 0's and 1's are proposed. Experimental results demonstrate that the generated key-streams are with long enough cycle length and very sensitive to the initial values and secret keys. For example, key-streams with the cycle length of 10 32 can easily be obtained. Moreover, even if the fluctuation to the initial values or secret keys is as small as 10 14 , uncorrelated key-streams will be generated. Experimental results also demonstrate that the generated key-streams have good randomness and they can pass all the standard criteria specified in FIPS PUB 140-2 with no less than 98%.展开更多
In the typical streaming media system, the streaming media server is system bottleneck with the expansion of Internet subscribers. This paper proposes an innovational high performance streaming media system architectu...In the typical streaming media system, the streaming media server is system bottleneck with the expansion of Internet subscribers. This paper proposes an innovational high performance streaming media system architecture (HPSMS) based on the logical separation of streaming media transport protocol. The system avoids expensive store-and-forward data copies between streaming media server and storage devices, improves the system performance greatly. The system bandwidth continuously increases with the expansion of storage system capacity is the highlight. The performance of the proposed HPSMS is evaluated through a practical prototype implementation.展开更多
A mobile transmission strategy, PMPatching (Proxy-based Mobile Patching) transmission strat-egy is proposed, it applies to the proxy-based mobile streaming media system in Wideband Code Division Multiple Access (WCDMA...A mobile transmission strategy, PMPatching (Proxy-based Mobile Patching) transmission strat-egy is proposed, it applies to the proxy-based mobile streaming media system in Wideband Code Division Multiple Access (WCDMA) network. Performance of the whole system can be improved by using patching stream to transmit anterior part of the suffix that had been played back, and by batching all the demands for the suffix arrived in prefix period and patching stream transmission threshold period. Experimental results show that this strategy can efficiently reduce average network transmission cost and number of channels consumed in central streaming media server.展开更多
The stream and rain data acquisition system presented in this paper makes the mapping of hydro potentials in the region or of the country economically and practically possible. Moreover, it can also serve as a flood w...The stream and rain data acquisition system presented in this paper makes the mapping of hydro potentials in the region or of the country economically and practically possible. Moreover, it can also serve as a flood warning system.展开更多
The rapid growth of streaming media applications on the Internet is proposing higher requirements on energy consumption and I/O performance of the storage systems.However,the optimized I/O requests from different init...The rapid growth of streaming media applications on the Internet is proposing higher requirements on energy consumption and I/O performance of the storage systems.However,the optimized I/O requests from different initiators will be mixed disorderly when they are reaching the storage system concurrently,which leads to increasing energy consumption.This paper proposes an energy-saving scheduling scheme based on I/O Stream(ES-IOS).The ES-IOS scheme can take the advantage of the I/O characteristics of streaming media and reorganize the mixed and disordered I/O requests into "streams".Technically,The ES-IOS scheme includes two main points,a priority-based weighted stream scheduling algorithm(PWSS) and a regression-fitting-based popularity prediction algorithm(RFPP).The PWSS algorithm can schedule the I/O streams in weighted queue based on priority to limit energy consumption.The priority of each stream is determined by its popularity.According to the I/O access records over a period,the RFPP algorithm can predict the popularity of each stream via regression fitting.Based on the popularities,the PWSS algorithm assigns more continuous service time to the hot streams and reversely less service time to the cold ones.Trace-driven experiments show that the ES-IOS scheme can reduce the energy consumption by 38%and enhance the I/O throughput by 27%approximately.展开更多
基金supported by the Research Fund of National Key Laboratory of Computer Architecture under Grant No.CARCH201501the Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing under Grant No.2016A09
文摘In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.
基金Foundation item： Proj ects（61172184, 61173147） supported by the National Natural Science Foundation of China Project（12JJ6062） supported by Natural Science Foundation of Hunan Province, China+1 种基金 Project（121gpy31） supported by the Fundamental Research Funds for the Central Universities of China Project supported by the State Key Laboratory of Information Security （Institute of Software, Chinese Academy of Sciences）, China
文摘As known to all that Hénon chaotic system is not appropriate for generating the key-streams because it has non-uniformly distributed output signal, a new key-stream generation scheme based on Hénon chaotic system is presented. In order to get the key-streams with good statistics and long enough cycle length, a specific method for dividing the Hénon attractor into numerous non-overlapping sub-regions, and a new one-to-one mapping strategy between the divided sub-regions and elements of dynamically generated matrix consisting of 0's and 1's are proposed. Experimental results demonstrate that the generated key-streams are with long enough cycle length and very sensitive to the initial values and secret keys. For example, key-streams with the cycle length of 10 32 can easily be obtained. Moreover, even if the fluctuation to the initial values or secret keys is as small as 10 14 , uncorrelated key-streams will be generated. Experimental results also demonstrate that the generated key-streams have good randomness and they can pass all the standard criteria specified in FIPS PUB 140-2 with no less than 98%.
文摘In the typical streaming media system, the streaming media server is system bottleneck with the expansion of Internet subscribers. This paper proposes an innovational high performance streaming media system architecture (HPSMS) based on the logical separation of streaming media transport protocol. The system avoids expensive store-and-forward data copies between streaming media server and storage devices, improves the system performance greatly. The system bandwidth continuously increases with the expansion of storage system capacity is the highlight. The performance of the proposed HPSMS is evaluated through a practical prototype implementation.
基金Supported by: (1) Specialized Research Fund for the Doctoral Program of Higher Education (No. 20030013006) (2) National Specialized R&D Project for the Product of Mobile Communications (Develop-ment and Application of Next Generation Mobile Intel-ligent Network)+3 种基金 (3) Development Fund Key Project for Electronic and Information Industry (Core Service Plat-form for Next Generation Network) (4) Development Fund Project for Electronic and Information Industry (Value-added Service Platform and Application System for Mobile Communications) (5) Development Fund Project for Electronic and Information Industry (Con-tent-based Integrated Charging Platform for Telecom-munication Networks) (6) National Specific Project for Hi-tech Industrialization and Information Equipments (Mobile Intelligent Network Supporting Value-added Data Services).
文摘A mobile transmission strategy, PMPatching (Proxy-based Mobile Patching) transmission strat-egy is proposed, it applies to the proxy-based mobile streaming media system in Wideband Code Division Multiple Access (WCDMA) network. Performance of the whole system can be improved by using patching stream to transmit anterior part of the suffix that had been played back, and by batching all the demands for the suffix arrived in prefix period and patching stream transmission threshold period. Experimental results show that this strategy can efficiently reduce average network transmission cost and number of channels consumed in central streaming media server.
基金Supported by the State Key Program of National Natural Science Foundation of China （No. 60832009）, Natural Science Foundation of Bcijing （No. 4102044）, Innovative Project for Young Researchers in Central Higher Education Institutions, China （No. 2009RC0119） and New Generation of Broadband Wireless Mobile Communication Networks of Major Projects of National Science and Technology （No.2009ZX03003-003-01）.
文摘The stream and rain data acquisition system presented in this paper makes the mapping of hydro potentials in the region or of the country economically and practically possible. Moreover, it can also serve as a flood warning system.
基金Supported by the National High Technology Research and Development Programme of China(No.2011AA01A102)
文摘The rapid growth of streaming media applications on the Internet is proposing higher requirements on energy consumption and I/O performance of the storage systems.However,the optimized I/O requests from different initiators will be mixed disorderly when they are reaching the storage system concurrently,which leads to increasing energy consumption.This paper proposes an energy-saving scheduling scheme based on I/O Stream(ES-IOS).The ES-IOS scheme can take the advantage of the I/O characteristics of streaming media and reorganize the mixed and disordered I/O requests into "streams".Technically,The ES-IOS scheme includes two main points,a priority-based weighted stream scheduling algorithm(PWSS) and a regression-fitting-based popularity prediction algorithm(RFPP).The PWSS algorithm can schedule the I/O streams in weighted queue based on priority to limit energy consumption.The priority of each stream is determined by its popularity.According to the I/O access records over a period,the RFPP algorithm can predict the popularity of each stream via regression fitting.Based on the popularities,the PWSS algorithm assigns more continuous service time to the hot streams and reversely less service time to the cold ones.Trace-driven experiments show that the ES-IOS scheme can reduce the energy consumption by 38%and enhance the I/O throughput by 27%approximately.
