In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of ...In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.展开更多
Time-sensitive networks(TSNs)support not only traditional best-effort communications but also deterministic communications,which send each packet at a deterministic time so that the data transmissions of networked con...Time-sensitive networks(TSNs)support not only traditional best-effort communications but also deterministic communications,which send each packet at a deterministic time so that the data transmissions of networked control systems can be precisely scheduled to guarantee hard real-time constraints.No-wait scheduling is suitable for such TSNs and generates the schedules of deterministic communications with the minimal network resources so that all of the remaining resources can be used to improve the throughput of best-effort communications.However,due to inappropriate message fragmentation,the realtime performance of no-wait scheduling algorithms is reduced.Therefore,in this paper,joint algorithms of message fragmentation and no-wait scheduling are proposed.First,a specification for the joint problem based on optimization modulo theories is proposed so that off-the-shelf solvers can be used to find optimal solutions.Second,to improve the scalability of our algorithm,the worst-case delay of messages is analyzed,and then,based on the analysis,a heuristic algorithm is proposed to construct low-delay schedules.Finally,we conduct extensive test cases to evaluate our proposed algorithms.The evaluation results indicate that,compared to existing algorithms,the proposed joint algorithm improves schedulability by up to 50%.展开更多
Without considering security, existing message scheduling mechanisms may expose critical messages to malicious threats like confidentiality attacks. Incorporating confidentiality improvement into message scheduling, t...Without considering security, existing message scheduling mechanisms may expose critical messages to malicious threats like confidentiality attacks. Incorporating confidentiality improvement into message scheduling, this paper investigates the problem of scheduling aperiodc messages with time-critical and security-critical requirements. A risk-based security profit model is built to quantify the security quality of messages; and a dynamic programming based approximation algorithm is proposed to schedule aperiodic messages with guaranteed security performance. Experimental results illustrate the efficiency and effectiveness of the proposed algorithm.展开更多
Nowadays, we experience an abundance of Internet of Things middleware solutions that make the sensors and the actuators are able to connect to the Internet. These solutions, referred to as platforms to gain a widespre...Nowadays, we experience an abundance of Internet of Things middleware solutions that make the sensors and the actuators are able to connect to the Internet. These solutions, referred to as platforms to gain a widespread adoption, have to meet the expectations of different players in the IoT ecosystem, including devices [1]. Low cost devices are easily able to connect wirelessly to the Internet, from handhelds to coffee machines, also known as Internet of Things (IoT). This research describes the methodology and the development process of creating an IoT platform. This paper also presents the architecture and implementation for the IoT platform. The goal of this research is to develop an analytics engine which can gather sensor data from different devices and provide the ability to gain meaningful information from IoT data and act on it using machine learning algorithms. The proposed system is introducing the use of a messaging system to improve the overall system performance as well as provide easy scalability.展开更多
One of the most important features of modem minor satellites is to realize autonomous moving. The perfomance of the satellite autonomous computer operating system acting as the control center is of utrnost importance....One of the most important features of modem minor satellites is to realize autonomous moving. The perfomance of the satellite autonomous computer operating system acting as the control center is of utrnost importance. The recent trend in operating system development is adopting microkernel architecture that holds such advantages as microminiaturization, modularity, portability and extendibility. IPC is the key of microkernel design. Message-based IPC mechanism is generally used in existing microkernel Operating system. It is of consistency, safety and reliability.However, it can not provide efficient support for real-time applications in satellite systems and it only applies to loose coupling multi-processor architecture. In this paper, an improvement solution for existing message-based IPC is proposed at first to obtain real-time performance. Then a new IPC mechanism is designed. It particulary applies to shared memory tight coupling multi-processor architecture.展开更多
By thorough research on the prominent periodic and aperiodic scheduling algorithms,anon-line hard real-time scheduler is presented,which is applicable to the scheduling of packets over a link.This scheduler,based on b...By thorough research on the prominent periodic and aperiodic scheduling algorithms,anon-line hard real-time scheduler is presented,which is applicable to the scheduling of packets over a link.This scheduler,based on both Rate Monotonic,pinwheel scheduling algorithm Sr and Polling Serverscheduling algorithms,can rapidly judge the schedulability and then automatically generate a bus tablefor the scheduling algorithm to schedule the packets as the periodic packets.The implementation of thescheduler is simple and easy to use,and it is effective for the utilization of bus link.The orderly executionof the bus table can not only guarantee the performance of the hard real time but also avoid the blockageand interruption of the message transmission.So the scheduler perfectly meets the demand of hard real-time communication system on the field bus domain.展开更多
基金supported by the Research Fund of National Key Laboratory of Computer Architecture under Grant No.CARCH201501the Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing under Grant No.2016A09
文摘In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.
基金partially supported by National Key Research and Development Program of China(2018YFB1700200)National Natural Science Foundation of China(61972389,61903356,61803368,U1908212)+2 种基金Youth Innovation Promotion Association of the Chinese Academy of Sciences,National Science and Technology Major Project(2017ZX02101007-004)Liaoning Provincial Natural Science Foundation of China(2020-MS-034,2019-YQ-09)China Postdoctoral Science Foundation(2019M661156)。
文摘Time-sensitive networks(TSNs)support not only traditional best-effort communications but also deterministic communications,which send each packet at a deterministic time so that the data transmissions of networked control systems can be precisely scheduled to guarantee hard real-time constraints.No-wait scheduling is suitable for such TSNs and generates the schedules of deterministic communications with the minimal network resources so that all of the remaining resources can be used to improve the throughput of best-effort communications.However,due to inappropriate message fragmentation,the realtime performance of no-wait scheduling algorithms is reduced.Therefore,in this paper,joint algorithms of message fragmentation and no-wait scheduling are proposed.First,a specification for the joint problem based on optimization modulo theories is proposed so that off-the-shelf solvers can be used to find optimal solutions.Second,to improve the scalability of our algorithm,the worst-case delay of messages is analyzed,and then,based on the analysis,a heuristic algorithm is proposed to construct low-delay schedules.Finally,we conduct extensive test cases to evaluate our proposed algorithms.The evaluation results indicate that,compared to existing algorithms,the proposed joint algorithm improves schedulability by up to 50%.
基金supported by the National Natural Science Foundation of China (60673142)the National High Technology Research and Development Progrm of China (863 Program) (2006AA01Z1732007AA01Z131)
文摘Without considering security, existing message scheduling mechanisms may expose critical messages to malicious threats like confidentiality attacks. Incorporating confidentiality improvement into message scheduling, this paper investigates the problem of scheduling aperiodc messages with time-critical and security-critical requirements. A risk-based security profit model is built to quantify the security quality of messages; and a dynamic programming based approximation algorithm is proposed to schedule aperiodic messages with guaranteed security performance. Experimental results illustrate the efficiency and effectiveness of the proposed algorithm.
文摘Nowadays, we experience an abundance of Internet of Things middleware solutions that make the sensors and the actuators are able to connect to the Internet. These solutions, referred to as platforms to gain a widespread adoption, have to meet the expectations of different players in the IoT ecosystem, including devices [1]. Low cost devices are easily able to connect wirelessly to the Internet, from handhelds to coffee machines, also known as Internet of Things (IoT). This research describes the methodology and the development process of creating an IoT platform. This paper also presents the architecture and implementation for the IoT platform. The goal of this research is to develop an analytics engine which can gather sensor data from different devices and provide the ability to gain meaningful information from IoT data and act on it using machine learning algorithms. The proposed system is introducing the use of a messaging system to improve the overall system performance as well as provide easy scalability.
文摘One of the most important features of modem minor satellites is to realize autonomous moving. The perfomance of the satellite autonomous computer operating system acting as the control center is of utrnost importance. The recent trend in operating system development is adopting microkernel architecture that holds such advantages as microminiaturization, modularity, portability and extendibility. IPC is the key of microkernel design. Message-based IPC mechanism is generally used in existing microkernel Operating system. It is of consistency, safety and reliability.However, it can not provide efficient support for real-time applications in satellite systems and it only applies to loose coupling multi-processor architecture. In this paper, an improvement solution for existing message-based IPC is proposed at first to obtain real-time performance. Then a new IPC mechanism is designed. It particulary applies to shared memory tight coupling multi-processor architecture.
基金Supported by the Emphases Science and Technology Project Foundation of Sichuan Province(NO.02GG006-037)
文摘By thorough research on the prominent periodic and aperiodic scheduling algorithms,anon-line hard real-time scheduler is presented,which is applicable to the scheduling of packets over a link.This scheduler,based on both Rate Monotonic,pinwheel scheduling algorithm Sr and Polling Serverscheduling algorithms,can rapidly judge the schedulability and then automatically generate a bus tablefor the scheduling algorithm to schedule the packets as the periodic packets.The implementation of thescheduler is simple and easy to use,and it is effective for the utilization of bus link.The orderly executionof the bus table can not only guarantee the performance of the hard real time but also avoid the blockageand interruption of the message transmission.So the scheduler perfectly meets the demand of hard real-time communication system on the field bus domain.
文摘消息队列作为一种常见的数据结构,在应用程序和软件操作系统中得到了广泛应用,而随着处理器的不断迭代,传统的锁同步消息队列逐渐无法发挥多核处理器的性能。因此,研究人员尝试引入布谷鸟过滤器(Cuckoo Filter,CCF)和DQueue技术对消息队列进行优化升级,创建云开发消息队列(Cloud Development Message Queue,CDMQ)。基于上述两种技术同时生成多个队列写入数据,操作者利用CCF对数据进行过滤检验,并读取相关信息。通过对CDMQ的性能测试,研究人员证明该消息队列结构具有伸缩性强和吞吐量高等优势,对提高消息队列性能具有重要意义。
