Development of A New Lime Adding Technology in Two-Stream Digestion Process of Diasporic Bauxite

Indirect heating and intensified digestion technology can be applied to reduce greatly the energy consumption in Bayer process of diasporic bauxite. A great advantage of two-stream process is to avoid or reduce efficiently serious scaling problem of bauxite slurry on indirect heating surface, which certainly happens in the single stream process and brings about great troubles to the indirect heating. As a result of a great number of experiments and the theoretical analysis, a new lime adding technology for the two-stream digestion process is developed in this paper that lime is added into spent liquor stream instead of bauxite slurry, which is more suitable to the two-stream process of diasporic bauxite. The influences of the new lime addition technology on preheating and digestion process were discussed. It was deduced that the new technology can be used efficiently in the two-stream process of non-diasporic bauxite.

Improvement Design for Distributed Real-Time Stream Processing Systems

In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.

Strategy for Data Stream Processing Based on Measurement Metadata: An Outpatient Monitoring Scenario

In this work we discuss SDSPbMM, an integrated Strategy for Data Stream Processing based on Measurement Metadata, applied to an outpatient monitoring scenario. The measures associated to the attributes of the patient (entity) under monitoring, come from heterogeneous data sources as data streams, together with metadata associated with the formal definition of a measurement and evaluation project. Such metadata supports the patient analysis and monitoring in a more consistent way, facilitating for instance: i) The early detection of problems typical of data such as missing values, outliers, among others;and ii) The risk anticipation by means of on-line classification models adapted to the patient. We also performed a simulation using a prototype developed for outpatient monitoring, in order to analyze empirically processing times and variable scalability, which shed light on the feasibility of applying the prototype to real situations. In addition, we analyze statistically the results of the simulation, in order to detect the components which incorporate more variability to the system.

Quality Stability of Multi-Station Assembly Process Based on Variation Stream

To analyze the physical structure of assembly process and assure product quality, the quality stability of multi-station assembly process was investigated. First, the assembly process was modeled as a one-dimensional discrete variant system by state space equation based on variation stream. Then, the criterion to judge whether the process is stable or not and the index, stability degree, to show the level of stability were proposed by analyzing the bounded-input bounded-output (BIBO) stability of system. Finally, a simulated example of a sheet metal assembly process with three stations, was provided to verify the effectiveness of the proposed method.

FPGA-Based Stream Processing for Frequent Itemset Mining with Incremental Multiple Hashes

With the advent of the IoT era, the amount of real-time data that is processed in data centers has increased explosively. As a result, stream mining, extracting useful knowledge from a huge amount of data in real time, is attracting more and more attention. It is said, however, that real- time stream processing will become more difficult in the near future, because the performance of processing applications continues to increase at a rate of 10% - 15% each year, while the amount of data to be processed is increasing exponentially. In this study, we focused on identifying a promising stream mining algorithm, specifically a Frequent Itemset Mining (FIsM) algorithm, then we improved its performance using an FPGA. FIsM algorithms are important and are basic data- mining techniques used to discover association rules from transactional databases. We improved on an approximate FIsM algorithm proposed recently so that it would fit onto hardware architecture efficiently. We then ran experiments on an FPGA. As a result, we have been able to achieve a speed 400% faster than the original algorithm implemented on a CPU. Moreover, our FPGA prototype showed a 20 times speed improvement compared to the CPU version.

基于Spark Streaming的气象自动站实时流处理与存储系统

在当前大数据技术蓬勃发展的时代,人们对气象数据的实时处理、数据质量、数据存储及大规模查询等要求也越来越高。针对现有气象自动站数据业务落地环节多,任务处理耦合紧但系统部署分散等问题,文中基于Spark Streaming的流式计算框架,研究使用Flume解析收集自动站原始数据,在Spark Streaming中设计融入自动站数据质控算法,最终通过对分布式数据库存储的表设计,使气象自动站数据具备高效率、高质量、高可靠的应用服务能力。性能测试结果表明,基于Spark Streaming的气象自动站数据实时流处理与存储系统,数据从文件采集、解码、流处理至入库的全流程能够在秒级完成,TB级数据查询响应为毫秒级,加权查询为秒级,完全满足自动站数据业务应用需求,从而为进一步提高气象自动站数据质量与服务水平提供基础支撑。

地面自动气象站数据流式处理设计与实现

针对观测密度和频次日益增加的海量地面自动气象站数据,在气象大数据云平台(天擎)中设计了基于Storm的实时流式处理,利用大规模并行处理的优势提高地面自动气象站数据的处理时效。在流式处理中,设计处理拓扑直接解码标准格式的数据消息;消息确认采用手工确认的方式,将数据解码组件锚定数据接入组件,实现每条数据的可靠处理;数据解码时进行字节校验和时间检查等,过滤异常数据;应用批量加定时的发送策略,解决海量监控信息发送气象综合业务实时监控系统(天镜)的问题;集群部署时保留部分剩余资源,有效应对单节点异常。应用效果表明:国家气象站小时数据的服务时效由全国综合气象信息共享系统(CIMISS)的175 s提高至天擎的78 s,约6×10^(4)个区域气象站小时数据的服务时效由CIMISS的5 min提高至天警的2 min,实况分析系统将数据源切换至天擎后,相同时间检索可获取的站点数量较CIMISS增加1倍。2021年12月基于Storm的流式处理与天擎一同在国省业务化运行,实现了长期稳定运行,为MICAPS4、SWAN2.0、实况分析系统等用户提供高效稳定的地面自动气象站数据。

Determination of Di-n-Butyl Phosphate in Organic Streams of FBTR Mixed Carbide Fuel Reprocessing Solution by Gas Chromatographic Technique

The present work describes the amount of Di-n- butyl phosphate (DBP) produced when PUREX solvent (30%tri-n-butyl phosphate (TBP) mixed with 70% hydrocarbon diluent) is exposed to intensive radiolytic and chemical at- tack during the separation of uranium and plutonium from fission products of FBTR mixed carbide fuel reprocessing solution. DBP is the major degradation product of Tri-n-butyl phosphate (TBP). Amount of DBP formed in the lean organic streams of different fuel burn-up FBTR carbide fuel reprocessing solutions were analyzed by Gas Chromatographic technique. The method is based on the preparation of diazo methane and conversion of non-volatile Di-n-butyl phosphate in to volatile and stable derivatives by the action of diazomethane and then determined by Gas Chromatography (GC). A calibration graph was made for DBP over a concentration in the range from 200 to 1800 ppm with correlation coefficient of 0.99587 and RSD 1.2%. The degraded 30% TBP-NPH solvent loaded with heavy metal ions like uranium was analyzed after repeated use and results are compared with standard ion chromatographic technique. A column comparison study to select of proper gas chromatographic column for the separation of DBP from other components in a single aliquot of injection is also examined.

基于Spark Streaming的实时数据分析系统及其应用

为了实现对实时网络数据流的快速分析,设计一种分布式实时数据流分析系统(DRDAS),能有效解决并发访问数据流的收集、存储和实时分析问题,为大数据环境的网络安全检测提供了一种有效的数据分析平台;根据Spark Streaming运行的原理设计一种动态采样的K-Means并行算法,与DRDAS结合能实时有效地检测大数据环境下的各种分布式拒绝服务(DDo S)攻击。实验结果显示:DRDAS具有好的可扩展性、容错性和实时处理能力,与动态采样的K-Means并行算法结合能实时地检测各种DDo S攻击,缩短了攻击的检测时间。

支持实时流计算应用的关键技术研究进展

信息系统在进行知识的挖掘和管理时,需要处理各种形式的数据,流数据便是其中之一.流数据具有数据规模大、产生速度快且蕴含的知识具有较强时效性等特点,因而发展支持实时处理应用的流计算技术对于信息系统的知识管理十分重要.流计算系统可以追溯到29世纪90年代,至今已经经历了长足的发展.然而,当前多样化的知识管理需求和新一代的硬件架构为流计算系统带来了全新的挑战和机遇,催生出了一系列流计算领域的技术研究.首先介绍流计算系统的基本需求以及发展脉络,再按照编程接口、执行计划、资源调度和故障容错4个层次分别分析流计算系统领域的相关技术;最后,展望流计算技术在未来可能的研究方向和发展趋势.

Online Nonstop Task Management for Storm-Based Distributed Stream Processing Engines

Most distributed stream processing engines(DSPEs)do not support online task management and cannot adapt to time-varying data flows.Recently,some studies have proposed online task deployment algorithms to solve this problem.However,these approaches do not guarantee the Quality of Service(QoS)when the task deployment changes at runtime,because the task migrations caused by the change of task deployments will impose an exorbitant cost.We study one of the most popular DSPEs,Apache Storm,and find out that when a task needs to be migrated,Storm has to stop the resource(implemented as a process of Worker in Storm)where the task is deployed.This will lead to the stop and restart of all tasks in the resource,resulting in the poor performance of task migrations.Aiming to solve this problem,in this pa-per,we propose N-Storm(Nonstop Storm),which is a task-resource decoupling DSPE.N-Storm allows tasks allocated to resources to be changed at runtime,which is implemented by a thread-level scheme for task migrations.Particularly,we add a local shared key/value store on each node to make resources aware of the changes in the allocation plan.Thus,each resource can manage its tasks at runtime.Based on N-Storm,we further propose Online Task Deployment(OTD).Differ-ing from traditional task deployment algorithms that deploy all tasks at once without considering the cost of task migra-tions caused by a task re-deployment,OTD can gradually adjust the current task deployment to an optimized one based on the communication cost and the runtime states of resources.We demonstrate that OTD can adapt to different kinds of applications including computation-and communication-intensive applications.The experimental results on a real DSPE cluster show that N-Storm can avoid the system stop and save up to 87%of the performance degradation time,compared with Apache Storm and other state-of-the-art approaches.In addition,OTD can increase the average CPU usage by 51%for computation-intensive applications and reduce network communication costs by 88%for communication-intensive ap-plications.

我国“职教二十条”的政策形成过程分析——基于多源流理论的视角

2019年,国务院颁布实施我国“职教二十条”。它的形成是多种因素共同促成的结果。以多源流理论为指导,从问题、政策和政治等三大源流及政策之窗对促进“职教二十条”政策形成的关键因素、积极因素及其相互间动态关系进行深入分析。分析表明:人才培养数量质量与经济社会发展需求之间矛盾突出、科研技术创新服务产业企业能力不足、职业教育培训体系和教学模式不健全、现行政策标准不健全与法律制度不完善构成“问题源流”;国家领导人的指示、“两会”代表的政策建议、大众传媒的报道及专家学者的研究成果构成“政策源流”;新一代领导集体高度重视职业教育改革的执政理念构成“政治源流”。在三大源流汇聚过程中,2017年习近平总书记在中国共产党第十九次全国代表大会上的报告是揭开我国“职教二十条”形成的政策之窗,不仅促使三大源流交互耦合,还合力促使“职教二十条”政策形成。

基于Spark Streaming的实时日志处理平台设计与实现

本文基于Hadoop Yarn平台,通过Spark Streaming流数据处理技术实时处理Web日志、系统日志、监控日志等,很好地解决了目前存在的问题,并使用HBase(Hadoop No SQL Database)作为数据库存储,更好地适应日志数据规模的增长。

基于Spark Streaming流技术的机动车缉查布控系统设计

机动车缉查布控系统以治安交通卡口系统为基础,将公路运行车辆的构成、流量分布、违章情况等信息进行汇总,实现对车辆轨迹的查询分析,从而为交通规划、管理和案件侦破等提供有效决策。由于机动车的数量增长迅猛,且机动车违规违章具有流动性、偶发性的特点,造成卡口系统录入的数据量巨大,传统的机动车缉查布控技术只能在预设条件下对数据进行查询,不能实现快速地实时分析查询。通过采用Spark Streaming技术进行大数据实时处理,实现实时显示和告警机动车违规违章活动热点,分析和预测机动车活动轨迹。

大规模物联网数据的实时处理与存储

详细探讨大规模物联网数据的实时处理与存储技术及其在智能城市交通管理等应用中的重要性。针对物联网数据的多样性、高并发、时变性以及异构性等特点,介绍流式处理、复杂事件处理(Complex Event Processing,CEP)、时序数据库以及分布式文件系统等主流技术,并结合智能交通管理案例,阐述这些技术在实时监测、交通调控、事件预警和长期趋势分析等方面的应用效果,为构建高效、实时的物联网数据处理与存储系统提供了理论支持和实践指导。

基于“湖仓一体”技术的城轨大数据平台设计与升级改造实践

为了探寻城市轨道交通行业大数据平台建设与升级改造的最优方案,本文以城轨大数据平台为研究对象,从城轨大数据平台发展历程出发,梳理城轨大数据平台发展的3个阶段,分析各阶段大数据平台所采用的技术与优缺点,重点总结当前阶段“湖仓一体”大数据技术所具备的湖仓一体、流批一体、OLTP+OLAP、多重负载等优点,研究了基于该技术的大数据平台架构升级改造设计要点,并将该技术在北京地铁数据中心的大数据平台升级改造中进行应用验证。结果表明:“湖仓一体”大数据平台技术兼具数据湖的低成本、数据仓库的高性能等优点,解决了原大数据平台在性能、容量与多用途支持上的不足,为城轨行业大数据平台建设与升级改造提供了新的解决思路。

基于流处理的重型柴油车排放预警监控系统设计

为了能够实时监控重型柴油车污染物排放水平,快速筛查出排放超标的车辆。利用以Flink计算引擎为核心的Apache开源大数据框架,根据重型车排放车载终端发回的数据计算出车辆污染物排放水平。开发完成后,该预警监控系统会根据计算结果向车辆用户、售后部门、研发部门推送排放超标车辆信息,同时根据需求部门的要求,统计超标车辆的运行工况、配置信息、故障码信息等,实现快速定位排放超标原因,有效提高了社会和经济效益。

CEStream:一种复杂事件流处理语言

复杂事件处理是支持大数据处理的流式计算平台的核心技术之一。CEStream语言作为一种新型的事件流处理语言,支持分布式环境下的复杂事件处理。该语言以XML等层次化数据为数据模型,为复杂事件检测提供了一种正规树模式匹配功能,并且支持结构连接和正规式匹配。同时,针对分布式的多个事件流,其能够将各个事件源模式匹配的结果按照时间顺序再一次进行正规式模式匹配,满足多源组合型复杂事件的检测需求,具有较强的事件处理能力。为了实现CEStream语言,研制了一个基于流数据处理集群和远端查询代理的执行引擎系统。该系统通过远程查询代理实现基于正规树模式的事件检测,通过流处理集群完成多源组合型复杂事件处理。实验表明,该系统实现了CEStream语言,有效地限制了各个节点之间的通信量,充分利用了集群的计算能力,整体性能能够满足应用需求。

基于卷积神经网络的学生课堂行为识别系统设计

精准的学生课堂行为识别结果有助于提升课堂教学效果,为此,设计一种基于卷积神经网络的学生课堂行为识别系统。系统的图像采集模块利用SZ-4K512M型摄像机,采集学生课堂行为的视频图像,并通过流式传输技术将标记后的采集图像传输至图像预处理模块;图像预处理模块对图像进行清洗和标准化处理后,传送至行为识别模块。行为识别模块通过卷积层、池化层和全连接层构建卷积神经网络,以已标记的学生课堂行为图像作为基础训练网络,利用完成训练的卷积神经网络识别学生课堂行为。实验结果表明,所设计系统可以精准识别学生玩手机、睡觉、举手等不同课堂行为,识别精度高于97%,说明该系统可以更好地掌握学生的心理活动变化。

基于流计算和大数据平台的实时交通流预测

目前交通流预测实时性差,很难满足在线分析和预测任务的需求,基于此提出一种Flink流计算框架和大数据平台结合的实时交通流预测方法。基于流计算框架实时捕捉和预处理数据,包括采用Flink的transform算子对数据进行校验和处理,将处理后的数据sink到大数据的HDFS文件系统,交由下一步的大数据并行框架进行分析建模与训练,实现基于流计算和大数据平台的实时交通流预测。实验结果表明,Flink能够实时捕捉和预处理交通流数据,把数据准时无误送入分布式文件系统中,在此基础上借助大数据框架下的并行分析和建模优势,在实时性数据分析与预测方面取得了较好的效果。