Multi-source heterogeneous data access management framework and key technologies for electric power Internet of Things

The power Internet of Things(IoT)is a significant trend in technology and a requirement for national strategic development.With the deepening digital transformation of the power grid,China’s power system has initially built a power IoT architecture comprising a perception,network,and platform application layer.However,owing to the structural complexity of the power system,the construction of the power IoT continues to face problems such as complex access management of massive heterogeneous equipment,diverse IoT protocol access methods,high concurrency of network communications,and weak data security protection.To address these issues,this study optimizes the existing architecture of the power IoT and designs an integrated management framework for the access of multi-source heterogeneous data in the power IoT,comprising cloud,pipe,edge,and terminal parts.It further reviews and analyzes the key technologies involved in the power IoT,such as the unified management of the physical model,high concurrent access,multi-protocol access,multi-source heterogeneous data storage management,and data security control,to provide a more flexible,efficient,secure,and easy-to-use solution for multi-source heterogeneous data access in the power IoT.

Image Processing on Geological Data in Vector Format and Multi-Source Spatial Data Fusion

The geological data are constructed in vector format in geographical information system (GIS) while other data such as remote sensing images, geographical data and geochemical data are saved in raster ones. This paper converts the vector data into 8 bit images according to their importance to mineralization each by programming. We can communicate the geological meaning with the raster images by this method. The paper also fuses geographical data and geochemical data with the programmed strata data. The result shows that image fusion can express different intensities effectively and visualize the structure characters in 2 dimensions. Furthermore, it also can produce optimized information from multi-source data and express them more directly.

Multi-Source Data Privacy Protection Method Based on Homomorphic Encryption and Blockchain

Multi-Source data plays an important role in the evolution of media convergence.Its fusion processing enables the further mining of data and utilization of data value and broadens the path for the sharing and dissemination of media data.However,it also faces serious problems in terms of protecting user and data privacy.Many privacy protectionmethods have been proposed to solve the problemof privacy leakage during the process of data sharing,but they suffer fromtwo flaws:1)the lack of algorithmic frameworks for specific scenarios such as dynamic datasets in the media domain;2)the inability to solve the problem of the high computational complexity of ciphertext in multi-source data privacy protection,resulting in long encryption and decryption times.In this paper,we propose a multi-source data privacy protection method based on homomorphic encryption and blockchain technology,which solves the privacy protection problem ofmulti-source heterogeneous data in the dissemination ofmedia and reduces ciphertext processing time.We deployed the proposedmethod on theHyperledger platformfor testing and compared it with the privacy protection schemes based on k-anonymity and differential privacy.The experimental results showthat the key generation,encryption,and decryption times of the proposedmethod are lower than those in data privacy protection methods based on k-anonymity technology and differential privacy technology.This significantly reduces the processing time ofmulti-source data,which gives it potential for use in many applications.

Research on Data Fusion of Adaptive Weighted Multi-Source Sensor

Data fusion can effectively process multi-sensor information to obtain more accurate and reliable results than a single sensor.The data of water quality in the environment comes from different sensors,thus the data must be fused.In our research,self-adaptive weighted data fusion method is used to respectively integrate the data from the PH value,temperature,oxygen dissolved and NH3 concentration of water quality environment.Based on the fusion,the Grubbs method is used to detect the abnormal data so as to provide data support for estimation,prediction and early warning of the water quality.

基于数据-模型混合驱动的电力系统机电暂态快速仿真方法

数据驱动建模方法改变了发电机传统的建模范式,导致传统的机电暂态时域仿真方法无法直接应用于新范式下的电力系统。为此,该文提出一种基于数据-模型混合驱动的机电暂态时域仿真(data and physics driven time domain simulation,DPD-TDS)算法。算法中发电机状态变量与节点注入电流通过数据驱动模型推理计算,并通过网络方程完成节点电压计算,两者交替求解完成仿真。算法提出一种混合驱动范式下的网络代数方程组预处理方法,用以改善仿真的收敛性;算法设计一种中央处理器单元-神经网络处理器单元(central processing unit-neural network processing unit,CPU-NPU)异构计算框架以加速仿真,CPU进行机理模型的微分代数方程求解;NPU作协处理器完成数据驱动模型的前向推理。最后在IEEE-39和Polish-2383系统中将部分或全部发电机替换为数据驱动模型进行验证,仿真结果表明,所提出的仿真算法收敛性好,计算速度快,结果准确。

多源异构数据融合关键技术与政务大数据治理体系

随着信息技术的飞速发展,各级政府和大型企业掌握的数据量正在以指数级别增长。然而,数据来源多样会导致格式差异,数据质量参差不齐会影响应用效果,数据分散管理会弱化关联汇集,数据形态异构会造成语义鸿沟。在此背景下,多源异构数据融合负责将来源不同的多模态数据进行有效整合,完成数据互补与关联,进而实现信息增强。目前,大多数已有研究的关注重点集中在大数据治理流程与多模态深度学习,很少有工作研究讨论完整的多源异构数据融合技术框架。因此,在综述关键技术的基础上,文中提出了一整套涵盖“数据引接-数据清洗-数据集成-数据融合”全过程的多源异构数据融合关键技术框架,并对各个环节需要解决的问题与重点任务进行介绍。然后,通过一个政务应用实例场景,给出了政务大数据治理体系的设计,以解决政务数据来源广泛、质量参差不齐、管理分散、形态异构的问题,并进一步阐述了多源异构数据融合的重要价值。最后总结全文并展望未来。

面向直升机装配生产线虚拟运行的多源数据感知与融合应用技术

针对直升机装配生产线中存在的数据来源繁多、描述形式不统一、利用率低等问题,对直升机装配生产线运行过程中的多源数据感知与融合应用技术进行研究。首先,定义生产线运行中的多源数据感知与融合技术内涵及其整体框架,包括生产线多源数据采集系统构建、多源数据融合技术构建、装配过程多源数据分析预测模型构建,以及生产线多源数据分析应用模块;其次,采用标准化的生产物联、基于本体的语义描述与基于多源数据的质量映射等方法,为实际装配生产线多源数据的感知与融合提供可行的解决方案;最后,阐述上述技术在生产线虚拟运行状态分析、设备使用状态监测和装配质量提升等方面的应用措施,为直升机装配生产线的数字孪生构建与虚拟高效运行提供优质数据基础。

LHAASO模拟作业从X86到ARM计算集群的移植

【目的】随着高能物理实验的推进与先进探测器的研发,产生的科学大数据显著增加,通过对这些数据的分析和模拟,可以发现宇宙运行规律并进一步探索宇宙的起源。【应用背景】科学数据的爆炸式增长对计算资源的规模和性能提出了更多的需求。例如,高海拔宇宙线观测站(LHAASO)自2020年实验开始运行以来,其宇宙线事例模拟作业一直在Intel X86集群上运行,但由于CPU资源有限,仅生产了第一阶段计划数据的一部分。【方法】基于对计算资源的需求和国际局势的变化,利用中国广东省东莞市的ARM架构计算集群,探索了异构计算服务设备在高能物理领域的应用。【结果】本文构建了一个完整的支持高能物理离线数据处理的应用程序生态环境。将基于平方公里阵列(KM2A)、水切伦科夫探测器阵列(WCDA)和广角切伦科夫望远镜阵列(WFCTA)实验的离线软件移植到ARM机器上运行,制定跨异地站点和异构计算集群的数据传输和作业调度策略,并评估了模拟作业在Intel X86和ARM集群中的性能和功耗差异。【结论】该环境中,移植的LHAASO模拟作业在ARM计算集群可以正确运行;虽然基于Intel X86架构的CPU单核性能优于ARM CPU,但是对于多核架构的整个服务器来说,ARM服务器性能更好。

OODAFlow:面向智能无人系统的流式数据处理框架

智能无人系统是一种能够在复杂环境中自主进行实时推理、决策和制定行动方案的计算系统。智能无人系统实现实时决策的关键在于对流式数据的实时处理,然而随着人工智能技术和传感器技术的快速发展,智能无人系统需要处理的数据规模不断增长,数据类型变得更加复杂。面对不断增长的数据处理性能需求,智能无人系统需要一个充分优化的专用流式数据处理框架来提升其数据处理性能。针对该问题,本文提出了一种面向智能无人系统的流式数据处理框架OODAFlow,该框架将智能无人系统的硬件特征和智能计算任务的数据特征与观察-判断-决策-行动(OODA)模型思想相融合,实现了OODA任务创建、任务调度、资源调度等功能,能够实现对智能无人系统异构资源的调度和智能计算任务的处理。本文在智能无人系统上搭建了一套OODA任务处理系统,验证了所提OODAFlow框架的可行性。通过提出的图像预处理过程优化、流水线优化以及判断节点并行加速优化等方法,提高了系统的数据吞吐性能和资源利用率。无人机智能控制任务的实验表明,采用本文提出的OODAFlow框架后,智能无人系统的数据处理性能提升了73倍。

卫星图像数据高速传输与处理系统的设计与验证

传统星上系统开发与研制通常是基于特定的专用宇航器件,这使得其开发周期长且可移植性较差。针对该问题,提出了一种基于Xilinx ZYNQ UltraScale+平台的ARM+可编程逻辑的解决方案。该方案用Vivado配置FPGA硬件架构,采用SDK(Software Development Kit)配置ARM实现两者数据交互。在整个系统设计中,通过集成在芯片内部的高速串行收发器和Aurora协议作为载体进行高速片间传输;采用自定义帧协议保证传输可靠性及安全性;采用AXI-Stream接口使其具有各类算法即插即用的灵活性;最后为验证本系统可以支持各类算法进行星上实时图像处理,通过在逻辑侧添加Sobel边缘检测算法进行验证。测试结果表明该系统图像数据传输无误码,星上串行高速传输速率较高,系统总体传输延时较低,且系统具有较强的算法通用性。

多源异构数据统一存储共享管理系统

大数据环境下,基于多源异构的专业数据,针对数据存储与共享管理业务,对多源异构数据统一存储共享系统及其功能组成进行了梳理,设计了一种面向文本、图片、专业领域数据等多源异构数据的跨模态统一存储与共享访问系统架构。采用分布式异构数据存储技术、数据模型管理技术,集数据获取、数据准备、可视化分析、数据共享与访问管理于一体,实现多源异构数据的统一存储与共享,并对系统的存储与访问性能进行评估分析,具备较高的稳定性和可靠性。验证了系统的实用性,从而达到了更好地为业务应用赋能,强力支撑各类应用开发的目的,具有广泛的适用性。

Managing Computing Infrastructure for IoT Data

Digital data have become a torrent engulfing every area of business, science and engineering disciplines, gushing into every economy, every organization and every user of digital technology. In the age of big data, deriving values and insights from big data using rich analytics becomes important for achieving competitiveness, success and leadership in every field. The Internet of Things (IoT) is causing the number and types of products to emit data at an unprecedented rate. Heterogeneity, scale, timeliness, complexity, and privacy problems with large data impede progress at all phases of the pipeline that can create value from data issues. With the push of such massive data, we are entering a new era of computing driven by novel and ground breaking research innovation on elastic parallelism, partitioning and scalability. Designing a scalable system for analysing, processing and mining huge real world datasets has become one of the challenging problems facing both systems researchers and data management researchers. In this paper, we will give an overview of computing infrastructure for IoT data processing, focusing on architectural and major challenges of massive data. We will briefly discuss about emerging computing infrastructure and technologies that are promising for improving massive data management.

Optimizing Query Results Integration Process Using an Extended Fuzzy C-Means Algorithm

Cleaning duplicate data is a major problem that persists even though many works have been done to solve it, due to the exponential growth of data amount treated and the necessity to use scalable and speed algorithms. This problem depends on the type and quality of data, and differs according to the volume of data set manipulated. In this paper we are going to introduce a novel framework based on extended fuzzy C-means algorithm by using topic ontology. This work aims to improve the OLAP querying process over heterogeneous data warehouses that contain big data sets, by improving query results integration, eliminating redundancies by using the extended classification algorithm, and measuring the loss of information.

面向异质性医学图像处理的深度学习算法综述

近年来深度学习技术在诸多计算机视觉任务上取得了令人瞩目的进步,也让越来越多的研究者尝试将其应用于医学图像处理领域,如面向高通量医学图像(CT、MRI)的解剖结构分割等,旨在为医生提供诊断辅助,提高其阅片效率.由于训练医学图像处理的深度学习模型同样需要大量的标注数据,同一医疗机构的数据往往不能满足需求,而受设备和采集协议的差异的影响,不同医疗机构的数据具有很大的异质性,这导致通过某些医疗机构的数据训练得到模型很难在其他医疗机构的数据上取得可靠的结果.此外,不同的医疗数据在患者个体病情阶段的分布上也往往是十分不均匀的,这同样会降低模型的可靠性.为了减少数据异质性的影响,提高模型的泛化能力,域适应、多站点学习等技术应运而生.其中域适应技术作为迁移学习中的研究热点,旨在将源域上学习的知识迁移到未标记的目标域数据上;多站点学习和数据非独立同分布的联邦学习技术则旨在在多个数据集上学习一个共同的表示,以提高模型的鲁棒性.从域适应、多站点学习和数据非独立同分布的联邦学习技术入手,对近年来的相关方法和相关数据集进行了综述、分类和总结,为相关研究提供参考.

专用数据处理器综述

随着网络传输带宽的增加,数据中心复杂的基础设施操作占用着越来越多的计算资源,影响业务处理性能。专用数据处理器(DPU)是新近发展起来的一种专用处理器,提出背景就是应对数据中心的数据量和复杂性的指数级增长带来的性能瓶颈。DPU的出现是异构计算的一个阶段性标志,要解决的核心问题是针对基础设施降低整体系统的总体拥有成本,提升整个计算系统的效率,即将“中央处理器处理效率低下、图形处理器处理不了”的负载卸载到DPU。首先介绍了DPU的发展背景,基于网络处理模型分析了DPU的硬件架构,并与智能网卡及网络处理器进行了对比;之后介绍了DPU的编程模型,目前业界的DPU产品与应用;最后总结并展望了DPU未来的研究发展方向。

基于多源系统融合的结构形态感知技术

针对薄壁结构零件在装配过程中由于装配状态变化存在的装配受力变形问题,结合数字孪生发展背景,提出一种激光跟踪、视觉测量、光纤监测多源系统融合的结构形态感知技术。首先,建立多源系统多站位坐标统一模型,实现坐标基准统一;其次,建立异构数据融合模型,完成光纤监测波长与空间点坐标异构数据统一,基于高斯过程实现多源数据融合,预测变形点云,实现产品结构形态感知;最后,以蒙皮薄壁结构为例,模拟装配变形实验。结果表明,融合方法所感知数据更好地反应实际变形,其平均相对误差为4.66%,绝对误差保持在0.016 mm。多源异构数据融合基于实测数据预测形变点云,可实现结构实时变形监测,从而简化视觉测量方式,提高了曲面信息保真度,为动态孪生模型构建提供新思路。

Computing infrastructure for big data processing

With computing systems undergone a fundamen- tal transformation from single-processor devices at the turn of the century to the ubiquitous and networked devices and the warehouse-scale computing via the cloud, the parallelism has become ubiquitous at many levels. At micro level, par- allelisms are being explored from the underlying circuits, to pipelining and instruction level parallelism on multi-cores or many cores on a chip as well as in a machine. From macro level, parallelisms are being promoted from multiple ma- chines on a rack, many racks in a data center, to the glob- ally shared infrastructure of the Internet. With the push of big data, we are entering a new era of parallel computing driven by novel and ground breaking research innovation on elas- tic parallelism and scalability. In this paper, we will give an overview of computing infrastructure for big data processing, focusing on architectural, storage and networking challenges of supporting big data paper. We will briefly discuss emerging computing infrastructure and technologies that are promising for improving data parallelism, task parallelism and encour- aging vertical and horizontal computation parallelism.

基于改进FCM的多源异构能源数据预处理与去噪

为了提升电力调控系统中能源数据精度和融合效果,提出构建参数估计优化模型。在进行参数寻优处理后,基于改进混沌算法(GGA)和马尔科夫蒙特卡罗算法(MCMC)分别对电力调度系统中的缺失数据进行填补和估计,以提升后续参数处理准确性;利用基于模糊聚类的FCM算法,提出基于改进FCM的数据去噪方法,减少多源异构能源数据中的噪声等干扰信息,以提升后续数据融合效果;通过实验验证提出算法的有效性。实验结果表明,提出的算法检出率为94.51%,数据融合效果较好,稳定性强,对于不确定的数据具有较好的融合效果。

基于异构计算平台的背景噪声预处理并行算法

背景噪声地震学利用地震台站记录的背景噪声信号计算台站之间的互相关信息,以此推演地质结构信息,近年来广泛应用于地球结构和油气勘探等领域。地震噪声数据处理往往需要通过预处理计算来减少仪器、地震信号的干扰,这一过程需进行多种信号处理计算。随着我国地震台站布设的推广,地震波形文件持续积累,预处理计算的耗时大大增加。针对计算耗时问题,基于图形处理器异构计算平台,提出了一种并行地震噪声预处理算法。并行算法在台站、时间和分段3个维度设计了并行计算框架,针对预处理中的计算过程实现了计算核函数,并且通过分批计算实现了对大批量文件的自适应处理。实验结果表明,并行预处理算法取得了约95倍的加速比,且具备良好的并行性。

立体交通工程大体量三维空间数据管理方法研究

交通工程行业数据平台普遍存在数据来源广泛、数据类型异构、数据存储量巨大以及数据展示效果差等问题。本文针对上述平台存在的问题进行研究,基于三维GIS、物联网、大数据等新一代信息技术,提出适用于交通工程行业大体量三维空间数据的管理方法,对大体量三维空间数据进行轻量化、web GL渲染及交通边缘数据处理,有效解决了模型渲染瓶颈并提高了渲染效率,对于交通工程的数字化发展产生积极的作用。