Turbo Message Passing Based Burst Interference Cancellation for Data Detection in Massive MIMO-OFDM Systems

This paper investigates the fundamental data detection problem with burst interference in massive multiple-input multiple-output orthogonal frequency division multiplexing(MIMO-OFDM) systems. In particular, burst interference may occur only on data symbols but not on pilot symbols, which means that interference information cannot be premeasured. To cancel the burst interference, we first revisit the uplink multi-user system and develop a matrixform system model, where the covariance pattern and the low-rank property of the interference matrix is discussed. Then, we propose a turbo message passing based burst interference cancellation(TMP-BIC) algorithm to solve the data detection problem, where the constellation information of target data is fully exploited to refine its estimate. Furthermore, in the TMP-BIC algorithm, we design one module to cope with the interference matrix by exploiting its lowrank property. Numerical results demonstrate that the proposed algorithm can effectively mitigate the adverse effects of burst interference and approach the interference-free bound.

A study on fast post-processing massive data of casting numerical simulation on personal computers

When castings become complicated and the demands for precision of numerical simulation become higher,the numerical data of casting numerical simulation become more massive.On a general personal computer,these massive numerical data may probably exceed the capacity of available memory,resulting in failure of rendering.Based on the out-of-core technique,this paper proposes a method to effectively utilize external storage and reduce memory usage dramatically,so as to solve the problem of insufficient memory for massive data rendering on general personal computers.Based on this method,a new postprocessor is developed.It is capable to illustrate filling and solidification processes of casting,as well as thermal stess.The new post-processor also provides fast interaction to simulation results.Theoretical analysis as well as several practical examples prove that the memory usage and loading time of the post-processor are independent of the size of the relevant files,but the proportion of the number of cells on surface.Meanwhile,the speed of rendering and fetching of value from the mouse is appreciable,and the demands of real-time and interaction are satisfied.

Research on data load balancing technology of massive storage systems for wearable devices

Because of the limited memory of the increasing amount of information in current wearable devices,the processing capacity of the servers in the storage system can not keep up with the speed of information growth,resulting in low load balancing,long load balancing time and data processing delay.Therefore,a data load balancing technology is applied to the massive storage systems of wearable devices in this paper.We first analyze the object-oriented load balancing method,and formally describe the dynamic load balancing issues,taking the load balancing as a mapping problem.Then,the task of assigning each data node and the request of the corresponding data node’s actual processing capacity are completed.Different data is allocated to the corresponding data storage node to complete the calculation of the comprehensive weight of the data storage node.According to the load information of each data storage node collected by the scheduler in the storage system,the load weight of the current data storage node is calculated and distributed.The data load balancing of the massive storage system for wearable devices is realized.The experimental results show that the average time of load balancing using this method is 1.75h,which is much lower than the traditional methods.The results show the data load balancing technology of the massive storage system of wearable devices has the advantages of short data load balancing time,high load balancing,strong data processing capability,short processing time and obvious application.

Parallelized User Clicks Recognition from Massive HTTP Data Based on Dependency Graph Model

With increasingly complex website structure and continuously advancing web technologies,accurate user clicks recognition from massive HTTP data,which is critical for web usage mining,becomes more difficult.In this paper,we propose a dependency graph model to describe the relationships between web requests.Based on this model,we design and implement a heuristic parallel algorithm to distinguish user clicks with the assistance of cloud computing technology.We evaluate the proposed algorithm with real massive data.The size of the dataset collected from a mobile core network is 228.7GB.It covers more than three million users.The experiment results demonstrate that the proposed algorithm can achieve higher accuracy than previous methods.

Study on Massive Vegetation Data Processing of FY-3 Based on RAM (h)

The vegetation data of the Fengyun meteorological satellite are segmented according to the latitude and longitude, and can be written into 648 blocks. However, the vegetation data processing efficiency is low because the data belongs to massive data. This paper presents a data processing method based on RAM (h) for Fengyun-3 vegetation data. First of all, we introduce the Locality-Aware model to segment the input data, then locate the data based on geographic location, and finally fuse the independent data based on geographical location. Experimental results show that the proposed method can effectively improve the data processing efficiency.

An Unsupervised Method for Short-Text Sentiment Analysis Based on Analysis of Massive Data

Common forms of short text are microblogs, Twitter posts, short product reviews, short movie reviews and instant messages. Sentiment analysis of them has been a hot topic. A highly-accurate model is proposed in this paper for short-text sentiment analysis. The researches target microblog, product review and movie reviews. Words, symbols or sentences with emotional tendencies are proved important indicators in short-text sentiment analysis based on massive users’ data. It is an effective method to predict emotional tendencies of short text using these features. The model has noticed the phenomenon of polysemy in single-character emotional word in Chinese and discusses singlecharacter and multi-character emotional word separately. The idea of model can be used to deal with various kinds of short-text data. Experiments show that this model performs well in most cases.

基于数据挖掘的5G Massive MIMO天线权值优化方法研究

本文基于4G/5G数据挖掘分析,给出了一种NSA组网下5G Massive MIMO天线权值智能优化方法。该方法结合4G MDT和5G MR数据,通过聚类和成形算法分析得到待优化小区理想权值集合,可以在海量权值因子中快速寻优得到最优权值组合,采用基于风险控制的调整算法实现Massive MIMO天线权值智能自动化迭代寻优。

Optimal decorrelated score subsampling for generalized linear models with massive data

In this paper, we consider the unified optimal subsampling estimation and inference on the lowdimensional parameter of main interest in the presence of the nuisance parameter for low/high-dimensionalgeneralized linear models (GLMs) with massive data. We first present a general subsampling decorrelated scorefunction to reduce the influence of the less accurate nuisance parameter estimation with the slow convergencerate. The consistency and asymptotic normality of the resultant subsample estimator from a general decorrelatedscore subsampling algorithm are established, and two optimal subsampling probabilities are derived under theA- and L-optimality criteria to downsize the data volume and reduce the computational burden. The proposedoptimal subsampling probabilities provably improve the asymptotic efficiency of the subsampling schemes in thelow-dimensional GLMs and perform better than the uniform subsampling scheme in the high-dimensional GLMs.A two-step algorithm is further proposed to implement, and the asymptotic properties of the correspondingestimators are also given. Simulations show satisfactory performance of the proposed estimators, and twoapplications to census income and Fashion-MNIST datasets also demonstrate its practical applicability.

The Interdisciplinary Research of Big Data and Wireless Channel: A Cluster-Nuclei Based Channel Model

Recently,internet stimulates the explosive progress of knowledge discovery in big volume data resource,to dig the valuable and hidden rules by computing.Simultaneously,the wireless channel measurement data reveals big volume feature,considering the massive antennas,huge bandwidth and versatile application scenarios.This article firstly presents a comprehensive survey of channel measurement and modeling research for mobile communication,especially for 5th Generation(5G) and beyond.Considering the big data research progress,then a cluster-nuclei based model is proposed,which takes advantages of both the stochastical model and deterministic model.The novel model has low complexity with the limited number of cluster-nuclei while the cluster-nuclei has the physical mapping to real propagation objects.Combining the channel properties variation principles with antenna size,frequency,mobility and scenario dug from the channel data,the proposed model can be expanded in versatile application to support future mobile research.

Design and development of real-time query platform for big data based on hadoop

This paper designs and develops a framework on a distributed computing platform for massive multi-source spatial data using a column-oriented database(HBase).This platform consists of four layers including ETL(extraction transformation loading) tier,data processing tier,data storage tier and data display tier,achieving long-term store,real-time analysis and inquiry for massive data.Finally,a real dataset cluster is simulated,which are made up of 39 nodes including 2 master nodes and 37 data nodes,and performing function tests of data importing module and real-time query module,and performance tests of HDFS's I/O,the MapReduce cluster,batch-loading and real-time query of massive data.The test results indicate that this platform achieves high performance in terms of response time and linear scalability.

Managing Computing Infrastructure for IoT Data

Digital data have become a torrent engulfing every area of business, science and engineering disciplines, gushing into every economy, every organization and every user of digital technology. In the age of big data, deriving values and insights from big data using rich analytics becomes important for achieving competitiveness, success and leadership in every field. The Internet of Things (IoT) is causing the number and types of products to emit data at an unprecedented rate. Heterogeneity, scale, timeliness, complexity, and privacy problems with large data impede progress at all phases of the pipeline that can create value from data issues. With the push of such massive data, we are entering a new era of computing driven by novel and ground breaking research innovation on elastic parallelism, partitioning and scalability. Designing a scalable system for analysing, processing and mining huge real world datasets has become one of the challenging problems facing both systems researchers and data management researchers. In this paper, we will give an overview of computing infrastructure for IoT data processing, focusing on architectural and major challenges of massive data. We will briefly discuss about emerging computing infrastructure and technologies that are promising for improving massive data management.

基于深度自回归模型的电网异常流量检测算法

针对电网中行为种类复杂多样且数量众多的问题,提出了一种基于自回归模型的电网异常流量检测算法。该算法利用深度自编码网络自动提取网络流量数据的特征,降低异常流量检测的分析周期,并自动挖掘数据的层次关系。通过支持向量机对提取的特征进行分类,实现对异常流量的检测。仿真实验结果表明,所提算法可以分析不同攻击向量,避免噪声数据的干扰,进而提高电网异常流量检测的精度,对于流量数据处理具有重要意义。

分布式技术在大模型训练和推理中的应用

近几年,人工智能被广泛应用于多个领域,大语言模型(以下简称大模型)的“预训练-微调”成为人工智能的最新范式。分布式技术存在于大模型生命周期的每一环,为大模型的发展助力。在数据获取环节,针对海量小文件的存储问题,研发了文件系统SuperFS,能够同时满足低延迟和可扩展的要求。在数据预处理环节,针对从分布式文件系统读取数据开销大的问题,研发了高效大数据处理引擎“诸葛弩”。在模型训练环节,针对检查点文件读写性能差的问题,提出了分布式检查点策略,加快了检查点文件的读写速度。在模型推理环节,针对KVCache对存储系统的挑战,研发了高吞吐推理方案FastDecode以及大模型推理架构Mooncake。分布式技术的应用,使大模型能够充分利用计算资源,加快训练速度,有利于人工智能领域的发展。

海量数据的分布式主成分分析算法及其在共同富裕测度中的应用

基于两轮型方法的分布式PCA算法(TR-DPCA),每台局部机器计算出和向量,并将它们传输到中央机器计算全样本数据的均值向量,再将它们传输给每台局部机器;然后,每台局部机器计算出散度矩阵,并将它们传输到中央机器计算全样本数据的协方差矩阵;最后根据协方差矩阵进行特征分解获得特征向量。通过数值模拟发现,TR-DPCA算法的表现与全样本PCA一致,且优于基于单轮型方法的分布式PCA算法。此外,将TR-DPCA算法应用到中国共同富裕测度中发现,中国的共同富裕水平呈现上升趋势,且个体差距在不断缩小。

大数据对临床检验诊断专业研究生教学的影响

随着第二代测序技术的飞速发展,大数据逐渐应用于临床诊断,基因组学、宏基因组学、转录组学和蛋白质组学等先进技术为医学检验的发展提供了更高的技术平台和成长空间。检测数据的爆炸式增长、结果分析的复杂化均对临床检验诊断专业研究生的教学提出了严峻考验和挑战,如何提高创新思维和处理大数据的实践能力是临床检验诊断专业研究生需要尽快解决的问题。为培养高层次复合型人才,使学生紧跟时代发展的脚步,医学院校需要将生物信息学列入教学中。将以授课为基础的学习(lecture-based learning,LBL)、以案例为基础的学习(case-based learning,CBL)和大规模在线开放课程(massive open online course,MOOC)教学模式相结合,极大提高了学生对生物信息学的兴趣和利用大数据的能力。文章围绕大数据对临床检验诊断研究生教学的影响展开讨论。

基于Fabric的海量交易数据上链预处理机制

Hyperledger Fabric是一种国内外广泛使用的联盟链框架,在基于Fabric技术的一些业务中具有协同组织众多、交易操作频繁、事务冲突增加等特点。Fabric采用的多版本并发控制技术能够在一定程度上解决部分交易冲突,提升系统并发性,但其机制不完善,会出现部分交易数据无法正常上链存储的问题。为了实现海量交易数据完整、高效、可信的上链存储,提出一种基于Fabric预言机的数据上链预处理机制。设计海量数据冲突预处理(MCPP)方法,通过检测、监听、延时提交、事务加锁、重排序缓存等方式实现主键冲突交易数据的完整上链。引入数据传输保障措施,在传输过程中利用非对称加密技术防止恶意节点伪造认证信息,确保交易数据链外处理前后的一致性。通过理论分析和实验结果表明,该机制可有效解决联盟链平台中海量交易数据上链时的并发冲突问题,当交易数据规模达到1 000和10 000时,MCPP的时效性比LMLS提高了38%和21.4%,且成功率接近100%,具有高效性和安全性,同时在无并发冲突情况下不影响Fabric系统性能。

一种FPGA集群轻量级深度学习计算架构设计及实现

传感器技术的发展带来了边缘、端设备功能的迅速迭代升级,也带来了战场前端的数据量成倍增长。针对边缘、端设备数据量的急剧增长和芯片计算处理能力的矛盾,结合Map/Reduce框架,提出了一种基于现场可编程门阵列(Field Programmable Gate Array,FPGA)计算集群资源的深度学习架构,能够实现多个深度学习算法的并行快捷部署和应用。该轻量级深度学习计算架构同时满足军事应用对“端”的智能处理能力提出的新要求,即不仅局限于数据采集和智能的应用,还必须具备分布式并行智能实时计算的能力。该FPGA集群轻量级深度学习计算框架部署不同类型算法容易,实时性高(ms级任务响应),可扩展性好,在多种类异构传感器、大场景大数据吞吐量的军事场景及森林防火等民用场景有广泛的应用前景。

双层索引驱动的隧洞海量点云高效管理方法

针对隧洞表观性态监测,三维激光扫描获取的点云具有数据量巨大、非结构化以及狭长线状非均匀分布等特点,给隧洞点云数据处理极大的压力,也制约了隧洞工程点云监测应用的发展。为此,本文结合隧洞工程空间分布特点,提出一种基于双层索引结构的隧洞海量点云管理方法。该方法设计了一种基于Hough变换的隧洞水平中线粗提取方法,指导隧洞点云数据沿水平中线进行点云自动分段;而后利用“自下而上”的归并构建策略建立分段点云八叉树索引。在此基础上,利用非冗余的多层次细节(LOD)建模方法和内外存动态调度技术实现海量点云数据快速可视化。实验结果显示,本文方法有效提高了隧洞点云水平轴线提取效率,基于双层索引结构的隧洞点云管理在点云检索、海量点云数据可视化等方面表现出优异性能。

面向深度神经网络大规模分布式数据并行训练的MC^(2)能耗模型

深度神经网络(deep neural network,DNN)在许多现代人工智能(artificial intelligence,AI)任务中取得了最高的精度.近年来,使用高性能计算平台进行大规模分布式并行训练DNN越来越普遍.能耗模型在设计和优化DNN大规模并行训练和抑制高性能计算平台过量能耗方面起着至关重要的作用.目前,大部分的能耗模型都是从设备的角度出发对单个设备或多个设备构成的集群进行能耗建模,由于缺乏从能耗角度对分布式并行DNN应用进行分解剖析,导致罕有针对分布式DNN应用特征进行建模的能耗模型.针对目前最常用的DNN分布式数据并行训练模式,从DNN模型训练本质特征角度出发,提出了“数据预处理(materials preprocessing)-前向与反向传播(computing)-梯度同步与更新(communicating)”三阶段MC^(2)能耗模型,并通过在国产E级原型机天河三号上使用最多128个MT节点和32个FT节点训练经典的VGG16和ResNet50网络以及最新的Vision Transformer网络验证了模型的有效性和可靠性.实验结果表明,MC^(2)与真实能耗测量结果相差仅为2.84%,相较4种线性比例能耗模型以及AR,SES,ARIMA时间预测模型准确率分别提升了69.12个百分点,69.50个百分点,34.58个百分点,13.47个百分点,5.23个百分点,22.13个百分点,10.53个百分点.通过使用的模型可以在超算平台得到DNN模型的各阶段能耗和总体能耗结果,为评估基于能耗感知的DNN大规模分布式数据并行训练及推理各阶段任务调度、作业放置、模型分割、模型裁剪等优化策略的效能提供了基础.

数字孪生流域三维数据底板建设研究及应用

三维数据底板是流域数字化映射的成果,更是数字化场景构建、智慧化模拟迭代的基础。为厘清三维数据底板建设的技术逻辑,系统性地阐述了三维数据底板的定位、建设任务和技术路线图,重点对建设过程中的海量数据融合、数据轻量处理、场景渲染发布、数据可视可算、数据共享共建等关键技术问题展开深入全面的论述,分析技术难点、解决路径和具体方法。基于以上研究,以浙江省曹娥江数字孪生流域为例,运用BIM+GIS等技术构建L2、L3三维数据底板,以三维数字化场景支撑流域“四预”可视化模型应用。相关成果对类似数据底板建设具有借鉴意义。