Super Resolution Perception for Improving Data Completeness in Smart Grid State Estimation

The smart grid is an evolving critical infrastructure,which combines renewable energy and the most advanced information and communication technologies to provide more economic and secure power supply services.To cope with the intermittency of ever-increasing renewable energy and ensure the security of the smart grid,state estimation,which serves as a basic tool for understanding the true states of a smart grid,should be performed with high frequency.More complete system state data are needed to support high-frequency state estimation.The data completeness problem for smart grid state estimation is therefore studied in this paper.The problem of improving data completeness by recovering highfrequency data from low-frequency data is formulated as a super resolution perception(SRP)problem in this paper.A novel machine-learning-based SRP approach is thereafter proposed.The proposed method,namely the Super Resolution Perception Net for State Estimation(SRPNSE),consists of three steps:feature extraction,information completion,and data reconstruction.Case studies have demonstrated the effectiveness and value of the proposed SRPNSE approach in recovering high-frequency data from low-frequency data for the state estimation.

Determining the Real Data Completeness of a Relational Dataset

Low quality of data is a serious problem in the new era of big data, which can severely reduce the usability of data, mislead or bias the querying, analyzing and mining, and leads to huge loss. Incomplete data is common in low quality data, and it is necessary to determine the data completeness of a dataset to provide hints for follow-up operations on it.Little existing work focuses on the completeness of a dataset, and such work views all missing values as unknown values. In this paper, we study how to determine real data completeness of a relational dataset. By taking advantage of given functional dependencies, we aim to determine some missing attribute values by other tuples and capture the really missing attribute cells. We propose a data completeness model, formalize the problem of determining the real data completeness of a relational dataset, and give a lower bound of the time complexity of this problem. Two optimal algorithms to determine the data completeness of a dataset for different cases are proposed. We empirically show the effectiveness and the scalability of our algorithms on both real-world data and synthetic data.

Research on completeness of earthquake data in the Chinese mainland（Ⅱ）──The regional distribution of the beginning years of basically complete earthquake data

Based on the concrete conditions of earthquake data in the west of China, East China and SOuth China, we studied the completeness of data in these regions by suitable methods to local conditions. Otherwise, we roughly estimated monitoring capability of local networks in China since 1970 and some outlying regions where the data is lack. Finally, we gave the regional distribution of the beginning years since which the data for different magnitude intervals are largely complete in the Chinese mainland.

Research on the completeness of earthquake data in the Chinese mainland（I）──North China

In terms of the temporal-spatial distribution features of earthquakes, we study the completeness of historical data in North China where there is the most plenty historical data and with the longest record history by some meth ods of analysis and comparison. The results are obtained for events with Ms≥4 are largely complete since 1484 in North China (except Huanghai sea region and remote districts, such as Nei Mongol Autonomous region), but quakes with Ms≥6 are largely complete since 1291 in the middle and lower reaches of the Yellow River.

Non-Linear Matrix Completion

Current methods for predicting missing values in datasets often rely on simplistic approaches such as taking median value of attributes, limiting their applicability. Real-world observations can be diverse, taking stock price as example, ranging from prices post-IPO to values before a company’s collapse, or instances where certain data points are missing due to stock suspension. In this paper, we propose a novel approach using Nonlinear Matrix Completion (NIMC) and Deep Matrix Completion (DIMC) to predict associations, and conduct experiment on financial data between dates and stocks. Our method leverages various types of stock observations to capture latent factors explaining the observed date-stock associations. Notably, our approach is nonlinear, making it suitable for datasets with nonlinear structures, such as the Russell 3000. Unlike traditional methods that may suffer from information loss, NIMC and DIMC maintain nearly complete information, especially in high-dimensional parameters. We compared our approach with state-of-the-art linear methods, including Inductive Matrix Completion, Nonlinear Inductive Matrix Completion, and Deep Inductive Matrix Completion. Our findings show that the nonlinear matrix completion method is particularly effective for handling nonlinear structured data, as exemplified by the Russell 3000. Additionally, we validate the information loss of the three methods across different dimensionalities.

Study on the Improvement of the Application of Complete Ensemble Empirical Mode Decomposition with Adaptive Noise in Hydrology Based on RBFNN Data Extension Technology

The complex nonlinear and non-stationary features exhibited in hydrologic sequences make hydrological analysis and forecasting difficult.Currently,some hydrologists employ the complete ensemble empirical mode decomposition with adaptive noise(CEEMDAN)method,a new time-frequency analysis method based on the empirical mode decomposition(EMD)algorithm,to decompose non-stationary raw data in order to obtain relatively stationary components for further study.However,the endpoint effect in CEEMDAN is often neglected,which can lead to decomposition errors that reduce the accuracy of the research results.In this study,we processed an original runoff sequence using the radial basis function neural network(RBFNN)technique to obtain the extension sequence before utilizing CEEMDAN decomposition.Then,we compared the decomposition results of the original sequence,RBFNN extension sequence,and standard sequence to investigate the influence of the endpoint effect and RBFNN extension on the CEEMDAN method.The results indicated that the RBFNN extension technique effectively reduced the error of medium and low frequency components caused by the endpoint effect.At both ends of the components,the extension sequence more accurately reflected the true fluctuation characteristics and variation trends.These advances are of great significance to the subsequent study of hydrology.Therefore,the CEEMDAN method,combined with an appropriate extension of the original runoff series,can more precisely determine multi-time scale characteristics,and provide a credible basis for the analysis of hydrologic time series and hydrological forecasting.

Tensor Completion for Recovering Multichannel Audio Signal with Missing Data

The quality of a multichannel audio signal may be reduced by missing data, which must be recovered before use. The data sets of multichannel audio can be quite large and have more than two axes of variation, such as channel, frame, and feature. To recover missing audio data, we propose a low-rank tensor completion method that is a high-order generalization of matrix completion. First, a multichannel audio signal with missing data is modeled by a three-order tensor. Next, tensor completion is formulated as a convex optimization problem by defining the trace norm of the tensor, and then an augmented Lagrange multiplier method is used for solving the constrained optimization problem. Finally, the missing data is replaced by alternating iteration with a tensor computation. Experiments were conducted to evaluate the effectiveness on data of a 5.1-channel audio signal. The results show that the proposed method outperforms state-of-the-art methods. Moreover, subjective listening tests with MUSHRA(Multiple Stimuli with Hidden Reference and Anchor) indicate that better audio effects were obtained by tensor completion.

Missing interpolation model for wind power data based on the improved CEEMDAN method and generative adversarial interpolation network

Randomness and fluctuations in wind power output may cause changes in important parameters(e.g.,grid frequency and voltage),which in turn affect the stable operation of a power system.However,owing to external factors(such as weather),there are often various anomalies in wind power data,such as missing numerical values and unreasonable data.This significantly affects the accuracy of wind power generation predictions and operational decisions.Therefore,developing and applying reliable wind power interpolation methods is important for promoting the sustainable development of the wind power industry.In this study,the causes of abnormal data in wind power generation were first analyzed from a practical perspective.Second,an improved complete ensemble empirical mode decomposition with adaptive noise(ICEEMDAN)method with a generative adversarial interpolation network(GAIN)network was proposed to preprocess wind power generation and interpolate missing wind power generation sub-components.Finally,a complete wind power generation time series was reconstructed.Compared to traditional methods,the proposed ICEEMDAN-GAIN combination interpolation model has a higher interpolation accuracy and can effectively reduce the error impact caused by wind power generation sequence fluctuations.

The Complete K-Level Tree and Its Application to Data Warehouse Filtering

This paper presents a simple complete K level tree (CKT) architecture for text database organization and rapid data filtering. A database is constructed as a CKT forest and each CKT contains data of the same length. The maximum depth and the minimum depth of an individual CKT are equal and identical to data’s length. Insertion and deletion operations are defined; storage method and filtering algorithm are also designed for good compensation between efficiency and complexity. Applications to computer aided teaching of Chinese and protein selection show that an about 30% reduction of storage consumption and an over 60% reduction of computation may be easily obtained.

矿井通风参数缺失数据插补方法

矿井智能通风系统对矿山智能化建设至关重要。为解决矿井通风参数在实际测量时,因为巷道不具备测试条件、仪器信号受到干扰、巷道断面风速不均一、人工操作不当等制约性因素,造成的矿井通风参数数据缺失问题,提出了1种基于随机森林−链式方程多重插补法的矿井通风参数缺失数据插补方法。采用链式方程多重插补法,通过迭代对每个缺失的属性值产生n个插补值,从而产生n个完整数据集,对n个完整数据集进行分析优化得到1个最终的完整数据集。为了提高缺失值插补精度,合理考虑了矿井通风参数缺失数据的不确定性对分析过程的影响,在随机森林的预测任务中,结合预测均值匹配模型对缺失数据进行插补。以潞新二矿为实验对象,利用智能矿井通风仿真系统IMVS对潞新二矿矿井通风参数原始数据集进行数据预处理,得到完整、准确的矿井通风参数完整数据集,对完整数据集分别进行了不同缺失属性、不同数据缺失率、不同迭代次数的对比试验。以多种模型评价指标对模型有效性进行评估。结果表明:基于随机森林的链式方程多重插补模型插补形成的完整数据集与原始数据集具有很好的相似性;对不同缺失列进行插补实验的结果显示插补模型可以轻松处理混合类型的数据,自主学习参数之间的相关性从而降低了插补复杂性;迭代后形成的n个数据集通过分析合并成一个最终数据集,提高了插补准确率;对初始插补后的完整数据集进行不同迭代次数的试验,发现迭代超过一定次数后,数据相关性一定会收敛。

基于双通道生成对抗网络的城市用电负荷缺失数据补全方法

用电负荷数据的完整性与有效性在负荷预测等应用中具有重要意义。传统的缺失数据补全方法缺乏对用电负荷和多种外部时空关联信息的挖掘,难以获得高精度的补全结果。文中提出了一种双通道生成对抗网络,对缺失的负荷数据进行补全。首先,根据负荷的周期性变化特征和时空关联性构建三阶负荷张量,并将影响负荷变化的多种外部因素构建为三阶辅助信息张量。然后,为满足两种张量的双输入需求,在生成对抗网络的输入层引入双通道机制,通过卷积与反卷积运算提取张量的特征;为提升网络对张量数据的训练效果和补全精度,将张量分解损失引入原始损失函数,并采用改进的混沌映射粒子群优化算法联合优化超参数和网络。最后,在真实负荷数据集上开展数据补全实验。结果表明,所提方法能够对随机缺失率不超过50%、连续缺失不超过3天的负荷数据进行准确补全。

结合矩阵补全的宽度协同过滤推荐算法

协同过滤是推荐系统中最经典的方法之一,能够满足人们对个性化推荐任务的需求,但许多协同过滤算法在面对评分数据稀疏性问题时推荐效果不佳。为解决此问题,提出一种结合矩阵补全的宽度协同过滤推荐算法。先使用矩阵补全技术对用户项目评分矩阵进行补全,再利用补全后的矩阵对已评分的用户和项目分别寻找其近邻项,进而构造用户与项目的评分协同向量,最后使用宽度学习系统来构建用户项目与评分之间的复杂的非线性关系。在MovieLens和filmtrust数据集上对所提出算法的有效性进行检验。试验结果表明,与当前最先进的方法相比,该方法能够有效地缓解数据稀疏性问题,具有较低的计算复杂度,在一定程度上提升了推荐系统的性能。

基于车载三维激光扫描的城市道路竣工测量探讨

车载三维激光扫描系统融合了多种传感器和数据源,可以自动、迅速地获取道路的全方位信息。其扫描速度迅捷、数据信息丰富、精确度高、采集过程安全简单,并能节省人力。此技术显著提高了外业生产效率,并降低了生产成本。对车载三维激光扫描技术在道路工程竣工测量中的内外业处理流程的研究结果表明:该技术的精度可达到1∶500测图精度要求,满足城市高架路竣工规划测绘的精度需求。该技术方案是切实可行的,且能高效地提高生产效率。

SLAM激光扫描技术在地铁隧道竣工测量中的应用

轴线偏差和断面测量一直是隧道竣工测量的两个重要指标。传统使用全站仪或断面仪进行测量的方式不仅作业效率低,采集的数据量也较少,难以反映隧道内的真实情况,无法满足隧道竣工测量的要求。本文提出了一种基于SLAM手持激光扫描仪进行地铁隧道竣工测量的方法,并在上海市某在建地铁隧道内进行了现场试验。比较激光点云后处理得到的断面中心与设计数据,二者平均点位精度小于3 cm。这表明使用SLAM手持激光扫描仪进行地铁隧道竣工测量的方法有效可行,精度能够满足竣工测量的要求,具有广阔的应用前景。

基于Transformer的街道停车位数据补全和预测

随着城市汽车数量的持续增长,街道停车难已经成为一个热点问题。解决街道停车问题的关键在于准确预测街道未来的停车位信息。移动群智感知方式(CrowdSensing)通过在车辆上安装声呐以感知路边的停车位情况,是一种低成本、高效益的感知停车位的方式,然而这种方式感知的停车位数据在时间上存在高稀疏性问题,传统模型无法直接用于预测。针对此问题,提出了一种基于Transformer的停车位序列补全和预测网络,此网络通过编码器生成缺失停车位序列的记忆,进而解码器以自回归的方式补全停车位序列中缺失的部分,同时预测出未来的停车位信息。实验结果表明,所提方法在两个高缺失的街道停车位数据集上的补全和预测效果都优于传统的机器学习和深度学习方法。

乌梁素海湖冰完整生消过程的数值模拟

为探究气候变暖背景下乌梁素海湖冰的冰厚、冰温和冰物候特征的变化,利用ERA5-Land和MERRA-2再分析数据作为大气强迫场,结合多源卫星遥感反演得到的初始模拟日期,通过一维高分辨率热力学雪冰(HIGHTSI)模型模拟了2015—2016年乌梁素海湖冰的完整生消过程。结果显示:①研究期内,最大冰厚达到41.7 cm,初冰日和终冰日分别为2015年11月21日和2016年3月25日,冰期为126 d。②对于水深较浅而冬季日照充足的乌梁素海,气温是影响冰厚的主要因素,太阳辐射次之,两者的昼夜周期循环显著影响冰层的厚度和温度;当冰面有雪覆盖时,积雪的低导热和高反照率会明显削弱气温和太阳辐射对冰层的影响。③即使缺失现场观测数据,采用气象数据和遥感反演的初始模拟日期仍能准确地表征现场真实冰雪的完整演变过程。该研究可为中纬度干旱区季节性冰封浅水湖冰的计算和湖冰生消的年际变化研究奠定基础。

基于双通道回声状态网络的时间序列补全及单步预测

随着物联网的发展,众多传感器采集到大量具有丰富数据相关性的时间序列,为各种数据挖掘应用提供强大的数据支持。然而,一些客观或主观原因(如设备故障、稀疏感知等)往往会造成采集到的数据出现不同程度的缺失。虽然已有很多方法被提出用于解决这一问题,但这些方法在数据相关性方面或考虑不够全面,或计算成本过高。而且,现有方法仅关注对缺失值的补全,未能兼顾下游应用。针对上述不足,设计了一种兼顾补全与预测任务的双通道回声状态网络。两个通道的网络虽共用输入层,但具有各自的储备池和输出层。两者最大的区别是左/右通道的输出层分别表示输入层前/后一个时刻对应的目标值或预补值。最后将两个通道的估计值进行融合,充分利用来自缺失时刻之前和之后的数据相关性以进一步提升性能。两种缺失现象下(随机缺失和分段缺失)不同缺失率的实验结果表明,所提模型无论是在补全精度还是预测精度上都优于目前流行的各类方法。

基于CEEMDAN-GMDH-ARIMA的大坝变形预测模型研究

为提高大坝变形预测精度,针对大坝变形数据的复杂性和非线性等特征,基于自适应噪声完备集成经验模态分解(CEEMDAN)、数据处理群集法(GMDH)和差分自回归移动平均模型算法(ARIMA)进行大坝变形预测研究。采用CEEMDAN将大坝变形原始数据分解为高频随机分量、中频周期分量和低频趋势分量,再分别采用GMDH模型、ARIMA模型对高中频分量、低频分量进行预测,建立基于CEEMDAN-GMDH-ARIMA的大坝变形预测模型。以江西上犹江水电站为例,将该模型预测结果与反向传播(BP)、径向基函数(RBF)、GMDH和CEEMDAN-GMDH模型的预测结果进行对比分析。结果表明:CEEMDAN-GMDH-ARIMA模型的均方根误差(E_(RMS))、平均绝对误差(E_(MA))、相关系数(r)分别为0.048 mm、0.035 mm、0.994,均优于BP、RBF、GMDH、CEEMDAN-GMDH模型,模型预测效果最好,能够很好地体现监测点水平位移变化趋势。

基于数据挖掘探析《景岳全书·脱肛》用药规律

目的研究《景岳全书·脱肛》的组方用药规律及科学内涵。方法研读《景岳全书·脱肛》,收集并整理其治疗脱肛的方药组成,运用SPSS Modeler 18.0等软件针对其使用频率较高的中药性味归经、配伍组合等方面进行聚类分析、关联规则等处理,以探究并分析张介宾治疗脱肛的用药思想及规律。结果《景岳全书·脱肛》中所载治疗脱肛的方剂共26首,所涉中药61味,药性涉及寒、热、温、凉、平共5性,以温、寒、平为主;药味涉及酸、苦、甘、辛、咸、淡、涩7味,以甘、辛、苦最为频繁;药物归经共涉及12条经络,以脾、肝、肾、肺、胃经为主。关联规则得出15个药物组合。聚类分析得出5组聚类:升麻,柴胡,陈皮;人参,白术,甘草;甘草,黄芩;白芍,川芎,当归;山药,山茱萸,熟地黄。结论张介宾治疗脱肛主要采用温补之法,阴阳气血并补,对于兼有实邪者注意辨证施治。

基于扩散Transformer网络的激光雷达数据补全方法

由于设备故障和环境干扰等因素,激光雷达在数据采集过程中常常存在数据缺失或噪声干扰的问题,这些问题严重影响后续数据的解析和应用。为了解决这一难题,引入了扩散Transformer网络(DT-Net),将DT-Net用作生成器,与自注意单元判别器相结合。此外,还设计了一种扩散机制用于激光雷达数据补全。实验结果表明:相较于Poin Tr方法,所提出的方法在激光雷达数据补全任务方面取得了显著的改进,平均Chamfer距离(CD)值降低了约1.79%,F-Score值提升了约1.88%。