Optimizing Big Data Retrieval and Job Scheduling Using Deep Learning Approaches

Big data analytics in business intelligence do not provide effective data retrieval methods and job scheduling that will cause execution inefficiency and low system throughput.This paper aims to enhance the capability of data retrieval and job scheduling to speed up the operation of big data analytics to overcome inefficiency and low throughput problems.First,integrating stacked sparse autoencoder and Elasticsearch indexing explored fast data searching and distributed indexing,which reduces the search scope of the database and dramatically speeds up data searching.Next,exploiting a deep neural network to predict the approximate execution time of a job gives prioritized job scheduling based on the shortest job first,which reduces the average waiting time of job execution.As a result,the proposed data retrieval approach outperforms the previous method using a deep autoencoder and Solr indexing,significantly improving the speed of data retrieval up to 53%and increasing system throughput by 53%.On the other hand,the proposed job scheduling algorithmdefeats both first-in-first-out andmemory-sensitive heterogeneous early finish time scheduling algorithms,effectively shortening the average waiting time up to 5%and average weighted turnaround time by 19%,respectively.

Similarity matching method of power distribution system operating data based on neural information retrieval

Operation control of power systems has become challenging with an increase in the scale and complexity of power distribution systems and extensive access to renewable energy.Therefore,improvement of the ability of data-driven operation management,intelligent analysis,and mining is urgently required.To investigate and explore similar regularities of the historical operating section of the power distribution system and assist the power grid in obtaining high-value historical operation,maintenance experience,and knowledge by rule and line,a neural information retrieval model with an attention mechanism is proposed based on graph data computing technology.Based on the processing flow of the operating data of the power distribution system,a technical framework of neural information retrieval is established.Combined with the natural graph characteristics of the power distribution system,a unified graph data structure and a data fusion method of data access,data complement,and multi-source data are constructed.Further,a graph node feature-embedding representation learning algorithm and a neural information retrieval algorithm model are constructed.The neural information retrieval algorithm model is trained and tested using the generated graph node feature representation vector set.The model is verified on the operating section of the power distribution system of a provincial grid area.The results show that the proposed method demonstrates high accuracy in the similarity matching of historical operation characteristics and effectively supports intelligent fault diagnosis and elimination in power distribution systems.

A study of multiyear ice concentration retrieval algorithms using AMSR-E data

In recent years, the rapid decline of Arctic sea ice area （SIA） and sea ice extent （SIE）, especially for the multiyear （MY） ice, has led to significant effect on climate change. The accurate retrieval of MY ice concentration retrieval is very important and challenging to understand the ongoing changes. Three MY ice concentration retrieval algorithms were systematically evaluated. A similar total ice concentration was yielded by these algorithms, while the retrieved MY sea ice concentrations differs from each other. The MY SIA derived from NASA TEAM algorithm is relatively stable. Other two algorithms created seasonal fluctuations of MY SIA, particularly in autumn and winter. In this paper, we proposed an ice concentration retrieval algorithm, which developed the NASA TEAM algorithm by adding to use AMSR-E 6.9 GHz brightness temperature data and sea ice concentration using 89.0 GHz data. Comparison with the reference MY SIA from reference MY ice, indicates that the mean difference and root mean square （rms） difference of MY SIA derived from the algorithm of this study are 0.65×10^6 km^2 and 0.69×10^6 km^2 during January to March, -0.06×10^6 km^2 and 0.14×10^6 km^2during September to December respectively. Comparison with MY SIE obtained from weekly ice age data provided by University of Colorado show that, the mean difference and rms difference are 0.69×10^6 km^2 and 0.84×10^6 km^2, respectively. The developed algorithm proposed in this study has smaller difference compared with the reference MY ice and MY SIE from ice age data than the Wang＇s, Lomax＇ and NASA TEAM algorithms.

Retrieval of Atmospheric Horizontal Visibility by Statistical Regression from NOAA/AVHRR Satellite Data

Based on the atmospheric horizontal visibility data from forty-seven observational stations along the eastern coast of China near the Taiwan Strait and simultaneous NOAA/AVHRR multichannel satellite data during January 2001 to December 2002, the spectral characters associated with visibility were investigated. Successful retrieval of visibility from multichannel NOAA/AVHRR data was performed using the principal component regression (PCR) method. A sample of retrieved visibility distribution was discussed with a sea fog process. The correlation coefficient between the observed and retrieved visibility was about 0.82, which is far higher than the 99.[KG-*7]9% confidence level by statistical test. The rate of successful retrieval is 94.98% of the 458 cases during 2001-2002. The error distribution showed that high visibilities were usually under-estimated and low visibilities were over-estimated and the relative error between the observed and retrieved visibilities was about 21.4%.

Study of M-estimator Variational Retrieval Using Simulated Feng Yun-3A Data

This paper adopts satellite channel brightness temperature simulation to study M-estimator variational retrieval. This approach combines both the advantages of classical variational inversion and robust M-estimators. Classical variational inversion depends on prior quality control to elim- inate outliers, and its errors follow a Gaussian distribution. We coupled the M-estimators to the framework of classical variational inversion to obtain a M-estimator variational inversion. The cost function contains the M-estimator to guarantee the robustness to outliers and improve the retrieval re- sults. The experimental evaluation adopts Feng Yun-3A （FY-3A） simulated data to add to the Gaussian and Non-Gaussian error. The variational in- version is used to obtain the inversion brightness temperature, and temperature and humidity data are used for validation. The preliminary results demonstrate the potential of M-estimator variational retrieval.

Data driven composite shape descriptor design for shape retrieval with a VoR-Tree

We develop a data driven method(probability model) to construct a composite shape descriptor by combining a pair of scale-based shape descriptors. The selection of a pair of scale-based shape descriptors is modeled as the computation of the union of two events, i.e.,retrieving similar shapes by using a single scale-based shape descriptor. The pair of scale-based shape descriptors with the highest probability forms the composite shape descriptor. Given a shape database, the composite shape descriptors for the shapes constitute a planar point set.A VoR-Tree of the planar point set is then used as an indexing structure for efficient query operation. Experiments and comparisons show the effectiveness and efficiency of the proposed composite shape descriptor.

眩晕病中医古今文献检索系统构建研究

从数据来源、构建方法、技术架构及功能模块等方面介绍眩晕病中医古今文献检索系统的构建情况,该系统全面采集、整理了涉及眩晕病因病机、证候分类、方药治疗等方面的历代古籍及部分现代期刊文献,可以对眩晕文献数据进行快速数据挖掘分析及可视化展示,还实现了眩晕相关古今文献的互通检索,可以为临床医生和科研人员研究提供丰富的原始资料,为进一步开展眩晕病文献相关特色分析研究提供可靠的数据支持与信息保障。

A Transfer Function Design for Medical Volume Data Using a Knowledge Database Based on Deep Image and Primitive Intensity Profile Features Retrieval

Direct volume rendering(DVR)is a technique that emphasizes structures of interest(SOIs)within a volume visually,while simultaneously depicting adjacent regional information,e.g.,the spatial location of a structure concerning its neighbors.In DVR,transfer function(TF)plays a key role by enabling accurate identification of SOIs interactively as well as ensuring appropriate visibility of them.TF generation typically involves non-intuitive trial-and-error optimization of rendering parameters,which is time-consuming and inefficient.Attempts at mitigating this manual process have led to approaches that make use of a knowledge database consisting of pre-designed TFs by domain experts.In these approaches,a user navigates the knowledge database to find the most suitable pre-designed TF for their input volume to visualize the SOIs.Although these approaches potentially reduce the workload to generate the TFs,they,however,require manual TF navigation of the knowledge database,as well as the likely fine tuning of the selected TF to suit the input.In this work,we propose a TF design approach,CBR-TF,where we introduce a new content-based retrieval(CBR)method to automatically navigate the knowledge database.Instead of pre-designed TFs,our knowledge database contains volumes with SOI labels.Given an input volume,our CBR-TF approach retrieves relevant volumes(with SOI labels)from the knowledge database;the retrieved labels are then used to generate and optimize TFs of the input.This approach largely reduces manual TF navigation and fine tuning.For our CBR-TF approach,we introduce a novel volumetric image feature which includes both a local primitive intensity profile along the SOIs and regional spatial semantics available from the co-planar images to the profile.For the regional spatial semantics,we adopt a convolutional neural network to obtain high-level image feature representations.For the intensity profile,we extend the dynamic time warping technique to address subtle alignment differences between similar profiles(SOIs).Finally,we propose a two-stage CBR scheme to enable the use of these two different feature representations in a complementary manner,thereby improving SOI retrieval performance.We demonstrate the capabilities of our CBR-TF approach with comparison with a conventional approach in visualization,where an intensity profile matching algorithm is used,and also with potential use-cases in medical volume visualization.

基于多尺度注意力的生成式信息隐藏算法

针对现有生成式信息隐藏算法嵌入容量低且提取的秘密图像视觉质量欠佳的问题,提出基于多尺度注意力的生成式信息隐藏算法。首先,设计基于多尺度注意力的双编码-单解码生成器,载体图像与秘密图像的特征在编码端分两个支路独立提取,在解码端通过多尺度注意力模块进行融合,并利用跳跃连接为解码端提供不同尺度的细节特征,从而获得高质量的载密图像。其次,在U-Net结构的提取器中引入自注意力模块,以弱化载体图像特征、增强秘密图像深层特征,并利用跳跃连接弥补秘密图像细节特征,提高秘密信息提取的准确率;同时,多尺度判决器与生成器的对抗训练可以有效提升载密图像的视觉质量。实验结果表明,所提算法在嵌入容量为24 bpp的情况下,生成的载密图像峰值信噪比(PSNR)和结构相似性(SSIM)平均可达到40.93 dB和0.9883,且提取的秘密图像PSNR和SSIM平均可达到30.47 dB和0.9543。

面向医药卫生知识服务系统的学术知识图谱构建与应用研究

目的/意义探析知识服务系统中学术知识图谱的构建框架和方法,为开展基于科技大数据的新型知识服务提供参考。方法/过程提出面向知识服务系统的学术知识图谱构建框架及方案,以医药卫生知识服务系统为例,详细阐述构建步骤及构建过程中的关键技术。结果/结论基于多类型科技数据资源构建的学术知识图谱有助于更好地开展学术实体的关联挖掘与知识推理,提高科技数据组织管理效率,提供更高效的知识服务。

知识图谱技术实现流程及相关应用

目前,人类已经迈入信息时代的大门,大数据技术得到广泛化应用,信息数据成为一项宝贵的资源,通过知识图谱可以对海量数据进行整合、利用,挖掘高价值信息,以便为决策制定等多方面提供重要支持。基于此,该文围绕知识图谱的技术实现及相关应用展开深入、全面的探析。

基于本体库的知识抽取及图谱构建技术

该文首先介绍领域内本体库的组成架构,以及基础数据分析和WordNet节选的主要功能。随后,提出一种基于本体库的实体数据抽取技术,在不同的实体之间建立语义关系,为知识抽取做好铺垫。在实体信息抽取过程中,首先要判断网页是否在领域内,在确定网页属于领域后按照特定的标签划分网页内容,进而抽取出有价值的实体数据。将抽取到的实体数据存储到Neo4j数据库中,定期更新知识图谱内的数据。当需要调用数据时,可以从知识图谱中检索需要的数据,从而实现数据资源的整合利用,发挥数据的价值。

基于数据挖掘的智慧图书馆信息自动化检索系统设计

由于检索系统信息种类检索准确性低,提出基于数据挖掘的智慧图书馆信息自动化检索系统。优化中央控制器的开发板和芯片,选定NAND Flash芯片构建系统数据存储器。采用MAX232芯片完成信息传输电路。通过数据挖掘算法计算图书信息中的关键词相似性并进行滤波处理,使用数据挖掘算法中的关联规则算法优化信息分类模块,实现图书馆信息自动化检索系统。实验表明,系统信息检索精确度较高,有效提高了信息种类检索准确性。

基于随机森林算法的FY-4A地表入射太阳辐射空间订正

筛选时次、天顶角、FY-4A AGRI全圆盘地表入射太阳辐射(SSI)、云覆盖率(CFR)、云检测(CLM)以及云类型(CLT)产品构建特征向量,采用随机森林方法开展FY-4A SSI逐时产品的空间订正研究。结果表明、随机森林算法对于提升FY-4A SSI产品的空间分布精度具有明显作用,订正后相关系数、平均偏差、平均绝对误差均有不同程度的改善,且能很好地解决FY-4A SSI产品的过高估计及FY-4A SSI产品在太阳天顶角大于70°时无观测问题,有效提高FY-4A SSI产品在高纬度地区的可用性。

k-best维特比解耦合知识蒸馏的命名实体识别模型

为提升命名实体识别(NER)模型的性能,可采用知识蒸馏方法,但是传统知识蒸馏损失函数因内部存在的耦合关系会导致蒸馏效果较差。为了解除耦合关系,有效提升输出层特征知识蒸馏的效果,提出一种结合k-best维特比解码的解耦合知识蒸馏方法(kvDKD),该方法利用k-best维特比算法提高计算效率,能够有效提升模型性能。另外,基于深度学习的命名实体识别在数据增强时易引入噪声,因此提出了融合数据筛选和实体再平衡算法的数据增强方法,旨在减少因原数据集引入噪声和增强数据错误标注的问题,提高数据集质量,减少过度拟合。最后在上述方法的基础上,提出了一种新的命名实体识别模型NER-kvDKD。在MSRA、Resume、Weibo、CLUENER和CoNLL-2003数据集上的对比实验结果表明,该方法能够提高模型的泛化能力,同时也有效提高了学生模型性能。

半配对的多模态询问哈希方法

多模态哈希能够将异构的多模态数据转化为联合的二进制编码串。由于其具有低存储成本、快速的汉明距离排序的优点,已经在大规模多媒体检索中受到了广泛的关注。现有的多模态哈希方法假设所有的询问数据都具备完整的多种模态信息以生成它们的联合哈希码。然而,实际应用中很难获得全完整的多模态信息,针对存在模态信息缺失的半配对询问场景,该文提出一种新颖的半配对询问哈希(SPQH),以解决半配对的询问样本的联合编码问题。首先,提出的方法执行投影学习和跨模态重建学习以保持多模态数据间的语义一致性。然后,标签空间的语义相似结构信息和多模态数据间的互补信息被有效地捕捉以学习判别性的哈希函数。在询问编码阶段,通过学习到的跨模态重构矩阵为未配对的样本数据补全缺失的模态特征,然后再经习得的联合哈希函数生成哈希特征。相比最先进的基线方法,在Pascal Sentence,NUS-WIDE和IAPR TC-12数据集上的平均检索精度提高了2.48%。实验结果表明该算法能够有效编码半配对的多模态询问数据,取得了优越的检索性能。

模型约束与机器学习下的植物类胡萝卜素和叶绿素含量反演方法

叶绿素和类胡萝卜素含量是评价植物健康状况的一个重要指标。PROSPECT模型与机器学习耦合反演植被生化特性已得到广泛应用。但由于叶片方向半球反射率因子(DHRF)光谱和二向反射率因子(BRF)光谱之间的差异,耦合模型的应用范围受到限制。为此,以北美地区植物叶片光谱数据库(EcoSIS)作为实验数据集,提出PROSPECT模型作为机器学习的附加约束形成混合数据集,对此混合数据集利用连续小波变换(CWT)产生的小波系数谱和一阶导(FD)产生谱,提出三种全光谱域和VNIR光谱子域下的植物叶片叶绿素、类胡萝卜素的光谱特征变量筛选策略,即是:竞争性自适应重加权算法(CARS)、连续投影算法(SPA)和主成分分析法(PCA)。由此,基于上述2×2×3=12种不同光谱处理方法、特征提取方法组合,分别建立了植物叶片叶绿素和类胡萝卜素含量的人工神经网络(ANN)预测模型。进而开展了不同模型下的预测精度对比分析,结果表明:PROSPECT模型约束下的模拟数据一定程度增强了机器学习的训练集质量;经一阶导、小波变换处理的光谱能较好地减少DHRF模拟光谱和BRF实测光谱间的偏差,并在结合特征提取算法CARS后进一步提升了预测表现。在全光谱域下的FD+CARS组合对叶片叶绿素的反演效果最佳,测试集R2为0.8064,RMSE为2.9114;在VNIR光谱子域下的CWT+CARS组合对叶片类胡萝卜素最佳,测试集R2为0.7972,RMSE为0.4141。该方法可为研究人员从叶片BRF光谱及其他近端反射率图像更精确、高效地提取植物叶片生化特征提供参考。

基于检索增强的噪声标签细粒度图像分类方法

在互联网音视频内容分析的应用中,快速建立低标注代价的图像细粒度分类方法具有重要意义。由于类别间具有相似的外观特征,并且存在光照、视角、背景遮挡等干扰因素,细粒度图像分类面临类别数量多、类间差异性小,以及标注代价高、标签信噪比低等挑战。为改善在带有噪声标签的数据环境下海量图像细粒度分类的效果,提出一种基于检索增强的图像细粒度分类方法,在迭代清洗噪声标签的基础上,利用检索范式通过简单类别标注获取更具表达性的特征,提升分类器的识别能力,并在包含1 500个细粒度的食物类别和超过50万张图像的数据集上取得良好的效果。

异步BiSS-C协议的FPGA解码

为了在BiSS-C协议的处理器设备(Master,MA)时钟和现场可编程门阵列(Field Programmable Gate Array,FPGA)时钟异步的情况下获得BiSS-C协议传输的数据,提出一种异步时钟下FPGA解码BiSS-C协议的实现方法。首先,在FPGA中对BiSS-C传输的编码器数据进行16倍采样,用状态机解码BiSS-C数据,并对数据进行循环冗余校验(Cyclic Redundancy Check,CRC)。其次,在ModelSim中对FPGA软件进行仿真,对状态机的功能、FPGA软件的数据判读能力进行验证。最后,搭建光栅编码器数据采集系统,对FPGA的解码效果进行验证。仿真结果和试验结果表明:在异步时钟下FPGA能正确解码BiSS-C协议传输数据,解码得到的编码器角度位置误差不大于0.1'',误码率低、解码精度高。