基于深度学习的数据接入方法研究

Research on Data Ingestion Method Based on Deep Learning

大数据中心在接入多源异构数据时,需要分析处理不同标准数据的标准转换问题。以往这项工作大都依赖人工分析解决。近几年快速发展的深度学习技术具有优异的特征提取和分类能力,文章基于深度学习技术设计了一套将数据形态模型和语义模型相结合的数据接入匹配算法,并利用部分样例数据进行测试验证。结果表明,该方法能够达到一定的准确率水平,并具备持续提升的空间,可以在实际应用中替代部分人工工作。

When ingesting multi-source heterogeneous data,big data centers need to analyze standard conversion problems that deal with data of different standards.In the past,most of this work relied on manual analysis.In recent years,the fast-developing deep learning technology has excellent feature extraction and classification ability.This study designed a data access matching algorithm based on deep learning technology which combined data morphological model and semantic model,and completed verification by using sample data.The results show that the proposed method could achieve a reasonable level of accuracy,and has potential for future improvement,which could replace some manual work in many practical applications.