基于规则定制的数据质量检查系统设计与实现

Design and implementation of a data quality inspection system based on rule customization

数据质量对大数据分析决策具有重要影响,数据业务主管部门经常组织数据会审以检查和整改数据质量。针对现有技术检查工具功能不够全面、合理的问题,提出了一种基于规则定制的数据质量检查系统,构建了系统总体技术架构,形成了数据质量检查任务模型,设计了数据质量检查软件,建立了数据质量检查知识图谱,加强了检查规则管理功能,实现了基础字段、常规查询、扩展流程、外部插件四类检查方法,引入了图像智能检查技术。通过实际应用验证,系统增强了数据审核印证方法手段,有效提升了数据会审工作效果,促进了数据质量综合治理。

Data quality has an important impact on big data analysis and decision-making.The competent department of data business often organizes data joint review to check and rectify data quality.Aiming at the problem that the functions of existing technical inspection tools are not comprehensive and reasonable enough,this paper proposes a data quality inspection system based on rule customization,constructs the overall technical architecture of the system,forms the data quality inspection task model,designs the data quality inspection software,establishes the data quality inspection knowledge graph,strengthens the inspection rule management function,and realizes four types of inspection methods,including the basic fields,conventional queries,extended processes and external plugins.It includes an external plugins of image intelligent inspection technology.Through application practical verification,the system can enhance the methods of data review and verification,effectively improve the effectiveness of data joint review work,and promot comprehensive management of data quality.