新冠肺炎疫情信息累积更新采集系统的设计与实现

Design and Implementation of COVID-19 Information Cumulation Update Collection System

突如其来的新冠肺炎疫情打乱了人们的正常生活和工作节奏,面对各种不同来源的新冠肺炎疫情数据,如何及时获取疫情信息这对人们的工作、生活有很大帮助。本文使用Python编程语言,结合Scrapy框架,基于Splash对页面中的JavaScript代码进行渲染的技术,设计并实现了一种新冠肺炎疫情信息累积更新采集系统。实验表明,该系统能同时采集国内31个省(自治区、直辖市)官方网站发布的疫情数据,并能累积更新。同时,将采集的数据清洗后存入数据库中,为下游任务疫情数据可视化、趋势预测等提供数据支撑。

The sudden COVID-19 epidemic has disrupted people′s normal life and work rhythm.In the face of COV⁃ID-19 epidemic data,how to timely obtain the epidemic information is of great help to people′s work and life.This paper uses python programming language,combined with scrapy framework,based on splash to render the JavaScript code in the page,designs and implements a collection system for cumulative update of new epidemic information.The experimental results show that the system can simultaneously collect the epidemic data published by official websites of 31 provinces(autonomous regions and municipalities)at the same time,and accumulate and update them.At the same time,the collected data are stored in the database after clearing the method,which provides data support for the visualization and trend prediction of epidemic situation data in downstream tasks.