基于网络爬虫的招聘信息可视化系统设计与实现

Design and Implementation of Recruitment Information Visualization System Based on Web Crawler

近些年,随着互联网行业快速发展,社会信息量呈爆炸式增长,特别是受新冠肺炎疫情的影响,网络招聘规模急剧增长,大量的招聘网站出现,每天都会有海量的招聘信息发布在网络上。如何深层次地发掘出招聘数据源价值,建立可视化分析系统,帮助高职院校客观且全面地了解相关企业招聘实际需求,有针对性地调整专业学科建设以及人才培养方案的制定,是当前研究的热点和难点。系统通过网络爬虫技术对大专院校计算机专业相关岗位进行爬取,并利用大数据技术对数据进行清洗去噪以及MapReduce Top-K算法统计出十大热门岗位和薪水数据,最后借助商业级图表数据库Echarts将分析结果可视化输出,为大专院校的计算机人才培养提供更加客观且全面的数据支撑。

In recent years,as the Internet industry develops rapidly,information has been growing dramatically,especially due to the Covid-19,a large number of recruitment websites have appeared,and a huge amount of recruitment information is released on the Internet every day.How to deeply discover the value of the recruitment data source and establish a visual analysis system to help higher vocational colleges and universities objectively and comprehensively understand the actual needs of the relevant enterprises'recruitment,and purposefully adjust the construction of professional disciplines as well as the development of talent cultivation programs is the hot topic and difficulty of the current research.The system crawls the relevant positions of computer majors in colleges and universities through network crawler technology,and uses big data technology to clean the data as well as the MapReduce Top-K algorithm to statistically identify the top ten popular positions and their salary data,and finally outputs the results of the analysis through the commercial-grade charting database Echarts,which can provide more objective and comprehensive data support for the cultivation of computer talents in colleges and universities.