基于Selenium框架实现Boss直聘网数据爬取与分析

Data Crawling and Analysis of Boss Employment Network Based on Selenium Framework

本文基于Selenium框架绕过网站设置的反爬机制,实现Boss直聘网爬虫工程师岗位的自动化爬取,将爬取后的数据存储到csv文件,之后利用pandas库对岗位数据进行数据分析并将分析结果可视化展示,在《数据爬取》课程中以此为教学案例,可以提升学生的专业和职业认同感,同时为学生未来就业提供了参考。

Based on selenium framework,this paper passes the anti crawling mechanism set by the website,realizes the automatic crawling of the web crawler engineer,then stores the crawled data to CSV file and analyzes the crawled data by pandas,at last,visual display of analysis results is carries out.Taking this as a teaching case in the course of data crawling,it can enhance students?professional and professional identity,at the same time,it provides a reference for students?future employment.