Python环境下利用Selenium与JavaScript逆向技术爬虫研究

Research on Crawler Using Selenium and JavaScript Reverse Technology in Python Environment

针对使用调试检测、数据加密等技术的网站,解析工具Beautiful Soup难以对网页进行解析爬取数据。本研究基于Python环境,结合JavaScript逆向技术、Beautiful Soup网页解析等网络爬虫技术,利用中间人攻击工具Mitmproxy(man-in-the-middle attack proxy)在本地指定设备端口开启本地代理,拦截并修改网页响应。同时,运用Web自动化工具Selenium来启动浏览器,设置使用代理服务器,连接到本地Mitmproxy代理,访问被修改的响应网页进行网页调试和解析,并对加密数据进行还原,解决网络爬虫中调试检测和数据加密难题,从而爬取数据。

For websites that use debugging detection,data encryption and other technologies,the parsing tool beautiful soup is difficult to parse web pages and crawl data Based on the python environment,combined with JavaScript reverse technology,beautiful soup web page parsing and other web crawler technologies,this study uses the man in the middle attack proxy(mitmproxy)to open the local proxy at the local designated device port,intercept and modify the web page response,and uses the web automation tool selenium to start the browser,set up the proxy server and connect to the local mitmproxy proxy.Visit the modified middle note to debug and analyze the web page,restore the encrypted data,solve the problem of debugging and data encryption in web crawler,and then crawl data.