基于动态解析方法的多线程数据高效抓取仿真

Multi-threaded Data Efficient Crawling Simulation Based on Dynamic Analysis Method

目前数据获取方法存在准确性和安全性较差的问题,为此提出基于动态解析的多线程数据高效抓取方法。依据多线程数据异常告警空间定义,利用混沌粒子群算法对异常告警进行聚类,初始化粒子群,设置聚类数量和粒子数量,针对各粒子随机指派一个类别,并获取聚类中心,利用聚类评判准则中最小均方根误差准则实现异常告警划分聚类,使数据抓取过程中能够有效避开异常数据。基于数据抓取安全性分析,通过词项的共现思想,针对多线程数据构建动态解析网络,依据网络边权重随时间延长而呈线性衰减的理念,通过加权度数获取多线程数据特征权重,抓取其中权重值较大的数据。实验表明,该方法抓取精度高,安全性好。

At present, the accuracy and security of data capture method is low. Therefore, this paper puts forward a method to efficiently fetch multi-thread data based on dynamic analysis. According to the definition of abnormal alarm space of multi-thread data, we used the chaotic particle swarm algorithm to cluster abnormal alarm and initialize particle swarm. Then, we set the number of clustering and particles. After that, we randomly assigned a class for each particle and obtained the clustering center. Moreover, we used the minimum root mean square error criterion in clustering evaluation criterion to realize the division and clustering of abnormal alarm, so that the abnormal data could be avoided effectively during the data capture. Based on the security analysis of data capture and the co-occurrence idea of lexical item, we built the dynamic analytic network according to multi-thread data. According to the idea that the weight of network edge decreased linearly with the time expanding, we obtained the multi - thread data feature weight. Finally, we captured the data with the larger weight value. Simulations prove that this method has high data capture precision and good security.