Web crawlers are an important part of modern search engines.With the development of the times,data has exploded and humans have entered a“big data era”.For example,Wikipedia carries the knowledge from all over the w...Web crawlers are an important part of modern search engines.With the development of the times,data has exploded and humans have entered a“big data era”.For example,Wikipedia carries the knowledge from all over the world,records the realtime news that occurs every day,and provides users with a good database of data,but because of the large amount of data,it puts a lot of pressure on users to search.At present,single-threaded crawling data can no longer meet the requirements of text crawling.In order to improve the performance and program versatility of single-threaded crawlers,a high-speed multi-threaded web crawler is designed to crawl the network hyper-scale text database.Multi-threaded crawling uses multiple threads to process web pages in parallel,combining breadth-first and depth-first algorithms to control web crawling.The practice project is based on the Python language to achieve multi-threaded optimization network hyper-large-scale text database-Wikipedia book crawling method,the project is inspired by the article on the Wikipedia article in the Big Data Digest public number.展开更多
首先分析潮流转移的原因及伴随的现象。其次讨论潮流转移区域以及区域界定,对传统广度优先遍历(breadth first search,BFS)算法进行改进,提出潮流转移影响区域的界定方法。对安全评估工作的理论基础——3个基本概念(模型量化、平均功率...首先分析潮流转移的原因及伴随的现象。其次讨论潮流转移区域以及区域界定,对传统广度优先遍历(breadth first search,BFS)算法进行改进,提出潮流转移影响区域的界定方法。对安全评估工作的理论基础——3个基本概念(模型量化、平均功率角和潮流转移灵敏度)分别进行定义。提出潮流转移模型及其灵敏度的表达式。提出安全评估的评估方法,建立安全评估的数学模型,最终得到安全评估的综合指标,并阐述了指标的使用。开发潮流转移灵敏度及安全评估程序,利用该程序对真实电网算例进行仿真验证。展开更多
基金This research is funded by the Open Foundation for the University Innovation Platform in the Hunan Province,grant number 16K013Hunan Provincial Natural Science Foundation of China,grant number 2017JJ2016+2 种基金2016 Science Research Project of Hunan Provincial Department of Education,grant number 16C0269.Accurate crawler design and implementation with a data cleaning function,National Students innovation and entrepreneurship of training program,grant number 201811532010.This research work is implemented at the 2011 Collaborative Innovation Center for Development and Utilization of Finance and Economics Big Data Property,Universities of Hunan Province.Open Foundation for the University Innovation Platform in the Hunan Province,grant number 16K013Hunan Provincial Natural Science Foundation of China,grant number 2017JJ20162016 Science Research Project of Hunan Provincial Department of Education,grant number 16C0269.This research work is implemented at the 2011 Collaborative Innovation Center for Development and Utilization of Finance and Economics Big Data Property,Universities of Hunan Province.Open project,grant number 20181901CRP03,20181901CRP04,20181901CRP05.
文摘Web crawlers are an important part of modern search engines.With the development of the times,data has exploded and humans have entered a“big data era”.For example,Wikipedia carries the knowledge from all over the world,records the realtime news that occurs every day,and provides users with a good database of data,but because of the large amount of data,it puts a lot of pressure on users to search.At present,single-threaded crawling data can no longer meet the requirements of text crawling.In order to improve the performance and program versatility of single-threaded crawlers,a high-speed multi-threaded web crawler is designed to crawl the network hyper-scale text database.Multi-threaded crawling uses multiple threads to process web pages in parallel,combining breadth-first and depth-first algorithms to control web crawling.The practice project is based on the Python language to achieve multi-threaded optimization network hyper-large-scale text database-Wikipedia book crawling method,the project is inspired by the article on the Wikipedia article in the Big Data Digest public number.
文摘首先分析潮流转移的原因及伴随的现象。其次讨论潮流转移区域以及区域界定,对传统广度优先遍历(breadth first search,BFS)算法进行改进,提出潮流转移影响区域的界定方法。对安全评估工作的理论基础——3个基本概念(模型量化、平均功率角和潮流转移灵敏度)分别进行定义。提出潮流转移模型及其灵敏度的表达式。提出安全评估的评估方法,建立安全评估的数学模型,最终得到安全评估的综合指标,并阐述了指标的使用。开发潮流转移灵敏度及安全评估程序,利用该程序对真实电网算例进行仿真验证。
