基于爬虫的数据监控系统

Spider-Based Data Monitor System

近年来,随着互联网技术的快速发展,云计算服务开始全面普及,大型互联网公司以及中小型企业已经开始提供自己的云计算服务,中国科学院也提供了云计算服务平台——中科院"科技云".本文针对中国科学院"科技云"项目的实际需求,参考已有商业云监控系统的功能和运行模式,设计并实现了一种基于爬虫的数据监控系统.该系统相比商业云监控系统,支持URL(布尔)类型数据监测的同时,增加了对数值类型、文本类型的监测,更好的支持第三方服务的监测,并实现了服务故障警报,监测数据可视化以及监测数据分布式存储.

In recent years, with the rapid development of Internet technology, cloud computing services begin to widely spread. Large-scale Internet companies, small and medium enterprises have begun to provide their own cloud computing services. The Chinese Academy of Sciences also provides a cloud computing service platform-CAS ＂science-cloud＂.Concerned with the actual needs of the ＂Science Cloud＂ project of the Chinese Academy of Sciences, this paper designs and implements a spider-based data monitoring system referencing the function and operation mode of the existing commercial cloud monitoring systems. Compared with the commercial cloud monitoring system, this system supports the monitoring of URL（Boolean） type data, adds the monitoring of numeric types and text types, supports better monitoring of the third party services, and realizes service failure alarm and monitoring data Visualization and monitoring data distributed storage.