基于Storm的实时用户日志管理系统

Real-time User Log Management System Based on Storm

研究并开发了具有实时响应的大数据用户日志管理分析平台。采用Storm流式计算框架,完成用户日志分析的实时处理平台架构;采用Elastic Search作为实时搜索引擎,完成实时响应处理;使用Flume、Kafka技术完成海量日志采集、聚合和传输等功能。以某城市监狱管理系统的用户日志信息为实验数据,系统取得了良好的实验效果。为同时处理海量数据和实时响应用户请求的系统提供了可行性解决方案。

The paper proposed an approach to research and develop a user log data management and analysis platform with real-time response,utilizing streaming computational framework named Storm as the realtime processing platform of log analysis;using Elastic Search as a real-time search engine to achieve realtime responses;using Flume and Kafka technologies to complete the massive log collection,aggregation and transmission functions.Taking the experimental data from an information management system of a city prison,the proposed platform achieved good experimental results.The paper provides a feasible solution to handle huge amounts of data offline and real-time responses to users＇ requests at the same time,which has a good prospect for the future applications.