面向物流服务的海量日志实时流处理平台

Real-Time Stream Processing Platform for Massive Logs of Logistics Services

随着电商平台的快速发展,物流行业增长迅猛,其中物流服务平台的访问日志能够反映用户的行为规律,从而挖掘潜藏信息助力物流服务平台优化业务已至关重要.目前,针对于此类大规模日志数据处理提出了更高的实时性需求,本文综合考量多种实时计算的流处理框架、大规模存储数据库以及日志采集工具等,选取Flume及Kafka作为日志采集工具与消息队列,并利用Flink及HBase进行流数据实时计算以及大规模数据存储.同时,对平台设计了数据去重、异常告警、容错策略以及负载调度的功能.经实验测试证明,本处理平台可以有效处理物流服务平台的日志数据,具有较强的创新思路以及实际价值.

With the rapid development of e-commerce platforms, the logistics industry is at a high rate of growth. The access logs of the logistics service platform can reflect user behavior, so it is very important to tap the hidden information to help the logistics service platform optimize the business. At present, higher real-time requirements are imposed on large-scale log data processing. This study comprehensively considers a variety of stream processing frameworks capable of real-time computing, large-scale storage databases, log collection tools, etc. It chooses Flume and Kafka as the log collection tools and message queues and uses Flink and HBase for real-time calculation of streaming data and large-scale data storage. At the same time, the functions including data deduplication, abnormal alarms, fault tolerance strategy, and load scheduling are designed for the platform. Experimental tests have proved that this processing platform can efficiently process log data of the logistics service platform, with innovative ideas and practical value.