基于Zookeeper的计费系统多级故障隔离方案

Multi level fault isolation scheme of billing system based on zookeeper

本课题针对移动通信BOSS业务支撑系统领域,在系统维护中存在的当内存库出现故障时故障节点难以得到高效隔离和有关应用恢复效率低的痛点,创新地设计出一种内存库不可访问场景下,业务故障隔离的使用方法,该方法创新关键点为:①数据节点状态集中管理:通过ZooKeeper监控并记录各个内存库节点的状态,并通过多级连接尝试,保障结果准确性,避免检测误报;②业务分级逐步恢复:计费系统各个业务模块,根据状态变更情况,指示业务进程依次对故障期间的数据进行逐步恢复;③跨节点故障隔离:应用及时获取各内存库节点状态,保障跨节点业务及时作出消息反馈,减少等待时间,甚至是多节点的业务中断。

This topic innovatively designs a service fault isolation method in the inaccessible memory scenario.The key points of this method are as follows:①Centralized management of data node status:monitoring and recording the status of each memory node through ZooKeeper,and ensuring the accuracy of the results and avoiding false positives through multi-level connection attempts;②Gradual recovery of business classification:each business module of the charging system instructs the business process to gradually recover the data during the failure period according to the state change;③Cross-node fault isolation:the application obtains the node status of each memory bank in time,which ensures the cross-node business to make timely message feedback,reduces the waiting time,and even interrupts the multi-node business.