CMS实验元数据发现的数据聚集系统

Data Aggregation System for Meta-data Discovery in CMS Experiment

在大型强子对撞机上的紧凑繆子螺线管探测器实验,具有数据量大(PB级规模)、数据类型复杂与数据地理上全球分布的特点。记录上述数据的元数据达到TB级的规模,并且以不同的格式保存在不同的关系和非关系数据源中,通过在这些异构数据源上添加一个缓存层的方法,实现一个提供精确的关键词查询的数据聚集系统。根据多重映射和聚集的方式支持用户的查询,并利用有效的缓存管理策略来提升查询的命中率。实验结果表明,该系统能够通过缓存的方式响应超过70%的用户查询,具有良好的查询性能。

The Compact Muon Solenoid（CMS） experiment on the Large Hadron Collider（LHC） produces PBs of physics data. Those data not only are huge on volume, but also have complex types and being distributed all over the world. Therefore the recta-data about how to organize those physics data reach TB in size. Those meta-data are kept in different relational or non-relational data sources in different format. In order to meet the data discovery requirement, it is important to provide a unified query interface. By adding a caching layer upon those data sources, this paper implements a data aggregation system, which provides precise keyword style search interface. It demonstrates how to support user queries by multiple mapping and aggregation, and how to manage the cache efficiently. Experimental result shows that more than 70% user queries can be answered by the cache system, and it has well queries performance.