MapReduce is a popular framework for large- scale data analysis. As data access is critical for MapReduce＇s performance, some recent work has applied different storage models, such as column-store or PAX-store, to Ma...MapReduce is a popular framework for large- scale data analysis. As data access is critical for MapReduce＇s performance, some recent work has applied different storage models, such as column-store or PAX-store, to MapReduce platforms. However, the data access patterns of different queries are very different. No storage model is able to achieve the optimal performance alone. In this paper, we study how MapReduce can benefit from the presence of two different column-store models - pure column-store and PAX-store. We propose a hybrid storage system called hybrid columnstore （HC-store）. Based on the characteristics of the incoming MapReduce tasks, our storage model can determine whether to access the underlying pure column-store or PAX-store. We studied the properties of the different storage models and create a cost model to decide the data access strategy at runtime. We have implemented HC-store on top of Hadoop. Our experimental results show that HC-store is able to outperform PAX-store and column-store, especially when confronted with diverse workload.展开更多
基金Acknowledgements This work was sponsored by the National Key Basic Research Program of China （973 Program） （2014CB340403）, the National Natural Science Foundation of China （Grant Nos. 61170013, 61272138 and 61232007）.
文摘MapReduce is a popular framework for large- scale data analysis. As data access is critical for MapReduce＇s performance, some recent work has applied different storage models, such as column-store or PAX-store, to MapReduce platforms. However, the data access patterns of different queries are very different. No storage model is able to achieve the optimal performance alone. In this paper, we study how MapReduce can benefit from the presence of two different column-store models - pure column-store and PAX-store. We propose a hybrid storage system called hybrid columnstore （HC-store）. Based on the characteristics of the incoming MapReduce tasks, our storage model can determine whether to access the underlying pure column-store or PAX-store. We studied the properties of the different storage models and create a cost model to decide the data access strategy at runtime. We have implemented HC-store on top of Hadoop. Our experimental results show that HC-store is able to outperform PAX-store and column-store, especially when confronted with diverse workload.
