Recent advances in big and streaming data systems have enabled real-time analysis of data generated by Internet of Things(IoT)systems and sensors in various domains.In this context,many applications require integratin...Recent advances in big and streaming data systems have enabled real-time analysis of data generated by Internet of Things(IoT)systems and sensors in various domains.In this context,many applications require integrating data from several heterogeneous sources,either stream or static sources.Frameworks such as Apache Spark are able to integrate and process large datasets from different sources.However,these frameworks are hard to use when the data sources are heterogeneous and numerous.To address this issue,we propose a system based on mediation techniques for integrating stream and static data sources.The integration process of our system consists of three main steps:configuration,query expression and query execution.In the configuration step,an administrator designs a mediated schema and defines mapping between the mediated schema and local data sources.In the query expression step,users express queries using customized SQL grammar on the mediated schema.Finally,our system rewrites the query into an optimized Spark application and submits the application to a Spark cluster.The results are continuously returned to users.Our experiments show that our optimizations can improve query execution time by up to one order of magnitude,making complex streaming and spatial data analysis more accessible.展开更多
The problem of storage and querying of large volumes of spatial grids is an issue to solve.In this paper,we propose a method to optimize queries to aggregate raster grids stored in databases.In our approach,we propose...The problem of storage and querying of large volumes of spatial grids is an issue to solve.In this paper,we propose a method to optimize queries to aggregate raster grids stored in databases.In our approach,we propose to estimate the exact result rather than calculate the exact result.This approach reduces query execution time.One advantage of our method is that it does not require implementing or modifying functionalities of database management systems.Our approach is based on a new data structure and a specific model of SQL queries.Our work is applied here to relational data warehouses.展开更多
基金financed by the French government IDEX-ISITE initiative 16-IDEX-0001(CAP 20-25)the PhD is funded by the European Regional Development Fund(FEDER).
文摘Recent advances in big and streaming data systems have enabled real-time analysis of data generated by Internet of Things(IoT)systems and sensors in various domains.In this context,many applications require integrating data from several heterogeneous sources,either stream or static sources.Frameworks such as Apache Spark are able to integrate and process large datasets from different sources.However,these frameworks are hard to use when the data sources are heterogeneous and numerous.To address this issue,we propose a system based on mediation techniques for integrating stream and static data sources.The integration process of our system consists of three main steps:configuration,query expression and query execution.In the configuration step,an administrator designs a mediated schema and defines mapping between the mediated schema and local data sources.In the query expression step,users express queries using customized SQL grammar on the mediated schema.Finally,our system rewrites the query into an optimized Spark application and submits the application to a Spark cluster.The results are continuously returned to users.Our experiments show that our optimizations can improve query execution time by up to one order of magnitude,making complex streaming and spatial data analysis more accessible.
基金This work is funded by Auvergne region,Feder,Agaetis,and Irstea.
文摘The problem of storage and querying of large volumes of spatial grids is an issue to solve.In this paper,we propose a method to optimize queries to aggregate raster grids stored in databases.In our approach,we propose to estimate the exact result rather than calculate the exact result.This approach reduces query execution time.One advantage of our method is that it does not require implementing or modifying functionalities of database management systems.Our approach is based on a new data structure and a specific model of SQL queries.Our work is applied here to relational data warehouses.
