Meteorological Sensor Data Storage Mechanism Based on TimescaleDB and Kafka

The scale of meteorological sensor data increases at TB level every week. Traditional relational database is inefficient in storing and processing such data and cannot satisfy many soft requirements. However, the heterogeneity and diversity of the numerous existing NoSQL systems impede the well-informed comparison and selection of a data store appropriate for a given application context. Implementing a meteorological sensor data storage mechanism is a key challenge. Therefore, a meteorological sensor data storage mechanism based on TimescaleDB and Kafka is proposed. In this solution, meteorological sensor data was acquired and transmitted by Kafka and was sent to TimescaleDB for storage and analysis. Based on simulated meteorological sensor dataset, it compared the solution with other NoSQL stores such as Redis, MongoDB, Cassandra, HBase and Riak TS. The experimental results show that the storage mechanism proposed is superior in the storage and processing of massive meteorological sensor data.