一种基于Spark大数据处理平台的查询方法

Query method based on Spark big data processing platform

Spark SQL在获取Spark核心的计算结果后,需要进行一些格式转化和数据拷贝才能正式输出,将造成内存中相同或近似相同的数据有多个拷贝,浪费了内存资源,也降低了性能,直接影响了用户响应和结果存储容量,并且这种影响会随着输出结果的增大而增大。为此,文中提出了基于Spark大数据处理平台的创新查询方法,以实现查询首结果立即输出,极大缩短客户响应时间。

After Spark SQL obtains the core calculation results of the Spark,it is necessary to perform some format conversion and data copy before the official output.It will cause multiple copies of the same or nearly the same data in the memory.It is a waste of memory resources,reducing the performance,and directly affecting the response capacity and the stored results.And this influence will increase with the increase of the output result.An innovative query method based on the Spark big data processing platform is proposed.The method is used to achieve immediate output of the first result of the query,thus greatly reducing customer response time.