摘要
Spark SQL在获取Spark核心的计算结果后,需要进行一些格式转化和数据拷贝才能正式输出,将造成内存中相同或近似相同的数据有多个拷贝,浪费了内存资源,也降低了性能,直接影响了用户响应和结果存储容量,并且这种影响会随着输出结果的增大而增大。为此,文中提出了基于Spark大数据处理平台的创新查询方法,以实现查询首结果立即输出,极大缩短客户响应时间。
After Spark SQL obtains the core calculation results of the Spark,it is necessary to perform some format conversion and data copy before the official output.It will cause multiple copies of the same or nearly the same data in the memory.It is a waste of memory resources,reducing the performance,and directly affecting the response capacity and the stored results.And this influence will increase with the increase of the output result.An innovative query method based on the Spark big data processing platform is proposed.The method is used to achieve immediate output of the first result of the query,thus greatly reducing customer response time.
作者
张海峰
魏可欣
ZHANG Haifeng;WEI Kexin(China Information Consulting&Designing Institute Co.Ltd.,Nanjing 210019,China;School of Business,Nanjing University,Nanjing 210093,China;School of Politics and Public Administration,Soochow University,Suzhou 215123,China)
出处
《南京邮电大学学报(自然科学版)》
北大核心
2021年第4期82-90,共9页
Journal of Nanjing University of Posts and Telecommunications:Natural Science Edition