Learned Distributed Query Optimizer:Architecture and Challenges

The query processing in distributed database management systems(DBMS)faces more challenges,such as more operators,and more factors in cost models and meta-data,than that in a single-node DMBS,in which query optimization is already an NP-hard problem.Learned query optimizers(mainly in the single-node DBMS)receive attention due to its capability to capture data distributions and flexible ways to avoid hard-craft rules in refinement and adaptation to new hardware.In this paper,we focus on extensions of learned query optimizers to distributed DBMSs.Specifically,we propose one possible but general architecture of the learned query optimizer in the distributed context and highlight differences from the learned optimizer in the single-node ones.In addition,we discuss the challenges and possible solutions.

Query Optimization Framework for Graph Database in Cloud Dew Environment

The query optimizer uses cost-based optimization to create an execution plan with the least cost,which also consumes the least amount of resources.The challenge of query optimization for relational database systems is a combinatorial optimization problem,which renders exhaustive search impossible as query sizes rise.Increases in CPU performance have surpassed main memory,and disk access speeds in recent decades,allowing data compression to be used—strategies for improving database performance systems.For performance enhancement,compression and query optimization are the two most factors.Compression reduces the volume of data,whereas query optimization minimizes execution time.Compressing the database reduces memory requirement,data takes less time to load into memory,fewer buffer missing occur,and the size of intermediate results is more diminutive.This paper performed query optimization on the graph database in a cloud dew environment by considering,which requires less time to execute a query.The factors compression and query optimization improve the performance of the databases.This research compares the performance of MySQL and Neo4j databases in terms of memory usage and execution time running on cloud dew servers.

A Distributed DBMS Based Dynamic Programming Method for Query Optimization

Dynamic programming(DP) is an effective query optimization approach to select an appropriate join order for relational database management system(RDBMS) in multi-table joins. This method was extended and made available in distributed DBMS(D-DBMS). The structure of this optimal solution was firstly characterized according to the distributing status of tables and data, and then the recurrence relations between a problem and its sub-problems were recursively defined. DP in D-DBMS has the same time-complexity with that in centralized DBMS, while it has the capability to solve a much more sophisticated optimal problem of multi-table join in D-DBMS. The effectiveness of this optimal strategy has been proved by experiments.

An adaptive range-query optimization technique with distributed replicas

Replication is an approach often used to speed up the execution of queries submitted to a large dataset.A compile-time/run-time approach is presented for minimizing the response time of 2-dimensional range when a distributed replica of a dataset exists.The aim is to partition the query payload(and its range) into subsets and distribute those to the replica nodes in a way that minimizes a client's response time.However,since query size and distribution characteristics of data(data dense/sparse regions) in varying ranges are not known a priori,performing efficient load balancing and parallel processing over the unpredictable workload is difficult.A technique based on the creation and manipulation of dynamic spatial indexes for query payload estimation in distributed queries was proposed.The effectiveness of this technique was demonstrated on queries for analysis of archived earthquake-generated seismic data records.

Multi - weighted tree based query optimization method for parallel relational database systems

The author investigates the query optimization problem for parallel relational databases. A multi - weighted tree based query optimization method is proposed. The method consists of a multi - weighted tree based parallel query plan model, a cost model for parallel qury plans and a query optimizer. The parallel query plan model is the first one to model all basic relational operations, all three types of parallelism of query execution, processor and memory allocation to operations, memory allocation to the buffers between operations in pipelines and data redistribution among processors. The cost model takes the waiting time of the operations in pipelining execution into consideration and is computable in a bottom - up fashion. The query optimizer addresses the query optimization problem in the context of Select - Project - Join queries that are widely used in commercial DBMSs. Several heuristics determining the processor allocation to operations are derived and used in the query optimizer. The query optimizer is aware of memory resources in order to generate good - quality plans. It includes the heuristics for determining the memory allocation to operations and buffers between operations in pipelines so that the memory resourse is fully exploit. In addition, multiple algorithms for implementing join operations are consided in the query optimizer. The query optimizer can make an optimal choice of join algorithm for each join operation in a query. The proposed query optimization method has been used in a prototype parallel database management system designed and implemented by the author.

An Efficient Algorithm for Query Transformation in Semantic Query Optimization

Semantic query optimization (SQO) is comparatively a recent approach for the transformation of given query into equivalent alternative query using matching rules in order to select an optimal query based on the costs of executing alternative queries. The key aspect of the algorithm proposed here is that previous proposed SQO techniques can be considered equally in the uniform cost model, with which optimization opportunities will not be missed. At the same time, the authors used the implication closure to guarantee that any matched rule will not be lost. The authors implemented their algorithm for the optimization of decomposed sub-query in local database in Multi-Database Integrator (MDBI), which is a multidatabase project. The experimental results verify that this algorithm is effective in the process of SQO.

A solution of spatial query processing and query optimization for spatial databases

Recently, attention has been focused on spatial query language which is used to query spatial databases. A design of spatial query language has been presented in this paper by extending the standard relational database query language SQL. It recognizes the significantly different requirements of spatial data handling and overcomes the inherent problems of the application of conventional database query languages. This design is based on an extended spatial data model, including the spatial data types and the spatial operators on them. The processing and optimization of spatial queries have also been discussed in this design. In the end, an implementation of this design is given in a spatial query subsystem.

Immune Algorithm For Document Query Optimization

To efficiently retrieve relevant document from the rapid proliferation of large information collections, a novel immune algorithm for document query optimization is proposed. The essential ideal of the immune algorithm is that the crossover and mutation of operator are constructed according to its own characteristics of information retrieval. Immune operator is adopted to avoid degeneracy. Relevant documents retrieved are merged to a single document list according to rank formula. Experimental results show that the novel immune algorithm can lead to substantial improvements of relevant document retrieval effectiveness.

Optimizing Resource Allocation Framework for Multi-Cloud Environment

Cloud computingmakes dynamic resource provisioning more accessible.Monitoring a functioning service is crucial,and changes are made when particular criteria are surpassed.This research explores the decentralized multi-cloud environment for allocating resources and ensuring the Quality of Service(QoS),estimating the required resources,and modifying allotted resources depending on workload and parallelism due to resources.Resource allocation is a complex challenge due to the versatile service providers and resource providers.The engagement of different service and resource providers needs a cooperation strategy for a sustainable quality of service.The objective of a coherent and rational resource allocation is to attain the quality of service.It also includes identifying critical parameters to develop a resource allocation mechanism.A framework is proposed based on the specified parameters to formulate a resource allocation process in a decentralized multi-cloud environment.The three main parameters of the proposed framework are data accessibility,optimization,and collaboration.Using an optimization technique,these three segments are further divided into subsets for resource allocation and long-term service quality.The CloudSim simulator has been used to validate the suggested framework.Several experiments have been conducted to find the best configurations suited for enhancing collaboration and resource allocation to achieve sustained QoS.The results support the suggested structure for a decentralized multi-cloud environment and the parameters that have been determined.

面向分布式数据库的算子并行优化策略

随着网络技术的不断发展,数据规模呈现爆发式增长,使得传统的单机数据库逐步被分布式数据库所取代。分布式数据库采用节点协同工作方式解决了大规模数据存储问题,但由于增加了节点间通信开销,查询效率却不如单机数据库。分布式架构下,存储节点的数据仅用作多备份的冗余,为系统故障时提供数据恢复,并未被利用起来改善查询效率。针对上述问题,提出了一种面向分布式数据库的算子并行优化策略,通过对关键物理算子进行拆分,将拆分后的子请求均匀分配到存储层多个节点,由多个节点并行处理,从而减少查询响应时间。上述策略已经在分布式数据库CBase上进行了应用,实验表明,提出的并行优化策略可显著缩短SQL请求查询时间,并提高系统资源利用率。

基于查询特征表示学习的联邦复杂查询基数估计

准确的基数估计是实现最佳查询计划的关键因素,现有方法大多基于深度学习来解决基数估计问题。然而,这种基于RDF图模式的方法专注于具有特定拓扑结构的简单查询,适用范围有限,缺乏对现实场景中频繁使用的复杂类查询的支持。为了解决以上问题,提出一种基于查询特征表示学习的联邦复杂查询基数估计模型。该模型主要处理带有FILTER或DISTINCT关键字的复杂查询,使用新提出的FILTER查询特征化方法将SPARQL查询表示为特征向量,通过模型预测查询基数。同时使用模型预测DISITINCT查询中唯一行比率。在LUBM数据集上的实验表明,与最先进的基数估计方法相比,该模型在估计质量上表现优异,平均估计误差中位数可达1.16,并对多连接查询的基数估计表现出潜力和可扩展性。

企业信息集成中XQuery引擎的查询优化

通过分析研究已有的XQuery引擎和XQ uery查询优化的方法,从减少编译时间和执行时间角度出发,提出了一种基于XQuery语句自身优化的新型综合性优化方法,经过XQuery中某常见查询进行仿真实验,可以说明该新型综合优化方法有较强的优势,可以大大地提高XQuery引擎的查询效率。

面向用户偏好的动态网页数据交互式查询算法

为提高网页数据查询速度、精度及工作效率,提出一种面向用户偏好的动态网页数据交互式查询算法.首先,构建用户偏好模型,增加偏好组合的演化个体适应性,综合计算适配值;其次,为防止数据冗余和重复,基于兴趣相似性,分离相似度高的查询数据和重复数据,识别出网络数据的性质;最后,利用粒子群优化算法寻找最优的动态网页数据交互式查询方案.实验结果表明:在数据集基数影响下,该算法的查询结果集质量在0.95以上;在查询最大维数影响下,该算法的查询结果集质量在0.96以上,表明其查询使用时间短、结果集精度高、自适应能力强.

基于机器学习的基数估计技术综述

基数估计是数据库关系系统查询优化器的基础和核心.随着人工智能技术的发展,其在数据处理、提取数据之间的关系等方面显现出优越的性能.近年来,基于机器学习的基数估计技术取得了显著的进展,受到了学术界的广泛关注.首先总结了基于机器学习的技术估计技术的发展现状,其次给出了基数估计的相关概念及其特征编码技术.接着建立了基数估计技术的分类体系.在此基础上,进一步将基于机器学习的基数估计技术细分为查询驱动、数据驱动和混合模型这3类基数估计技术.然后重点分析了每一类技术的建模流程、典型技术和模型特点,并对其在SQL和NoSQL中的应用进行了分析和总结.最后讨论了基于机器学习的基数估计技术面临的挑战和未来的研究方向。

基于TreeLSTM的查询基数估计

针对传统的数据库管理系统无法很好地学习谓词之间的交互以及无法准确地估计复杂查询的基数问题,提出了一种树形结构的长短期记忆神经网络(Tree Long Short Term Memory, TreeLSTM)模型建模查询,并使用该模型对新的查询基数进行估计.所提出的模型考虑了查询语句中包含的合取和析取运算,根据谓词之间的操作符类型将子表达式构建为树形结构,根据组合子表达式向量来表示连续向量空间中的任意逻辑表达式.TreeLSTM模型通过捕捉查询谓词之间的顺序依赖关系从而提升基数估计的性能和准确度,将TreeLSTM与基于直方图方法、基于学习的MSCN和TreeRNN方法进行了比较.实验结果表明:TreeLSTM的估算误差比直方图、MSCN、TreeRNN方法的误差分别降低了60.41%,33.33%和11.57%,该方法显著提高了基数估计器的性能.

基于LSM-Tree的键值存储系统的读写性能优化

在写密集型工作环境中,日志结构合并树(LSM-Tree)已逐渐成为主流存储系统,LSM-Tree存在读操作速度慢、写操作成本高、范围查询操作效率低等问题;针对这些问题,为提升LSM-Tree的性能进行了研究,提出了一种基于LSM-Tree的键值存储系统的读写性能优化策略,通过键值分离策略设计vTree结构,并提出层内归并与消极的层间合并相结合的方法,以及范围查询优化合并的策略,从而优化系统的范围查询性能,在LSM-Tree和vTree采用不同的压缩结构,以实现系统读写性能的提升;实验结果表明,与RocksDB相比读性能提升30%,与RocksDB-vTree相比范围查询性能提升10%。

SQL Server性能优化的技术分析

随着客户端/服务器技术和多层架构的进步,系统效率问题日益引发广泛讨论。由于缺乏数据库管理系统(DBMS)提供的优化方法和工具,以及基于微软技术的数据库管理员和应用程序开发人员的知识,无法对设计和服务执行系统进行优化。文章回顾了提高SQL Server实例性能时需要考虑的目标,并阐述了用于优化查询的技术。

XML查询语言XQuery的分析与研究

随着XML日益广泛的应用,XML的查询语言XQuery变得尤为重要。阐述了XML的查询语言XQuery,设计并实现了一个基于XQuery的XML查询系统,同时介绍了它的结构框架和执行过程,提出了对其查询优化的具体方案。

XQuery语言查询优化策略研究

XQuery查询语言在XPath路径查询的基础上使用FLWR等表达式实现XML数据的复杂查询或重构。查询优化是查询处理中重要的组成部分。文章提出了两个优化策略:路径的关联优化和FLWOR查询的重写,XMark实验表明能有效提高XQuery查询的执行效率。