大数据并行计算框架

Parallel computing framework for big data

大数据是当前IT信息技术研究和应用的热点,但目前的研究多集中在系统和应用层面,而理论基础研究方面相对较少.本文以计算复杂性理论为基础,针对大数据量大、快速和多样性等挑战,着重研究大数据的可计算性及其计算原理.首先将多种类型的大数据抽象到度量空间进行统一化表示以解决多样性问题,其次在度量空间中基于距离对大数据进行划分,最后运用NC类计算理论等并行计算理论和方法对大数据问题进行并行求解,以解决量大和快速等问题.本文从更广的视角,根据大数据的特性和大数据整个生命周期,提出处理大数据的策略和技术以及需要变革思维方法研究大数据.

Big data has received a great deal of attention with respect to its use in research and application in information technology. However, most current efforts focus on systems and applications instead of the theoretical foundation. Based on computational complexity theory, according to the volume, velocity, and variety challenges of big data, we study the computability and computational principles of big data. First, various types of big data can be abstracted into metric space for universal representation to handle the variety challenge. Big data can then be partitioned in metric space according to distance. Finally, NC-class computing theory can be applied to solve big data problems in parallel and handle the volume and velocity challenges. Last of all, from a wider perspective, we propose a processing strategy according to the challenges and innovation of big data research methodology.