Optimal Estimation of High-Dimensional Covariance Matrices with Missing and Noisy Data

The estimation of covariance matrices is very important in many fields, such as statistics. In real applications, data are frequently influenced by high dimensions and noise. However, most relevant studies are based on complete data. This paper studies the optimal estimation of high-dimensional covariance matrices based on missing and noisy sample under the norm. First, the model with sub-Gaussian additive noise is presented. The generalized sample covariance is then modified to define a hard thresholding estimator , and the minimax upper bound is derived. After that, the minimax lower bound is derived, and it is concluded that the estimator presented in this article is rate-optimal. Finally, numerical simulation analysis is performed. The result shows that for missing samples with sub-Gaussian noise, if the true covariance matrix is sparse, the hard thresholding estimator outperforms the traditional estimate method.

Progress and future prospects of decadal prediction and data assimilation:A review

年代际预测,也称为“近期气候预测”,旨在预测未来1-10年内的气候变化,是气候预测和气候变化研究领域的一个新关注点.它位于季节至年际预测和长期气候变化预测之间,结合了初值问题和外部强迫问题的两个方面.年代际预测的核心技术在于用于模式初始化的同化方法的准确性和效率,其目标是为模式提供准确的初始条件,其中包含观测到的气候系统内部变率,年代际预测的初始化通常涉及在耦合框架内同化海洋观测,其中观测到的信号通过耦合过程传递到其他分量,如大气和海冰.然而,最近的研究越来越关注在海洋-大气耦合模式中探索耦合数据同化(CDA),有人认为CDA有潜力显著提高年代际预测技巧.本文综合评述了该领域的三个方面的研究现状:初始化方法,年代际气候预测的可预测性和预测技巧,以及年代际预测的未来发展和挑战.

Global aerodynamic design optimization based on data dimensionality reduction

In aerodynamic optimization, global optimization methods such as genetic algorithms are preferred in many cases because of their advantage on reaching global optimum. However,for complex problems in which large number of design variables are needed, the computational cost becomes prohibitive, and thus original global optimization strategies are required. To address this need, data dimensionality reduction method is combined with global optimization methods, thus forming a new global optimization system, aiming to improve the efficiency of conventional global optimization. The new optimization system involves applying Proper Orthogonal Decomposition（POD） in dimensionality reduction of design space while maintaining the generality of original design space. Besides, an acceleration approach for samples calculation in surrogate modeling is applied to reduce the computational time while providing sufficient accuracy. The optimizations of a transonic airfoil RAE2822 and the transonic wing ONERA M6 are performed to demonstrate the effectiveness of the proposed new optimization system. In both cases, we manage to reduce the number of design variables from 20 to 10 and from 42 to 20 respectively. The new design optimization system converges faster and it takes 1/3 of the total time of traditional optimization to converge to a better design, thus significantly reducing the overall optimization time and improving the efficiency of conventional global design optimization method.

Coupling Ensemble Kalman Filter with Four-dimensional Variational Data Assimilation

This study examines the performance of coupling the deterministic four-dimensional variational assimilation system （4DVAR） with an ensemble Kalman filter （EnKF） to produce a superior hybrid approach for data assimilation. The coupled assimilation scheme （E4DVAR） benefits from using the state-dependent uncertainty provided by EnKF while taking advantage of 4DVAR in preventing filter divergence： the 4DVAR analysis produces posterior maximum likelihood solutions through minimization of a cost function about which the ensemble perturbations are transformed, and the resulting ensemble analysis can be propagated forward both for the next assimilation cycle and as a basis for ensemble forecasting. The feasibility and effectiveness of this coupled approach are demonstrated in an idealized model with simulated observations. It is found that the E4DVAR is capable of outperforming both 4DVAR and the EnKF under both perfect- and imperfect-model scenarios. The performance of the coupled scheme is also less sensitive to either the ensemble size or the assimilation window length than those for standard EnKF or 4DVAR implementations.

Similarity measurement method of high-dimensional data based on normalized net lattice subspace

The performance of conventional similarity measurement methods is affected seriously by the curse of dimensionality of high-dimensional data.The reason is that data difference between sparse and noisy dimensionalities occupies a large proportion of the similarity,leading to the dissimilarities between any results.A similarity measurement method of high-dimensional data based on normalized net lattice subspace is proposed.The data range of each dimension is divided into several intervals,and the components in different dimensions are mapped onto the corresponding interval.Only the component in the same or adjacent interval is used to calculate the similarity.To validate this method,three data types are used,and seven common similarity measurement methods are compared.The experimental result indicates that the relative difference of the method is increasing with the dimensionality and is approximately two or three orders of magnitude higher than the conventional method.In addition,the similarity range of this method in different dimensions is [0,1],which is fit for similarity analysis after dimensionality reduction.

CABOSFV algorithm for high dimensional sparse data clustering

An algorithm, Clustering Algorithm Based On Sparse Feature Vector (CABOSFV),was proposed for the high dimensional clustering of binary sparse data. This algorithm compressesthe data effectively by using a tool 'Sparse Feature Vector', thus reduces the data scaleenormously, and can get the clustering result with only one data scan. Both theoretical analysis andempirical tests showed that CABOSFV is of low computational complexity. The algorithm findsclusters in high dimensional large datasets efficiently and handles noise effectively.

Similarity measure design for high dimensional data

Information analysis of high dimensional data was carried out through similarity measure application. High dimensional data were considered as the a typical structure. Additionally, overlapped and non-overlapped data were introduced, and similarity measure analysis was also illustrated and compared with conventional similarity measure. As a result, overlapped data comparison was possible to present similarity with conventional similarity measure. Non-overlapped data similarity analysis provided the clue to solve the similarity of high dimensional data. Considering high dimensional data analysis was designed with consideration of neighborhoods information. Conservative and strict solutions were proposed. Proposed similarity measure was applied to express financial fraud among multi dimensional datasets. In illustrative example, financial fraud similarity with respect to age, gender, qualification and job was presented. And with the proposed similarity measure, high dimensional personal data were calculated to evaluate how similar to the financial fraud. Calculation results show that the actual fraud has rather high similarity measure compared to the average, from minimal 0.0609 to maximal 0.1667.

A theoretical study of the multigrid three-dimensional variational data assimilation scheme using a simple bilinear interpolation algorithm

In order to solve the so-called ＂bull-eye＂ problem caused by using a simple bilinear interpolation as an observational mapping operator in the cost function in the multigrid three-dimensional variational （3DVAR） data assimilation scheme, a smoothing term, equivalent to a penalty term, is introduced into the cost function to serve as a means of troubleshooting. A theoretical analysis is first performed to figure out what on earth results in the issue of ＂bull-eye＂, and then the meaning of such smoothing term is elucidated and the uniqueness of solution of the multigrid 3DVAR with the smoothing term added is discussed through the theoretical deduction for one-dimensional （1D） case, and two idealized data assimilation experiments （one- and two-dimensional （2D） cases）. By exploring the relationship between the smoothing term and the recursive filter theoretically and practically, it is revealed why satisfied analysis results can be achieved by using such proposed solution for the issue of the multigrid 3DVAR.

Testing a Four-Dimensional Variational Data Assimilation Method Using an Improved Intermediate Coupled Model for ENSO Analysis and Prediction

A four-dimensional variational （4D-Var） data assimilation method is implemented in an improved intermediate coupled model （ICM） of the tropical Pacific. A twin experiment is designed to evaluate the impact of the 4D-Var data assimilation algorithm on ENSO analysis and prediction based on the ICM. The model error is assumed to arise only from the parameter uncertainty. The ＂observation＂ of the SST anomaly, which is sampled from a ＂truth＂ model simulation that takes default parameter values and has Gaussian noise added, is directly assimilated into the assimilation model with its parameters set erroneously. Results show that 4D-Var effectively reduces the error of ENSO analysis and therefore improves the prediction skill of ENSO events compared with the non-assimilation case. These results provide a promising way for the ICM to achieve better real-time ENSO prediction.

Constructing Three-Dimension Space Graph for Outlier Detection Algorithms in Data Mining

Outlier detection has very important applied value in data mining literature. Different outlier detection algorithms based on distinct theories have different definitions and mining processes. The three-dimensional space graph for constructing applied algorithms and an improved GridOf algorithm were proposed in terms of analyzing the existing outlier detection algorithms from criterion and theory. Key words outlier - detection - three-dimensional space graph - data mining CLC number TP 311. 13 - TP 391 Foundation item: Supported by the National Natural Science Foundation of China (70371015)Biography: ZHANG Jing (1975-), female, Ph. D, lecturer, research direction: data mining and knowledge discovery.

Nonlinear Dimensionality Reduction and Data Visualization:A Review

Dimensionality reduction and data visualization are useful and important processes in pattern recognition. Many techniques have been developed in the recent years. The self-organizing map （SOM） can be an efficient method for this purpose. This paper reviews recent advances in this area and related approaches such as multidimensional scaling （MDS）, nonlinear PC A, principal manifolds, as well as the connections of the SOM and its recent variant, the visualization induced SOM （ViSOM）, with these approaches. The SOM is shown to produce a quantized, qualitative scaling and while the ViSOM a quantitative or metric scaling and approximates principal curve/surface. The SOM can also be regarded as a generalized MDS to relate two metric spaces by forming a topological mapping between them. The relationships among various recently proposed techniques such as ViSOM, Isomap, LLE, and eigenmap are discussed and compared.

Multi-dimensional database design and implementation of dam safety monitoring system

To improve the effectiveness of dam safety monitoring database systems, the development process of a multi-dimensional conceptual data model was analyzed and a logic design wasachieved in multi-dimensional database mode. The optimal data model was confirmed by identifying data objects, defining relations and reviewing entities. The conversion of relations among entities to external keys and entities and physical attributes to tables and fields was interpreted completely. On this basis, a multi-dimensional database that reflects the management and analysis of a dam safety monitoring system on monitoring data information has been established, for which factual tables and dimensional tables have been designed. Finally, based on service design and user interface design, the dam safety monitoring system has been developed with Delphi as the development tool. This development project shows that the multi-dimensional database can simplify the development process and minimize hidden dangers in the database structure design. It is superior to other dam safety monitoring system development models and can provide a new research direction for system developers.

Multidimensional Data Querying on Tree-Structured Overlay

Multidimensional data query has been gaining much interest in database research communities in recent years, yet many of the existing studies focus mainly on ten tralized systems. A solution to querying in Peer-to-Peer（P2P） environment was proposed to achieve both low processing cost in terms of the number of peers accessed and search messages and balanced query loads among peers. The system is based on a balanced tree structured P2P network. By partitioning the query space intelligently, the amount of query forwarding is effectively controlled, and the number of peers involved and search messages are also limited. Dynamic load balancing can be achieved during space partitioning and query resolving. Extensive experiments confirm the effectiveness and scalability of our algorithms on P2P networks.

A DESIGN OF THREE-DIMENSIONAL SPATIAL DATA MODEL AND ITS DATA STRUCTURE IN GEOLOGICAL EXPLORATION ENGINEERING

The key to develop 3-D GISs is the study on 3-D data model and data structure. Some of the data models and data structures have been presented by scholars. Because of the complexity of 3-D spatial phenomenon, there are no perfect data structures that can describe all spatial entities. Every data structure has its own advantages and disadvantages. It is difficult to design a single data structure to meet different needs. The important subject in the3-D data models is developing a data model that has integrated vector and raster data structures. A special 3-D spatial data model based on distributing features of spatial entities should be designed. We took the geological exploration engineering as the research background and designed an integrated data model whose data structures integrats vector and raster data byadopting object-oriented technique. Research achievements are presented in this paper.

VISUALIZATION OF THREE-DIMENSIONAL DATA FIELD AND ITS APPLICATION IN MACHINE TESTING

In order to realize visualization of three-dimensional data field （TDDF） in instrument, two methods of visualization of TDDF and the usual manner of quick graphic and image processing are analyzed. And how to use OpenGL technique and the characteristic of analyzed data to construct a TDDF, the ways of reality processing and interactive processing are described. Then the medium geometric element and a related realistic model are constructed by means of the first algorithm. Models obtained for attaching the third dimension in three-dimensional data field are presented. An example for TDDF realization of machine measuring is provided. The analysis of resultant graphic indicates that the three-dimensional graphics built by the method developed is featured by good reality, fast processing and strong interaction

A nearest neighbor search algorithm of high-dimensional data based on sequential NPsim matrix

Problems existin similarity measurement and index tree construction which affect the performance of nearest neighbor search of high-dimensional data. The equidistance problem is solved using NPsim function to calculate similarity. And a sequential NPsim matrix is built to improve indexing performance. To sum up the above innovations,a nearest neighbor search algorithm of high-dimensional data based on sequential NPsim matrix is proposed in comparison with the nearest neighbor search algorithms based on KD-tree or SR-tree on Munsell spectral data set. Experimental results show that the proposed algorithm similarity is better than that of other algorithms and searching speed is more than thousands times of others. In addition,the slow construction speed of sequential NPsim matrix can be increased by using parallel computing.

A 3-DIMENSIONAL DATA MODEL FOR VISUALIZING CLOVERLEAF JUNCTION IN A CITY MODEL

The research work has been seldom done about cloverleaf junction expression in a 3-dimensional city model (3DCM). The main reason is that the cloverleaf junction is often in a complex and enormous construction. Its main body is bestraddle in air,and has aerial intersections between its parts. This complex feature made cloverleaf junction quite different from buildings and terrain, therefore, it is difficult to express this kind of spatial objects in the same way as for buildings and terrain. In this paper,authors analyze spatial characteristics of cloverleaf junction, propose an all-constraint points TIN algorithm to partition cloverleaf junction road surface, and develop a method to visualize cloverleaf junction road surface using TIN. In order to manage cloverleaf junction data efficiently, the authors also analyzed the mechanism of 3DCM data management, extended BLOB type in relational database, and combined R-tree index to manage 3D spatial data. Based on this extension, an appropriate data

Dimensionality Reduction of High-Dimensional Highly Correlated Multivariate Grapevine Dataset

Viticulturists traditionally have a keen interest in studying the relationship between the biochemistry of grapevines’ leaves/petioles and their associated spectral reflectance in order to understand the fruit ripening rate, water status, nutrient levels, and disease risk. In this paper, we implement imaging spectroscopy (hyperspectral) reflectance data, for the reflective 330 - 2510 nm wavelength region (986 total spectral bands), to assess vineyard nutrient status;this constitutes a high dimensional dataset with a covariance matrix that is ill-conditioned. The identification of the variables (wavelength bands) that contribute useful information for nutrient assessment and prediction, plays a pivotal role in multivariate statistical modeling. In recent years, researchers have successfully developed many continuous, nearly unbiased, sparse and accurate variable selection methods to overcome this problem. This paper compares four regularized and one functional regression methods: Elastic Net, Multi-Step Adaptive Elastic Net, Minimax Concave Penalty, iterative Sure Independence Screening, and Functional Data Analysis for wavelength variable selection. Thereafter, the predictive performance of these regularized sparse models is enhanced using the stepwise regression. This comparative study of regression methods using a high-dimensional and highly correlated grapevine hyperspectral dataset revealed that the performance of Elastic Net for variable selection yields the best predictive ability.

Design and Realization of Block Level Augmented Reality Three-Dimensional Map

In response to the construction needs of “Real 3D China”, the system structure, functional framework, application direction and product form of block level augmented reality three-dimensional map is designed. Those provide references and ideas for the later large-scale production of augmented reality three-dimensional map. The augmented reality three-dimensional map is produced based on skyline software. Including the map browsing, measurement and analysis and so on, the basic function of three-dimensional map is realized. The special functional module including housing management, pipeline management and so on is developed combining the need of residential quarters development, that expands the application fields of augmented reality three-dimensional map. Those lay the groundwork for the application of augmented reality three-dimensional map. .