This paper summarizes recent progress at the State Key Laboratory of Numerical Modeling for Atmospheric Sci- ences and Geophysical Fluid Dynamics (LASG), Institute of Atmospheric Physics, Chinese Academy of Sciences...This paper summarizes recent progress at the State Key Laboratory of Numerical Modeling for Atmospheric Sci- ences and Geophysical Fluid Dynamics (LASG), Institute of Atmospheric Physics, Chinese Academy of Sciences in studies on targeted observations, data assimilation, and ensemble prediction, which are three effective strategies to re- duce the prediction uncertainties and improve the forecast skill of weather and climate events. Considering the limita- tions of traditional targeted observation approaches, LASG researchers have developed a conditional nonlinear op- timal perturbation-based targeted observation strategy to optimize the design of the observing network. This strategy has been employed to identify sensitive areas for targeted observations of the E1 Nifio-Southern Oscillation, Indian Ocean dipole, and tropical cyclones, and has been demonstrated to be effective in improving the forecast skill of these events. To assimilate the targeted observations into the initial state of a numerical model, a dimension-reduced- projection-based four-dimensional variational data assimilation (DRP-4DVar) approach has been proposed and is used operationally to supply accurate initial conditions in numerical forecasts. The performance of DRP-4DVar is good, and its computational cost is much lower than the standard 4DVar approach. Besides, ensemble prediction, which is a practical approach to generate probabilistic forecasts of the future state of a particular system, can be used to reduce the prediction uncertainties of single forecasts by taking the ensemble mean of forecast members. In this field, LASG researchers have proposed an ensemble forecast method that uses nonlinear local Lyapunov vectors (NLLVs) to yield ensemble initial perturbations. Its application in simple models has shown that NLLVs are more useful than bred vectors and singular vectors in improving the skill of the ensemble forecast. Therefore, NLLVs rep- resent a candidate for possible development as an ensemble method in operational forecasts. Despite the consider- able efforts made towards developing these methods to reduce prediction uncertainties, much challenging but highly important work remains in terms of improving the methods to further increase the skill in forecasting such weather and climate events.展开更多
This work presents a comprehensive fourth-order predictive modeling (PM) methodology that uses the MaxEnt principle to incorporate fourth-order moments (means, covariances, skewness, kurtosis) of model parameters, com...This work presents a comprehensive fourth-order predictive modeling (PM) methodology that uses the MaxEnt principle to incorporate fourth-order moments (means, covariances, skewness, kurtosis) of model parameters, computed and measured model responses, as well as fourth (and higher) order sensitivities of computed model responses to model parameters. This new methodology is designated by the acronym 4<sup>th</sup>-BERRU-PM, which stands for “fourth-order best-estimate results with reduced uncertainties.” The results predicted by the 4<sup>th</sup>-BERRU-PM incorporates, as particular cases, the results previously predicted by the second-order predictive modeling methodology 2<sup>nd</sup>-BERRU-PM, and vastly generalizes the results produced by extant data assimilation and data adjustment procedures.展开更多
Cross entropy is a measure in machine learning and deep learning that assesses the difference between predicted and actual probability distributions. In this study, we propose cross entropy as a performance evaluation...Cross entropy is a measure in machine learning and deep learning that assesses the difference between predicted and actual probability distributions. In this study, we propose cross entropy as a performance evaluation metric for image classifier models and apply it to the CT image classification of lung cancer. A convolutional neural network is employed as the deep neural network (DNN) image classifier, with the residual network (ResNet) 50 chosen as the DNN archi-tecture. The image data used comprise a lung CT image set. Two classification models are built from datasets with varying amounts of data, and lung cancer is categorized into four classes using 10-fold cross-validation. Furthermore, we employ t-distributed stochastic neighbor embedding to visually explain the data distribution after classification. Experimental results demonstrate that cross en-tropy is a highly useful metric for evaluating the reliability of image classifier models. It is noted that for a more comprehensive evaluation of model perfor-mance, combining with other evaluation metrics is considered essential. .展开更多
A new numerical technique named interval finite difference method is proposed for the steady-state temperature field prediction with uncertainties in both physical parameters and boundary conditions. Interval variable...A new numerical technique named interval finite difference method is proposed for the steady-state temperature field prediction with uncertainties in both physical parameters and boundary conditions. Interval variables are used to quantitatively describe the uncertain parameters with limited information. Based on different Taylor and Neumann series, two kinds of parameter perturbation methods are presented to approximately yield the ranges of the uncertain temperature field. By comparing the results with traditional Monte Carlo simulation, a numerical example is given to demonstrate the feasibility and effectiveness of the proposed method for solving steady-state heat conduction problem with uncertain-but-bounded parameters.展开更多
This work (in two parts) will present a novel predictive modeling methodology aimed at obtaining “best-estimate results with reduced uncertainties” for the first four moments (mean values, covariance, skewness and k...This work (in two parts) will present a novel predictive modeling methodology aimed at obtaining “best-estimate results with reduced uncertainties” for the first four moments (mean values, covariance, skewness and kurtosis) of the optimally predicted distribution of model results and calibrated model parameters, by combining fourth-order experimental and computational information, including fourth (and higher) order sensitivities of computed model responses to model parameters. Underlying the construction of this fourth-order predictive modeling methodology is the “maximum entropy principle” which is initially used to obtain a novel closed-form expression of the (moments-constrained) fourth-order Maximum Entropy (MaxEnt) probability distribution constructed from the first four moments (means, covariances, skewness, kurtosis), which are assumed to be known, of an otherwise unknown distribution of a high-dimensional multivariate uncertain quantity of interest. This fourth-order MaxEnt distribution provides optimal compatibility of the available information while simultaneously ensuring minimal spurious information content, yielding an estimate of a probability density with the highest uncertainty among all densities satisfying the known moment constraints. Since this novel generic fourth-order MaxEnt distribution is of interest in its own right for applications in addition to predictive modeling, its construction is presented separately, in this first part of a two-part work. The fourth-order predictive modeling methodology that will be constructed by particularizing this generic fourth-order MaxEnt distribution will be presented in the accompanying work (Part-2).展开更多
This work illustrates the innovative results obtained by applying the recently developed the 2<sup>nd</sup>-order predictive modeling methodology called “2<sup>nd</sup>- BERRU-PM”, where the ...This work illustrates the innovative results obtained by applying the recently developed the 2<sup>nd</sup>-order predictive modeling methodology called “2<sup>nd</sup>- BERRU-PM”, where the acronym BERRU denotes “best-estimate results with reduced uncertainties” and “PM” denotes “predictive modeling.” The physical system selected for this illustrative application is a polyethylene-reflected plutonium (acronym: PERP) OECD/NEA reactor physics benchmark. This benchmark is modeled using the neutron transport Boltzmann equation (involving 21,976 uncertain parameters), the solution of which is representative of “large-scale computations.” The results obtained in this work confirm the fact that the 2<sup>nd</sup>-BERRU-PM methodology predicts best-estimate results that fall in between the corresponding computed and measured values, while reducing the predicted standard deviations of the predicted results to values smaller than either the experimentally measured or the computed values of the respective standard deviations. The obtained results also indicate that 2<sup>nd</sup>-order response sensitivities must always be included to quantify the need for including (or not) the 3<sup>rd</sup>- and/or 4<sup>th</sup>-order sensitivities. When the parameters are known with high precision, the contributions of the higher-order sensitivities diminish with increasing order, so that the inclusion of the 1<sup>st</sup>- and 2<sup>nd</sup>-order sensitivities may suffice for obtaining accurate predicted best- estimate response values and best-estimate standard deviations. On the other hand, when the parameters’ standard deviations are sufficiently large to approach (or be outside of) the radius of convergence of the multivariate Taylor-series which represents the response in the phase-space of model parameters, the contributions stemming from the 3<sup>rd</sup>- and even 4<sup>th</sup>-order sensitivities are necessary to ensure consistency between the computed and measured response. In such cases, the use of only the 1<sup>st</sup>-order sensitivities erroneously indicates that the computed results are inconsistent with the respective measured response. Ongoing research aims at extending the 2<sup>nd</sup>-BERRU-PM methodology to fourth-order, thus enabling the computation of third-order response correlations (skewness) and fourth-order response correlations (kurtosis).展开更多
This work presents a comprehensive second-order predictive modeling (PM) methodology based on the maximum entropy (MaxEnt) principle for obtaining best-estimate mean values and correlations for model responses and par...This work presents a comprehensive second-order predictive modeling (PM) methodology based on the maximum entropy (MaxEnt) principle for obtaining best-estimate mean values and correlations for model responses and parameters. This methodology is designated by the acronym 2<sup>nd</sup>-BERRU-PMP, where the attribute “2<sup>nd</sup>” indicates that this methodology incorporates second- order uncertainties (means and covariances) and second (and higher) order sensitivities of computed model responses to model parameters. The acronym BERRU stands for “Best-Estimate Results with Reduced Uncertainties” and the last letter (“P”) in the acronym indicates “probabilistic,” referring to the MaxEnt probabilistic inclusion of the computational model responses. This is in contradistinction to the 2<sup>nd</sup>-BERRU-PMD methodology, which deterministically combines the computed model responses with the experimental information, as presented in the accompanying work (Part I). Although both the 2<sup>nd</sup>-BERRU-PMP and the 2<sup>nd</sup>-BERRU-PMD methodologies yield expressions that include second (and higher) order sensitivities of responses to model parameters, the respective expressions for the predicted responses, for the calibrated predicted parameters and for their predicted uncertainties (covariances), are not identical to each other. Nevertheless, the results predicted by both the 2<sup>nd</sup>-BERRU-PMP and the 2<sup>nd</sup>-BERRU-PMD methodologies encompass, as particular cases, the results produced by the extant data assimilation and data adjustment procedures, which rely on the minimization, in a least-square sense, of a user-defined functional meant to represent the discrepancies between measured and computed model responses.展开更多
This work presents a comprehensive second-order predictive modeling (PM) methodology designated by the acronym 2<sup>nd</sup>-BERRU-PMD. The attribute “2<sup>nd</sup>” indicates that this met...This work presents a comprehensive second-order predictive modeling (PM) methodology designated by the acronym 2<sup>nd</sup>-BERRU-PMD. The attribute “2<sup>nd</sup>” indicates that this methodology incorporates second-order uncertainties (means and covariances) and second-order sensitivities of computed model responses to model parameters. The acronym BERRU stands for “Best- Estimate Results with Reduced Uncertainties” and the last letter (“D”) in the acronym indicates “deterministic,” referring to the deterministic inclusion of the computational model responses. The 2<sup>nd</sup>-BERRU-PMD methodology is fundamentally based on the maximum entropy (MaxEnt) principle. This principle is in contradistinction to the fundamental principle that underlies the extant data assimilation and/or adjustment procedures which minimize in a least-square sense a subjective user-defined functional which is meant to represent the discrepancies between measured and computed model responses. It is shown that the 2<sup>nd</sup>-BERRU-PMD methodology generalizes and extends current data assimilation and/or data adjustment procedures while overcoming the fundamental limitations of these procedures. In the accompanying work (Part II), the alternative framework for developing the “second- order MaxEnt predictive modelling methodology” is presented by incorporating probabilistically (as opposed to “deterministically”) the computed model responses.展开更多
Focusing on the rapid prediction of acoustic field uncertainty in environment with temporal and spatial sound speed perturbation, evolvement of sound speed structure over time is predicted based on the ocean-acoustic ...Focusing on the rapid prediction of acoustic field uncertainty in environment with temporal and spatial sound speed perturbation, evolvement of sound speed structure over time is predicted based on the ocean-acoustic coupled model to obtain the uncertainty distribution of the vertical structure of sound speed. Further, a method combining the arbitrary polynomial chaos expansion with the empirical orthogonal function is proposed to reduce the dimensionality of uncertain parameters and to obtain the uncertainty distribution of the acoustic field. Simulations have shown that the computational complexity can be reduced by 2 orders of magnitude compared to the conventional polynomial chaos expansion while ensures the same precision.Moreover, the computational complexity is not influenced by the complexity of the sound speed profile. The acoustic field and uncertainty predicted in uncertain environment by proposed method also have been tested with the experimental data.展开更多
An improved method using kernel density estimation (KDE) and confidence level is presented for model validation with small samples. Decision making is a challenging problem because of input uncertainty and only smal...An improved method using kernel density estimation (KDE) and confidence level is presented for model validation with small samples. Decision making is a challenging problem because of input uncertainty and only small samples can be used due to the high costs of experimental measurements. However, model validation provides more confidence for decision makers when improving prediction accuracy at the same time. The confidence level method is introduced and the optimum sample variance is determined using a new method in kernel density estimation to increase the credibility of model validation. As a numerical example, the static frame model validation challenge problem presented by Sandia National Laboratories has been chosen. The optimum bandwidth is selected in kernel density estimation in order to build the probability model based on the calibration data. The model assessment is achieved using validation and accreditation experimental data respectively based on the probability model. Finally, the target structure prediction is performed using validated model, which are consistent with the results obtained by other researchers. The results demonstrate that the method using the improved confidence level and kernel density estimation is an effective approach to solve the model validation problem with small samples.展开更多
基金Supported by the China Meteorological Administration Special Public Welfare Research Fund(GYHY201306018)National Natural Science Foundation of China(41525017 and 41506032)
文摘This paper summarizes recent progress at the State Key Laboratory of Numerical Modeling for Atmospheric Sci- ences and Geophysical Fluid Dynamics (LASG), Institute of Atmospheric Physics, Chinese Academy of Sciences in studies on targeted observations, data assimilation, and ensemble prediction, which are three effective strategies to re- duce the prediction uncertainties and improve the forecast skill of weather and climate events. Considering the limita- tions of traditional targeted observation approaches, LASG researchers have developed a conditional nonlinear op- timal perturbation-based targeted observation strategy to optimize the design of the observing network. This strategy has been employed to identify sensitive areas for targeted observations of the E1 Nifio-Southern Oscillation, Indian Ocean dipole, and tropical cyclones, and has been demonstrated to be effective in improving the forecast skill of these events. To assimilate the targeted observations into the initial state of a numerical model, a dimension-reduced- projection-based four-dimensional variational data assimilation (DRP-4DVar) approach has been proposed and is used operationally to supply accurate initial conditions in numerical forecasts. The performance of DRP-4DVar is good, and its computational cost is much lower than the standard 4DVar approach. Besides, ensemble prediction, which is a practical approach to generate probabilistic forecasts of the future state of a particular system, can be used to reduce the prediction uncertainties of single forecasts by taking the ensemble mean of forecast members. In this field, LASG researchers have proposed an ensemble forecast method that uses nonlinear local Lyapunov vectors (NLLVs) to yield ensemble initial perturbations. Its application in simple models has shown that NLLVs are more useful than bred vectors and singular vectors in improving the skill of the ensemble forecast. Therefore, NLLVs rep- resent a candidate for possible development as an ensemble method in operational forecasts. Despite the consider- able efforts made towards developing these methods to reduce prediction uncertainties, much challenging but highly important work remains in terms of improving the methods to further increase the skill in forecasting such weather and climate events.
文摘This work presents a comprehensive fourth-order predictive modeling (PM) methodology that uses the MaxEnt principle to incorporate fourth-order moments (means, covariances, skewness, kurtosis) of model parameters, computed and measured model responses, as well as fourth (and higher) order sensitivities of computed model responses to model parameters. This new methodology is designated by the acronym 4<sup>th</sup>-BERRU-PM, which stands for “fourth-order best-estimate results with reduced uncertainties.” The results predicted by the 4<sup>th</sup>-BERRU-PM incorporates, as particular cases, the results previously predicted by the second-order predictive modeling methodology 2<sup>nd</sup>-BERRU-PM, and vastly generalizes the results produced by extant data assimilation and data adjustment procedures.
文摘Cross entropy is a measure in machine learning and deep learning that assesses the difference between predicted and actual probability distributions. In this study, we propose cross entropy as a performance evaluation metric for image classifier models and apply it to the CT image classification of lung cancer. A convolutional neural network is employed as the deep neural network (DNN) image classifier, with the residual network (ResNet) 50 chosen as the DNN archi-tecture. The image data used comprise a lung CT image set. Two classification models are built from datasets with varying amounts of data, and lung cancer is categorized into four classes using 10-fold cross-validation. Furthermore, we employ t-distributed stochastic neighbor embedding to visually explain the data distribution after classification. Experimental results demonstrate that cross en-tropy is a highly useful metric for evaluating the reliability of image classifier models. It is noted that for a more comprehensive evaluation of model perfor-mance, combining with other evaluation metrics is considered essential. .
基金supported by the National Special Fund for Major Research Instrument Development(2011YQ140145)111 Project (B07009)+1 种基金the National Natural Science Foundation of China(11002013)Defense Industrial Technology Development Program(A2120110001 and B2120110011)
文摘A new numerical technique named interval finite difference method is proposed for the steady-state temperature field prediction with uncertainties in both physical parameters and boundary conditions. Interval variables are used to quantitatively describe the uncertain parameters with limited information. Based on different Taylor and Neumann series, two kinds of parameter perturbation methods are presented to approximately yield the ranges of the uncertain temperature field. By comparing the results with traditional Monte Carlo simulation, a numerical example is given to demonstrate the feasibility and effectiveness of the proposed method for solving steady-state heat conduction problem with uncertain-but-bounded parameters.
文摘This work (in two parts) will present a novel predictive modeling methodology aimed at obtaining “best-estimate results with reduced uncertainties” for the first four moments (mean values, covariance, skewness and kurtosis) of the optimally predicted distribution of model results and calibrated model parameters, by combining fourth-order experimental and computational information, including fourth (and higher) order sensitivities of computed model responses to model parameters. Underlying the construction of this fourth-order predictive modeling methodology is the “maximum entropy principle” which is initially used to obtain a novel closed-form expression of the (moments-constrained) fourth-order Maximum Entropy (MaxEnt) probability distribution constructed from the first four moments (means, covariances, skewness, kurtosis), which are assumed to be known, of an otherwise unknown distribution of a high-dimensional multivariate uncertain quantity of interest. This fourth-order MaxEnt distribution provides optimal compatibility of the available information while simultaneously ensuring minimal spurious information content, yielding an estimate of a probability density with the highest uncertainty among all densities satisfying the known moment constraints. Since this novel generic fourth-order MaxEnt distribution is of interest in its own right for applications in addition to predictive modeling, its construction is presented separately, in this first part of a two-part work. The fourth-order predictive modeling methodology that will be constructed by particularizing this generic fourth-order MaxEnt distribution will be presented in the accompanying work (Part-2).
文摘This work illustrates the innovative results obtained by applying the recently developed the 2<sup>nd</sup>-order predictive modeling methodology called “2<sup>nd</sup>- BERRU-PM”, where the acronym BERRU denotes “best-estimate results with reduced uncertainties” and “PM” denotes “predictive modeling.” The physical system selected for this illustrative application is a polyethylene-reflected plutonium (acronym: PERP) OECD/NEA reactor physics benchmark. This benchmark is modeled using the neutron transport Boltzmann equation (involving 21,976 uncertain parameters), the solution of which is representative of “large-scale computations.” The results obtained in this work confirm the fact that the 2<sup>nd</sup>-BERRU-PM methodology predicts best-estimate results that fall in between the corresponding computed and measured values, while reducing the predicted standard deviations of the predicted results to values smaller than either the experimentally measured or the computed values of the respective standard deviations. The obtained results also indicate that 2<sup>nd</sup>-order response sensitivities must always be included to quantify the need for including (or not) the 3<sup>rd</sup>- and/or 4<sup>th</sup>-order sensitivities. When the parameters are known with high precision, the contributions of the higher-order sensitivities diminish with increasing order, so that the inclusion of the 1<sup>st</sup>- and 2<sup>nd</sup>-order sensitivities may suffice for obtaining accurate predicted best- estimate response values and best-estimate standard deviations. On the other hand, when the parameters’ standard deviations are sufficiently large to approach (or be outside of) the radius of convergence of the multivariate Taylor-series which represents the response in the phase-space of model parameters, the contributions stemming from the 3<sup>rd</sup>- and even 4<sup>th</sup>-order sensitivities are necessary to ensure consistency between the computed and measured response. In such cases, the use of only the 1<sup>st</sup>-order sensitivities erroneously indicates that the computed results are inconsistent with the respective measured response. Ongoing research aims at extending the 2<sup>nd</sup>-BERRU-PM methodology to fourth-order, thus enabling the computation of third-order response correlations (skewness) and fourth-order response correlations (kurtosis).
文摘This work presents a comprehensive second-order predictive modeling (PM) methodology based on the maximum entropy (MaxEnt) principle for obtaining best-estimate mean values and correlations for model responses and parameters. This methodology is designated by the acronym 2<sup>nd</sup>-BERRU-PMP, where the attribute “2<sup>nd</sup>” indicates that this methodology incorporates second- order uncertainties (means and covariances) and second (and higher) order sensitivities of computed model responses to model parameters. The acronym BERRU stands for “Best-Estimate Results with Reduced Uncertainties” and the last letter (“P”) in the acronym indicates “probabilistic,” referring to the MaxEnt probabilistic inclusion of the computational model responses. This is in contradistinction to the 2<sup>nd</sup>-BERRU-PMD methodology, which deterministically combines the computed model responses with the experimental information, as presented in the accompanying work (Part I). Although both the 2<sup>nd</sup>-BERRU-PMP and the 2<sup>nd</sup>-BERRU-PMD methodologies yield expressions that include second (and higher) order sensitivities of responses to model parameters, the respective expressions for the predicted responses, for the calibrated predicted parameters and for their predicted uncertainties (covariances), are not identical to each other. Nevertheless, the results predicted by both the 2<sup>nd</sup>-BERRU-PMP and the 2<sup>nd</sup>-BERRU-PMD methodologies encompass, as particular cases, the results produced by the extant data assimilation and data adjustment procedures, which rely on the minimization, in a least-square sense, of a user-defined functional meant to represent the discrepancies between measured and computed model responses.
文摘This work presents a comprehensive second-order predictive modeling (PM) methodology designated by the acronym 2<sup>nd</sup>-BERRU-PMD. The attribute “2<sup>nd</sup>” indicates that this methodology incorporates second-order uncertainties (means and covariances) and second-order sensitivities of computed model responses to model parameters. The acronym BERRU stands for “Best- Estimate Results with Reduced Uncertainties” and the last letter (“D”) in the acronym indicates “deterministic,” referring to the deterministic inclusion of the computational model responses. The 2<sup>nd</sup>-BERRU-PMD methodology is fundamentally based on the maximum entropy (MaxEnt) principle. This principle is in contradistinction to the fundamental principle that underlies the extant data assimilation and/or adjustment procedures which minimize in a least-square sense a subjective user-defined functional which is meant to represent the discrepancies between measured and computed model responses. It is shown that the 2<sup>nd</sup>-BERRU-PMD methodology generalizes and extends current data assimilation and/or data adjustment procedures while overcoming the fundamental limitations of these procedures. In the accompanying work (Part II), the alternative framework for developing the “second- order MaxEnt predictive modelling methodology” is presented by incorporating probabilistically (as opposed to “deterministically”) the computed model responses.
基金supported by the National 530 Special 2015 First Batch of Research and Service Support Projectsthe National Defense Scientific and Technological Innovation Special Zone Project(17-H863-05-ZT-001-024-01)
文摘Focusing on the rapid prediction of acoustic field uncertainty in environment with temporal and spatial sound speed perturbation, evolvement of sound speed structure over time is predicted based on the ocean-acoustic coupled model to obtain the uncertainty distribution of the vertical structure of sound speed. Further, a method combining the arbitrary polynomial chaos expansion with the empirical orthogonal function is proposed to reduce the dimensionality of uncertain parameters and to obtain the uncertainty distribution of the acoustic field. Simulations have shown that the computational complexity can be reduced by 2 orders of magnitude compared to the conventional polynomial chaos expansion while ensures the same precision.Moreover, the computational complexity is not influenced by the complexity of the sound speed profile. The acoustic field and uncertainty predicted in uncertain environment by proposed method also have been tested with the experimental data.
基金Funding of Jiangsu Innovation Program for Graduate Education (CXZZ11_0193)NUAA Research Funding (NJ2010009)
文摘An improved method using kernel density estimation (KDE) and confidence level is presented for model validation with small samples. Decision making is a challenging problem because of input uncertainty and only small samples can be used due to the high costs of experimental measurements. However, model validation provides more confidence for decision makers when improving prediction accuracy at the same time. The confidence level method is introduced and the optimum sample variance is determined using a new method in kernel density estimation to increase the credibility of model validation. As a numerical example, the static frame model validation challenge problem presented by Sandia National Laboratories has been chosen. The optimum bandwidth is selected in kernel density estimation in order to build the probability model based on the calibration data. The model assessment is achieved using validation and accreditation experimental data respectively based on the probability model. Finally, the target structure prediction is performed using validated model, which are consistent with the results obtained by other researchers. The results demonstrate that the method using the improved confidence level and kernel density estimation is an effective approach to solve the model validation problem with small samples.