Impact of Data Quality on Question Answering System Performances

In contrast with the research of new models,little attention has been paid to the impact of low or high-quality data feeding a dialogue system.The present paper makes thefirst attempt tofill this gap by extending our previous work on question-answering(QA)systems by investigating the effect of misspelling on QA agents and how context changes can enhance the responses.Instead of using large language models trained on huge datasets,we propose a method that enhances the model's score by modifying only the quality and structure of the data feed to the model.It is important to identify the features that modify the agent performance because a high rate of wrong answers can make the students lose their interest in using the QA agent as an additional tool for distant learning.The results demonstrate the accuracy of the proposed context simplification exceeds 85%.Thesefindings shed light on the importance of question data quality and context complexity construct as key dimensions of the QA system.In conclusion,the experimental results on questions and contexts showed that controlling and improving the various aspects of data quality around the QA system can significantly enhance his robustness and performance.

Modeling data quality for risk assessment of GIS

This paper presents a methodology to determine three data quality (DQ) risk characteristics: accuracy, comprehensiveness and nonmembership. The methodology provides a set of quantitative models to confirm the information quality risks for the database of the geographical information system (GIS). Four quantitative measures are introduced to examine how the quality risks of source information affect the quality of information outputs produced using the relational algebra operations Selection, Projection, and Cubic Product. It can be used to determine how quality risks associated with diverse data sources affect the derived data. The GIS is the prime source of information on the location of cables, and detection time strongly depends on whether maps indicate the presence of cables in the construction business. Poor data quality in the GIS can contribute to increased risk or higher risk avoidance costs. A case study provides a numerical example of the calculation of the trade-offs between risk and detection costs and provides an example of the calculation of the costs of data quality. We conclude that the model contributes valuable new insight.

Digital Continuity Guarantee Approach of Electronic Record Based on Data Quality Theory

Since the British National Archive put forward the concept of the digital continuity in 2007,several developed countries have worked out their digital continuity action plan.However,the technologies of the digital continuity guarantee are still lacked.At first,this paper analyzes the requirements of digital continuity guarantee for electronic record based on data quality theory,then points out the necessity of data quality guarantee for electronic record.Moreover,we convert the digital continuity guarantee of electronic record to ensure the consistency,completeness and timeliness of electronic record,and construct the first technology framework of the digital continuity guarantee for electronic record.Finally,the temporal functional dependencies technology is utilized to build the first integration method to insure the consistency,completeness and timeliness of electronic record.

Novel method for the evaluation of data quality based on fuzzy control

One of the goals of data collection is preparing for decision-making, so high quality requirement must be satisfied. Rational evaluation of data quality is an effective way to identify data problem in time, and the quality of data after this evaluation is satisfactory with the requirement of decision maker. A fuzzy neural network based research method of data quality evaluation is proposed. First, the criteria for the evaluation of data quality are selected to construct the fuzzy sets of evaluating grades, and then by using the learning ability of NN, the objective evaluation of membership is carried out, which can be used for the effective evaluation of data quality. This research has been used in the platform of ＇data report of national compulsory education outlay guarantee＇ from the Chinese Ministry of Education. This method can be used for the effective evaluation of data quality worldwide, and the data quality situation can be found out more completely, objectively, and in better time by using the method.

On Statistical Measures for Data Quality Evaluation

Most GIS databases contain data errors. The quality of the data sources such as traditional paper maps or more recent remote sensing data determines spatial data quality. In the past several decades, different statistical measures have been developed to evaluate data quality for different types of data, such as nominal categorical data, ordinal categorical data and numerical data. Although these methods were originally proposed for medical research or psychological research, they have been widely used to evaluate spatial data quality. In this paper, we first review statistical methods for evaluating data quality, discuss under what conditions we should use them and how to interpret the results, followed by a brief discussion of statistical software and packages that can be used to compute these data quality measures.

A Short Review of the Literature on Automatic Data Quality

Several organizations have migrated to the cloud for better quality in business engagements and security. Data quality is crucial in present-day activities. Information is generated and collected from data representing real-time facts and activities. Poor data quality affects the organizational decision-making policy and customer satisfaction, and influences the organization’s scheme of execution negatively. Data quality also has a massive influence on the accuracy, complexity and efficiency of the machine and deep learning tasks. There are several methods and tools to evaluate data quality to ensure smooth incorporation in model development. The bulk of data quality tools permit the assessment of sources of data only at a certain point in time, and the arrangement and automation are consequently an obligation of the user. In ensuring automatic data quality, several steps are involved in gathering data from different sources and monitoring data quality, and any problems with the data quality must be adequately addressed. There was a gap in the literature as no attempts have been made previously to collate all the advances in different dimensions of automatic data quality. This limited narrative review of existing literature sought to address this gap by correlating different steps and advancements related to the automatic data quality systems. The six crucial data quality dimensions in organizations were discussed, and big data were compared and classified. This review highlights existing data quality models and strategies that can contribute to the development of automatic data quality systems.

Improve Data Quality by Processing Null Values and Semantic Dependencies

Today, the quantity of data continues to increase, furthermore, the data are heterogeneous, from multiple sources (structured, semi-structured and unstructured) and with different levels of quality. Therefore, it is very likely to manipulate data without knowledge about their structures and their semantics. In fact, the meta-data may be insufficient or totally absent. Data Anomalies may be due to the poverty of their semantic descriptions, or even the absence of their description. In this paper, we propose an approach to better understand the semantics and the structure of the data. Our approach helps to correct automatically the intra-column anomalies and the inter-col- umns ones. We aim to improve the quality of data by processing the null values and the semantic dependencies between columns.

Assessment of Knowledge and Practices of Community Health Nurses on Data Quality in the Ho Municipality of Ghana

Background: High data quality provides correct and up-to-date information which is critical to ensure, not only for the maintenance of health care at an optimal level, but also for the provision of high-quality clinical care, continuing health care, clinical and health service research, and planning and management of health systems. For the attainment of achievable improvements in the health sector, good data is core. Aim/Objective: To assess the level of knowledge and practices of Community Health Nurses on data quality in the Ho municipality, Ghana. Methods: A descriptive cross-sectional study was employed for the study, using a standard Likert scale questionnaire. A census was used to collect 77 Community Health Nurses’ information. The statistical software, Epi-Data 3.1 was used to enter the data and exported to STATA 12.0 for the analyses. Chi-square and logistic analyses were performed to establish associations between categorical variables and a p-value of less than 0.05 at 95% significance interval was considered statistically significant. Results: Out of the 77 Community Health Nurses studied, 49 (63.64%) had good knowledge on data accuracy, 51 (66.23%) out of the 77 Community Health Nurses studied had poor knowledge on data completeness, and 64 (83.12%) had poor knowledge on data timeliness out of the 77 studied. Also, 16 (20.78%) and 33 (42.86%) of the 77 Community Health Nurses responded there was no designated staff for data quality review and no feedback from the health directorate respectively. Out of the 16 health facilities studied for data quality practices, half (8, 50.00%) had missing values on copies of their previous months’ report forms. More so, 10 (62.50%) had no reminders (monthly data submission itineraries) at the facility level. Conclusion: Overall, the general level of knowledge of Community Health Nurses on data quality was poor and their practices for improving data quality at the facility level were woefully inadequate. Therefore, Community Health Nurses need to be given on-job training and proper education on data quality and its dimensions. Also, the health directorate should intensify its continuous supportive supervisory visits at all facilities and feedback should be given to the Community Health Nurses on the data submitted.

Data Quality Assurance Techniques for a Monitoring and Diagnosis System

By researching the data quality problem in the monitoring and diagnosis system （MDS） , the method of detecting non-condition data based on the development trend of equipment condition is proposed, and three requirements of criteria for detecting non-condition data： dynamic, syntheses and simplicity are discussed. According to the general mode of data management in MDS, a data quality assurance system （DQAS） comprising data quality monitoring, data quality diagnosis, detection criteria adjusting and artificial confirmation is set up. A route inspection system called MTREE realizes the DQAS. Aiming at vibration data of route inspection, two detecting criteria are made. One is the quality monitoring parameter, which is found through combining and optimizing some fundamental parameters by genetic programming （GP）. The other is the quality diagnosis criterion, i. e. pseudo distance of Spectral-Energy-Vector （SEV） named Adjacent J-divergence, which indicates the variation degree of adjacent data＇s spectral energy distribution. Results show that DQAS, including these two criteria, is effective to improve the data quality of MDS.

Imagery Data Quality of ZY Satellite Reached International Level

The in-orbit commissioning of ZY-1 02C satellite is proceeding smoothly. According to the relevant experts in this field, the imagery quality of the satellite has reached or nearly reached the level of international satellites of the same kind. ZY-1 02C satellite and ZY-3 satellite were successfully launched on December 22, 2011 and January 9, 2012 respectively. China Centre for Resources Satellite Data andApplication (CRSDA) was responsible for the building of a ground

Comprehensive Evaluation Method for Traffic Flow Data Quality Based on Grey Correlation Analysis and Particle Swarm Optimization

Nowadays,data are more and more used for intelligent modeling and prediction,and the comprehensive evaluation of data quality is getting more and more attention as a necessary means to measure whether the data are usable or not.However,the comprehensive evaluation method of data quality mostly contains the subjective factors of the evaluator,so how to comprehensively and objectively evaluate the data has become a bottleneck that needs to be solved in the research of comprehensive evaluation method.In order to evaluate the data more comprehensively,objectively and differentially,a novel comprehensive evaluation method based on particle swarm optimization(PSO)and grey correlation analysis(GCA)is presented in this paper.At first,an improved GCA evaluation model based on the technique for order preference by similarity to an ideal solution(TOPSIS)is proposed.Then,an objective function model of maximum difference of the comprehensive evaluation values is built,and the PSO algorithm is used to optimize the weights of the improved GCA evaluation model based on the objective function model.Finally,the performance of the proposed method is investigated through parameter analysis.A performance comparison of traffic flow data is carried out,and the simulation results show that the maximum average difference between the evaluation results and its mean value(MDR)of the proposed comprehensive evaluation method is 33.24%higher than that of TOPSIS-GCA,and 6.86%higher than that of GCA.The proposed method has better differentiation than other methods,which means that it objectively and comprehensively evaluates the data from both the relevance and differentiation of the data,and the results more effectively reflect the differences in data quality,which will provide more effective data support for intelligent modeling,prediction and other applications.

Classification and quantification of timestamp data quality issues and its impact on data quality outcome

Timestamps play a key role in process mining because it determines the chronology of which events occurred and subsequently how they are ordered in process modelling.The timestamp in process mining gives an insight on process performance,conformance,and modelling.This therefore means problems with the timestamp will result in misrepresentations of the mined process.A few articles have been published on the quantification of data quality problems but just one of the articles at the time of this paper is based on the quantification of timestamp quality problems.This article evaluates the quality of timestamps in event log across two axes using eleven quality dimensions and four levels of potential data quality problems.The eleven data quality dimensions were obtained by doing a thorough literature review of more than fifty process mining articles which focus on quality dimensions.This evaluation resulted in twelve data quality quantification metrics and the metrics were applied to the MIMIC-ll dataset as an illustration.The outcome of the timestamp quality quantification using the proposed typology enabled the user to appreciate the quality of the event log and thus makes it possible to evaluate the risk of carrying out specific data cleaning measures to improve the process mining outcome.

Prediction of blast furnace gas generation based on data quality improvement strategy

The real-time energy flow data obtained in industrial production processes are usually of low quality.It is difficult to accurately predict the short-term energy flow profile by using these field data,which diminishes the effect of industrial big data and artificial intelligence in industrial energy system.The real-time data of blast furnace gas(BFG)generation collected in iron and steel sites are also of low quality.In order to tackle this problem,a three-stage data quality improvement strategy was proposed to predict the BFG generation.In the first stage,correlation principle was used to test the sample set.In the second stage,the original sample set was rectified and updated.In the third stage,Kalman filter was employed to eliminate the noise of the updated sample set.The method was verified by autoregressive integrated moving average model,back propagation neural network model and long short-term memory model.The results show that the prediction model based on the proposed three-stage data quality improvement method performs well.Long short-term memory model has the best prediction performance,with a mean absolute error of 17.85 m3/min,a mean absolute percentage error of 0.21%,and an R squared of 95.17%.

Data quality assessment for studies investigating microplastics and nanoplastics in food products: Are current data reliable?

Data on the occurrence of microplastics and nanoplastics(MP/NPs)in foods have been used to assess the human health risk caused by the consumption of MP/NPs.The reliability of the data,however,remains unclear because of the lack of international standards for the analysis of MP/NPs in foods.Therefore,the data quality needs to be assessed for accurate health risk assessment.This study developed 10 criteria applicable to the quality assessment of data on MP/NPs in foods.Accordingly,the reliability of 71 data records(69 of them only focused on MPs)was assessed by assigning a score of 2(reliable without restrictions),1(reliable but with restrictions),or 0(unreliable)on each criterion.The results showed that only three data records scored 2 or 1 on all criteria,and six data records scored 0 on as many as six criteria.A total of 58 data records did not include information on positive controls,and 12 data records did not conduct the polymer identification,which could result in the overestimation or underestimation of MP/NPs.Our results also indicated that the data quality of unprocessed foods was more reliable than that of processed foods.Furthermore,we proposed a quality assurance and quality control protocol to investigate MP/NPs in foods.Notably,the characteristics of MP/NPs used in toxicological studies and those existing in foods showed a remarkable discrepancy,causing the uncertainty of health risk assessment.Therefore,both the estimated exposure of MP/NPs and the claimed potential health risks should be treated with caution.

Data quality evaluation and calibration of on-road remote sensing systems based on exhaust plumes

In recent years,with rapid increases in the number of vehicles in China,the contribution of vehicle exhaust emissions to air pollution has become increasingly prominent.To achieve the precise control of emissions,on-road remote sensing(RS)technology has been developed and applied for law enforcement and supervision.However,data quality is still an existing issue affecting the development and application of RS.In this study,the RS data from a cross-road RS system used at a single site(from 2012 to 2015)were collected,the data screening process was reviewed,the issues with data quality were summarized,a new method of data screening and calibration was proposed,and the effectiveness of the improved data quality control methods was finally evaluated.The results showed that this method reduces the skewness and kurtosis of the data distribution by up to nearly 67%,which restores the actual characteristics of exhaust diffusion and is conducive to the identification of actual clean and high-emission vehicles.The annual variability of emission factors of nitric oxide decreases by 60%-on average-eliminating the annual drift of fleet emissions and improving data reliability.

Correlation Analysis of Turbidity and Total Phosphorus in Water Quality Monitoring Data

At present,water pollution has become an important factor affecting and restricting national and regional economic development.Total phosphorus is one of the main sources of water pollution and eutrophication,so the prediction of total phosphorus in water quality has good research significance.This paper selects the total phosphorus and turbidity data for analysis by crawling the data of the water quality monitoring platform.By constructing the attribute object mapping relationship,the correlation between the two indicators was analyzed and used to predict the future data.Firstly,the monthly mean and daily mean concentrations of total phosphorus and turbidity outliers were calculated after cleaning,and the correlation between them was analyzed.Secondly,the correlation coefficients of different times and frequencies were used to predict the values for the next five days,and the data trend was predicted by python visualization.Finally,the real value was compared with the predicted value data,and the results showed that the correlation between total phosphorus and turbidity was useful in predicting the water quality.

Clinical data quality problems and countermeasure for real world study

Real world study （RWS） has become a hotspot for clinical research. Data quality plays a vital role in research achievement and other clinical research fields. In this paper, the common quality problems in the RWS of traditional Chinese medicine are discussed, and a countermeasure is proposed.

Calibration and Data Quality Analysis with Mobile C-Band Polarimetric Radar

A C-band mobile polarimetric radar with simultaneous horizontal and vertical transmission was built in the State Key Laboratory of Severe Weather, Chinese Academy of Meteorological Sciences. It was used in heavy rainfall and typhoon observations in 2008. It is well-known that radar calibration is essential and critical to high quality radar data and products. In this paper, the test and weather signals were used in calibration of reflectivity ZH, differential reflectivity ZDR and differential phase ФDP. Noise effects on correlation coefficient ρHV at low signal-noise-ratio （SNR） were analyzed. The polarimetric radar data for a heavy rain and a snow event were inspected to evaluate the performance of the calibration method and radar data quality, and S-band Doppler radar data were used to validate the refiectivity data quality collected by the polarimetric radar. The results show that the polarimetric and S-band Doppler radars have observed comparable reflectivity values and a similar structure of a heavy rainfall case at middle and low levels. The mismatch of two receivers produce obvious ZDR biases, which were verified by the radar data observed at vertical incidence. The ZDR correction improved the radar data quality. The usage range for PHV was defined. Application of the calibration method introduced in this paper can reduce the system biases caused by the difference of horizontal （H） and vertical （V） channels. After the calibration and correction, the polarimetric parameters observed by the polarimetric radar could be used in further relevant researches.

Data quality issues for synchrophasor applications PartⅠ:a review

Synchrophasor systems, providing low-latency,high-precision, and time-synchronized measurements to enhance power grid performances, are deployed globally.However, the synchrophasor system as a physical network,involves communication constraints and data quality issues, which will impact or even disable certain synchrophasor applications. This work investigates the data quality issue for synchrophasor applications. In Part I, the standards of synchrophasor systems and the classifications and data quality requirements of synchrophasor applications are reviewed. Also, the actual events of synchronization signal accuracy, synchrophasor data loss, and latency are counted and analyzed. The review and statistics are expected to provide an overall picture of data accuracy,loss, and latency issues for synchrophasor applications.

Steering data quality with visual analytics:The complexity challenge

Data quality management,especially data cleansing,has been extensively studied for many years in the areas of data management and visual analytics.In the paper,we first review and explore the relevant work from the research areas of data management,visual analytics and human-computer interaction.Then for different types of data such as multimedia data,textual data,trajectory data,and graph data,we summarize the common methods for improving data quality by leveraging data cleansing techniques at different analysis stages.Based on a thorough analysis,we propose a general visual analytics framework for interactively cleansing data.Finally,the challenges and opportunities are analyzed and discussed in the context of data and humans.