Through improving the redundant data filtering of unreliable data filter for radio frequency identification（RFID） with sliding-window,a data filter which integrates self-adaptive sliding-window and Euclidean distanc...Through improving the redundant data filtering of unreliable data filter for radio frequency identification（RFID） with sliding-window,a data filter which integrates self-adaptive sliding-window and Euclidean distance is proposed.The input data required being filtered have been shunt by considering a large number of redundant data existing in the unreliable data for RFID and the redundant data in RFID are the main filtering object with utilizing the filter based on Euclidean distance.The comparison between the results from the method proposed in this paper and previous research shows that it can improve the accuracy of the RFID for unreliable data filtering and largely reduce the redundant reading rate.展开更多
On-site programming big data refers to the massive data generated in the process of software development with the characteristics of real-time,complexity and high-difficulty for processing.Therefore,data cleaning is e...On-site programming big data refers to the massive data generated in the process of software development with the characteristics of real-time,complexity and high-difficulty for processing.Therefore,data cleaning is essential for on-site programming big data.Duplicate data detection is an important step in data cleaning,which can save storage resources and enhance data consistency.Due to the insufficiency in traditional Sorted Neighborhood Method(SNM)and the difficulty of high-dimensional data detection,an optimized algorithm based on random forests with the dynamic and adaptive window size is proposed.The efficiency of the algorithm can be elevated by improving the method of the key-selection,reducing dimension of data set and using an adaptive variable size sliding window.Experimental results show that the improved SNM algorithm exhibits better performance and achieve higher accuracy.展开更多
基金supported by the foundation of Science and Technology Commission of Shanghai Municipality (Grant No.13521103902)
文摘Through improving the redundant data filtering of unreliable data filter for radio frequency identification（RFID） with sliding-window,a data filter which integrates self-adaptive sliding-window and Euclidean distance is proposed.The input data required being filtered have been shunt by considering a large number of redundant data existing in the unreliable data for RFID and the redundant data in RFID are the main filtering object with utilizing the filter based on Euclidean distance.The comparison between the results from the method proposed in this paper and previous research shows that it can improve the accuracy of the RFID for unreliable data filtering and largely reduce the redundant reading rate.
基金supported by the National Key R&D Program of China(Nos.2018YFB1003905)the National Natural Science Foundation of China under Grant No.61971032,Fundamental Research Funds for the Central Universities(No.FRF-TP-18-008A3).
文摘On-site programming big data refers to the massive data generated in the process of software development with the characteristics of real-time,complexity and high-difficulty for processing.Therefore,data cleaning is essential for on-site programming big data.Duplicate data detection is an important step in data cleaning,which can save storage resources and enhance data consistency.Due to the insufficiency in traditional Sorted Neighborhood Method(SNM)and the difficulty of high-dimensional data detection,an optimized algorithm based on random forests with the dynamic and adaptive window size is proposed.The efficiency of the algorithm can be elevated by improving the method of the key-selection,reducing dimension of data set and using an adaptive variable size sliding window.Experimental results show that the improved SNM algorithm exhibits better performance and achieve higher accuracy.
