To address the shortage of public datasets for customs X-ray images of contraband and the difficulties in deploying trained models in engineering applications,a method has been proposed that employs the Extract-Transf...To address the shortage of public datasets for customs X-ray images of contraband and the difficulties in deploying trained models in engineering applications,a method has been proposed that employs the Extract-Transform-Load(ETL)approach to create an X-ray dataset of contraband items.Initially,X-ray scatter image data is collected and cleaned.Using Kafka message queues and the Elasticsearch(ES)distributed search engine,the data is transmitted in real-time to cloud servers.Subsequently,contraband data is annotated using a combination of neural networks and manual methods to improve annotation efficiency and implemented mean hash algorithm for quick image retrieval.The method of integrating targets with backgrounds has enhanced the X-ray contraband image data,increasing the number of positive samples.Finally,an Airport Customs X-ray dataset(ACXray)compatible with customs business scenarios has been constructed,featuring an increased number of positive contraband samples.Experimental tests using three datasets to train the Mask Region-based Convolutional Neural Network(Mask R-CNN)algorithm and tested on 400 real customs images revealed that the recognition accuracy of algorithms trained with Security Inspection X-ray(SIXray)and Occluded Prohibited Items X-ray(OPIXray)decreased by 16.3%and 15.1%,respectively,while the ACXray dataset trained algorithm’s accuracy was almost unaffected.This indicates that the ACXray dataset-trained algorithm possesses strong generalization capabilities and is more suitable for customs detection scenarios.展开更多
数据抽取、转换和装载(Extraction,Transformation and Loading,简称ETL)是数据仓库化的关键环节,对数据仓库数据质量有着至关重要的影响。随着信息化的发展,ETL已经成为当前较活跃的研究领域之一,但是ETL理论和技术的发展还不成熟。针...数据抽取、转换和装载(Extraction,Transformation and Loading,简称ETL)是数据仓库化的关键环节,对数据仓库数据质量有着至关重要的影响。随着信息化的发展,ETL已经成为当前较活跃的研究领域之一,但是ETL理论和技术的发展还不成熟。针对当前ETL研究中存在的一些问题和需要考虑的各种因素,从ETL各个阶段存在的主要问题出发,列举了各种研究方法及研究成果,并进行了分析。最后,总结并提出了ETL的未来研究方向和今后工作的建议。展开更多
基金supported by the National Natural Science Foundation of China(Grant No.51605069).
文摘To address the shortage of public datasets for customs X-ray images of contraband and the difficulties in deploying trained models in engineering applications,a method has been proposed that employs the Extract-Transform-Load(ETL)approach to create an X-ray dataset of contraband items.Initially,X-ray scatter image data is collected and cleaned.Using Kafka message queues and the Elasticsearch(ES)distributed search engine,the data is transmitted in real-time to cloud servers.Subsequently,contraband data is annotated using a combination of neural networks and manual methods to improve annotation efficiency and implemented mean hash algorithm for quick image retrieval.The method of integrating targets with backgrounds has enhanced the X-ray contraband image data,increasing the number of positive samples.Finally,an Airport Customs X-ray dataset(ACXray)compatible with customs business scenarios has been constructed,featuring an increased number of positive contraband samples.Experimental tests using three datasets to train the Mask Region-based Convolutional Neural Network(Mask R-CNN)algorithm and tested on 400 real customs images revealed that the recognition accuracy of algorithms trained with Security Inspection X-ray(SIXray)and Occluded Prohibited Items X-ray(OPIXray)decreased by 16.3%and 15.1%,respectively,while the ACXray dataset trained algorithm’s accuracy was almost unaffected.This indicates that the ACXray dataset-trained algorithm possesses strong generalization capabilities and is more suitable for customs detection scenarios.
文摘数据抽取、转换和装载(Extraction,Transformation and Loading,简称ETL)是数据仓库化的关键环节,对数据仓库数据质量有着至关重要的影响。随着信息化的发展,ETL已经成为当前较活跃的研究领域之一,但是ETL理论和技术的发展还不成熟。针对当前ETL研究中存在的一些问题和需要考虑的各种因素,从ETL各个阶段存在的主要问题出发,列举了各种研究方法及研究成果,并进行了分析。最后,总结并提出了ETL的未来研究方向和今后工作的建议。
