A Novel Efficient and Effective Preprocessing Algorithm for Text Classification

Text classification is an essential task of natural language processing. Preprocessing, which determines the representation of text features, is one of the key steps of text classification architecture. It proposed a novel efficient and effective preprocessing algorithm with three methods for text classification combining the Orthogonal Matching Pursuit algorithm to perform the classification. The main idea of the novel preprocessing strategy is that it combined stopword removal and/or regular filtering with tokenization and lowercase conversion, which can effectively reduce the feature dimension and improve the text feature matrix quality. Simulation tests on the 20 newsgroups dataset show that compared with the existing state-of-the-art method, the new method reduces the number of features by 19.85%, 34.35%, 26.25% and 38.67%, improves accuracy by 7.36%, 8.8%, 5.71% and 7.73%, and increases the speed of text classification by 17.38%, 25.64%, 23.76% and 33.38% on the four data, respectively.

Optimizing Facial Expression Recognition through Effective Preprocessing Techniques

Analyzing human facial expressions using machine vision systems is indeed a challenging yet fascinating problem in the field of computer vision and artificial intelligence. Facial expressions are a primary means through which humans convey emotions, making their automated recognition valuable for various applications including man-computer interaction, affective computing, and psychological research. Pre-processing techniques are applied to every image with the aim of standardizing the images. Frequently used techniques include scaling, blurring, rotating, altering the contour of the image, changing the color to grayscale and normalization. Followed by feature extraction and then the traditional classifiers are applied to infer facial expressions. Increasing the performance of the system is difficult in the typical machine learning approach because feature extraction and classification phases are separate. But in Deep Neural Networks (DNN), the two phases are combined into a single phase. Therefore, the Convolutional Neural Network (CNN) models give better accuracy in Facial Expression Recognition than the traditional classifiers. But still the performance of CNN is hampered by noisy and deviated images in the dataset. This work utilized the preprocessing methods such as resizing, gray-scale conversion and normalization. Also, this research work is motivated by these drawbacks to study the use of image pre-processing techniques to enhance the performance of deep learning methods to implement facial expression recognition. Also, this research aims to recognize emotions using deep learning and show the influences of data pre-processing for further processing of images. The accuracy of each pre-processing methods is compared, then combination between them is analysed and the appropriate preprocessing techniques are identified and implemented to see the variability of accuracies in predicting facial expressions. .

Data preprocessing and preliminary results of the Moon-based Ultraviolet Telescope on the CE-3 lander

The Moon-based Ultraviolet Telescope （MUVT） is one of the payloads on the Chang＇e-3 （CE-3） lunar lander. Because of the advantages of having no at- mospheric disturbances and the slow rotation of the Moon, we can make long-term continuous observations of a series of important celestial objects in the near ultra- violet band （245-340 nm）, and perform a sky survey of selected areas, which can- not be completed on Earth. We can find characteristic changes in celestial brightness with time by analyzing image data from the MUVT, and deduce the radiation mech- anism and physical properties of these celestial objects after comparing with a phys- ical model. In order to explain the scientific purposes of MUVT, this article analyzes the preprocessing of MUVT image data and makes a preliminary evaluation of data quality. The results demonstrate that the methods used for data collection and prepro- cessing are effective, and the Level 2A and 2B image data satisfy the requirements of follow-up scientific researches.

Diabetes Type 2: Poincaré Data Preprocessing for Quantum Machine Learning

Quantum Machine Learning(QML)techniques have been recently attracting massive interest.However reported applications usually employ synthetic or well-known datasets.One of these techniques based on using a hybrid approach combining quantum and classic devices is the Variational Quantum Classifier(VQC),which development seems promising.Albeit being largely studied,VQC implementations for“real-world”datasets are still challenging on Noisy Intermediate Scale Quantum devices(NISQ).In this paper we propose a preprocessing pipeline based on Stokes parameters for data mapping.This pipeline enhances the prediction rates when applying VQC techniques,improving the feasibility of solving classification problems using NISQ devices.By including feature selection techniques and geometrical transformations,enhanced quantum state preparation is achieved.Also,a representation based on the Stokes parameters in the PoincaréSphere is possible for visualizing the data.Our results show that by using the proposed techniques we improve the classification score for the incidence of acute comorbid diseases in Type 2 Diabetes Mellitus patients.We used the implemented version of VQC available on IBM’s framework Qiskit,and obtained with two and three qubits an accuracy of 70%and 72%respectively.

DATA PREPROCESSING AND RE KERNEL CLUSTERING FOR LETTER

Many classifiers and methods are proposed to deal with letter recognition problem. Among them, clustering is a widely used method. But only one time for clustering is not adequately. Here, we adopt data preprocessing and a re kernel clustering method to tackle the letter recognition problem. In order to validate effectiveness and efficiency of proposed method, we introduce re kernel clustering into Kernel Nearest Neighbor classification(KNN), Radial Basis Function Neural Network(RBFNN), and Support Vector Machine(SVM). Furthermore, we compare the difference between re kernel clustering and one time kernel clustering which is denoted as kernel clustering for short. Experimental results validate that re kernel clustering forms fewer and more feasible kernels and attain higher classification accuracy.

Power Data Preprocessing Method of Mountain Wind Farm Based on POT-DBSCAN

Due to the frequent changes of wind speed and wind direction,the accuracy of wind turbine(WT)power prediction using traditional data preprocessing method is low.This paper proposes a data preprocessing method which combines POT with DBSCAN(POT-DBSCAN)to improve the prediction efficiency of wind power prediction model.Firstly,according to the data of WT in the normal operation condition,the power prediction model ofWT is established based on the Particle Swarm Optimization(PSO)Arithmetic which is combined with the BP Neural Network(PSO-BP).Secondly,the wind-power data obtained from the supervisory control and data acquisition(SCADA)system is preprocessed by the POT-DBSCAN method.Then,the power prediction of the preprocessed data is carried out by PSO-BP model.Finally,the necessity of preprocessing is verified by the indexes.This case analysis shows that the prediction result of POT-DBSCAN preprocessing is better than that of the Quartile method.Therefore,the accuracy of data and prediction model can be improved by using this method.

D-IMPACT: A Data Preprocessing Algorithm to Improve the Performance of Clustering

In this study, we propose a data preprocessing algorithm called D-IMPACT inspired by the IMPACT clustering algorithm. D-IMPACT iteratively moves data points based on attraction and density to detect and remove noise and outliers, and separate clusters. Our experimental results on two-dimensional datasets and practical datasets show that this algorithm can produce new datasets such that the performance of the clustering algorithm is improved.

Preprocessing Model of Manuscripts in Javanese Characters

Manuscript preprocessing is the earliest stage in transliteration process of manuscripts in Javanese scripts. Manuscript preprocessing stage is aimed to produce images of letters which form the manuscripts to be processed further in manuscript transliteration system. There are four main steps in manuscript preprocessing, which are manuscript binarization, noise reduction, line segmentation, and character segmentation for every line image produced by line segmentation. The result of the test on parts of PB.A57 manuscript which contains 291 character images, with 95% level of confidence concluded that the success percentage of preprocessing in producing Javanese character images ranged 85.9% - 94.82%.

Adaptive Gradient-Based and Anisotropic Diffusion Equation Filtering Algorithm for Microscopic Image Preprocessing

In image acquisition process, the quality of microscopic images will be degraded by electrical noise, quantizing noise, light illumination etc. Hence, image preprocessing is necessary and important to improve the quality. The background noise and pulse noise are two common types of noise existing in microscopic images. In this paper, a gradient-based anisotropic filtering algorithm was proposed, which can filter out the background noise while preserve object boundary effectively. The filtering performance was evaluated by comparing that with some other filtering algorithms.

Hybrid 1DCNN-Attention with Enhanced Data Preprocessing for Loan Approval Prediction

In order to reduce the risk of non-performing loans, losses, and improve the loan approval efficiency, it is necessary to establish an intelligent loan risk and approval prediction system. A hybrid deep learning model with 1DCNN-attention network and the enhanced preprocessing techniques is proposed for loan approval prediction. Our proposed model consists of the enhanced data preprocessing and stacking of multiple hybrid modules. Initially, the enhanced data preprocessing techniques using a combination of methods such as standardization, SMOTE oversampling, feature construction, recursive feature elimination (RFE), information value (IV) and principal component analysis (PCA), which not only eliminates the effects of data jitter and non-equilibrium, but also removes redundant features while improving the representation of features. Subsequently, a hybrid module that combines a 1DCNN with an attention mechanism is proposed to extract local and global spatio-temporal features. Finally, the comprehensive experiments conducted validate that the proposed model surpasses state-of-the-art baseline models across various performance metrics, including accuracy, precision, recall, F1 score, and AUC. Our proposed model helps to automate the loan approval process and provides scientific guidance to financial institutions for loan risk control.

EEG connectivity analysis in infants:A Beginner's Guide on Preprocessing and Processing Techniques

Over the last decades,infantile brain networks have received increased scientific attention due to the elevated need to understand better the maturational processes of the human brain and the early forms of neural abnormalities.Electroencephalography(EEG)is becoming a popular tool for the investigation of functional connectivity(FC)of the immature brain,as it is easily applied in awake,non-sedated infants.However,there are still no universally accepted standards regarding the preprocessing and processing analyses which address the peculiarities of infantile EEG data,resulting in comparability difficulties between different studies.Nevertheless,during the last few years,there is a growing effort in overcoming these issues,with the creation of age-appropriate pipelines.Although FC in infants has been mostly measured via linear metrics and particularly coherence analysis,non-linear methods,such as cross-frequency-coupling(CFC),may be more valuable for the investigation of network communication and early network development.Additionally,graph theory analysis often accompanies linear and non-linear FC computation offering a more comprehensive understanding of the infantile network architecture.The current review attempts to gather the basic information on the preprocessing and processing techniques that are usually employed by infantile FC studies,while providing guidelines for future studies.

Predicting 3D Radiotherapy Dose-Volume Based on Deep Learning

Cancer is one of the most dangerous diseaseswith highmortality.One of the principal treatments is radiotherapy by using radiation beams to destroy cancer cells and this workflow requires a lot of experience and skill from doctors and technicians.In our study,we focused on the 3D dose prediction problem in radiotherapy by applying the deeplearning approach to computed tomography(CT)images of cancer patients.Medical image data has more complex characteristics than normal image data,and this research aims to explore the effectiveness of data preprocessing and augmentation in the context of the 3D dose prediction problem.We proposed four strategies to clarify our hypothesis in different aspects of applying data preprocessing and augmentation.In strategies,we trained our custom convolutional neural network model which has a structure inspired by the U-net,and residual blocks were also applied to the architecture.The output of the network is added with a rectified linear unit(Re-Lu)function for each pixel to ensure there are no negative values,which are absurd with radiation doses.Our experiments were conducted on the dataset of the Open Knowledge-Based Planning Challenge which was collected from head and neck cancer patients treatedwith radiation therapy.The results of four strategies showthat our hypothesis is rational by evaluating metrics in terms of the Dose-score and the Dose-volume histogram score(DVH-score).In the best training cases,the Dose-score is 3.08 and the DVH-score is 1.78.In addition,we also conducted a comparison with the results of another study in the same context of using the loss function.

Preprocessing of 2-Dimensional Gel Electrophoresis Images Applied to Proteomic Analysis： A Review

Various methods and specialized software programs are available for processing two- dimensional gel electrophoresis （2-DGE） images. However, due to the anomalies present in these images, a reliable, automated, and highly reproducible system for 2-DGE image analysis has still not been achieved. The most common anomalies found in 2-DGE images include vertical and hor- izontal streaking, fuzzy spots, and background noise, which greatly complicate computational anal- ysis. In this paper, we review the preprocessing techniques applied to 2-DGE images for noise reduction, intensity normalization, and background correction. We also present a quantitative comparison of non-linear filtering techniques applied to synthetic gel images, through analyzing the performance of the filters under specific conditions. Synthetic proteins were modeled into a two-dimensional Gaussian distribution with adjustable parameters for changing the size, intensity, and degradation. Three types of noise were added to the images： Gaussian, Rayleigh, and exponen- tial, with signal-to-noise ratios （SNRs） ranging 8-20 decibels （dB）. We compared the performanceof wavelet, contourlet, total variation （TV）, and wavelet-total variation （WTTV） techniques using parameters SNR and spot efficiency. In terms of spot efficiency, contourlet and TV were more sen- sitive to noise than wavelet and WTTV. Wavelet worked the best for images with SNR ranging 10- 20 dB, whereas WTTV performed better with high noise levels. Wavelet also presented the best per- formance with any level of Gaussian noise and low levels （20-14 dB） of Rayleigh and exponential noise in terms of SNR. Finally, the performance of the non-linear filtering techniques was evaluated using a real 2-DGE image with previously identified proteins marked. Wavelet achieved the best detection rate for the real image.

Preprocessing method of night vision image application in apple harvesting robot

Due to the low working efficiency of apple harvesting robots,there is still a long way to go for commercialization.The machine performance and extended operating time are the two research aspects for improving efficiencies of harvesting robots,this study focused on the extended operating time and proposed a round-the-clock operation mode.Due to the influences of light,temperature,humidity,etc.,the working environment at night is relatively complex,and thus restricts the operating efficiency of the apple harvesting robot.Three different artificial light sources(incandescent lamp,fluorescent lamp,and LED lights)were selected for auxiliary light according to certain rules so that the apple night vision images could be captured.In addition,by color analysis,night and natural light images were compared to find out the color characteristics of the night vision images,and intuitive visual and difference image methods were used to analyze the noise characteristics.The results showed that the incandescent lamp is the best artificial auxiliary light for apple harvesting robots working at night,and the type of noise contained in apple night vision images is Gaussian noise mixed with some salt and pepper noise.The preprocessing method can provide a theoretical and technical reference for subsequent image processing.

Land 3D-Seismic Data: Preprocessing Quality Control Utilizing Survey Design Specifications, Noise Properties, Normal Moveout, First Breaks, and Offset

The recent proliferation of the 3D reflection seismic method into the near-surface area of geophysical applications, especially in response to the emergence of the need to comprehensively characterize and monitor near-surface carbon dioxide sequestration in shallow saline aquifers around the world, justifies the emphasis on cost-effective and robust quality control and assurance （QC/QA） workflow of 3D seismic data preprocessing that is suitable for near-surface applications. The main purpose of our seismic data preprocessing QC is to enable the use of appropriate header information, data that are free of noise-dominated traces, and/or flawed vertical stacking in subsequent processing steps. In this article, I provide an account of utilizing survey design specifications, noise properties, first breaks, and normal moveout for rapid and thorough graphical QC/QA diagnostics, which are easy to apply and efficient in the diagnosis of inconsistencies. A correlated vibroseis time-lapse 3D-seismic data set from a CO2-flood monitoring survey is used for demonstrating QC diagnostics. An important by-product of the QC workflow is establishing the number of layers for a refraction statics model in a data-driven graphical manner that capitalizes on the spatial coverage of the 3D seismic data.

Impact of preprocessing on medical data classification

The significance of the preprocessing stage in any data mining task is well known. Before attempting medical data classification, characteristics of medical datasets, including noise, incompleteness, and the existence of multiple and possibly irrelevant features, need to be addressed. In this paper, we show that selecting the right combination of prepro- cessing methods has a considerable impact on the classification potential of a dataset. The preprocessing operations con- sidered include the discretization of numeric attributes, the selection of attribute subset（s）, and the handling of missing values. The classification is performed by an ant colony optimization algorithm as a case study. Experimental results on 25 real-world medical datasets show that a significant relative improvement in predictive accuracy, exceeding 60% in some cases, is obtained.

Control of vertical phase separation in high performance non-fullerene organic solar cell by introducing oscillating stratification preprocessing

Non-fullerene organic solar cell(NFOSC)has attracted tremendous attention due to their great potential for commercial applications.To improve its power conversion efficiency(PCE),generally,sequential solution deposition(SSD)methods have been employed to construct the graded vertical phase separation(VPS)of the bulk-heterojunction(BHJ)active layer for efficient exciton separation and charge transition.However,a variety of orthogonal solvents used in the SSD may lead to the unpredicted change in the BHJ morphology and introduce additional defects inside the BHJ bulk thus complicate the fabrication process.Here,a simple oscillating stratification preprocessing(OSP)is developed to facilitate the formation of graded VPS among the BHJ layer.As a result,a significant improvement is obtained in PCE from 10.96%to 12.03%,which is the highest value reported among PBDB-T:ITIC based NFOSC.

Background Interference Removal Algorithm for PIV Preprocessing Based on Improved Local Otsu Thresholding

Due to background light fluctuation,noise interference,voltage fluctuation,and other factors,there will be noise interference of different intensities in the background of the collected image.In this paper,a PIV image background interference removal algorithm based on improved neighborhood Otsu processing is proposed.The algorithm proposed in this paper separates the particle image from the background interference through the adaptive neighborhood improved Otsu threshold segmentation method and uses the common PIV analysis tools PIVLab and para PIV to analyze the flow pattern after the interference is removed.The experimental results demonstrated that the proposed algorithm can obviously improve the quality of PIV results in terms of both PSNR and SSIM in the case of background light interference,and the increase in average performance is nearly 50%compared with traditional preprocessing algorithms,which solves the problem of large flow pattern analysis error caused by poor background light removal effect in the case of irregular grating and other background light interference only using traditional preprocessing.

Untargeted LC–MS Data Preprocessing in Metabolomics

Liquid chromatography–mass spectrometry(LC–MS)has enabled the detection of thousands of metabolite features from a single biological sample that produces large and complex datasets.One of the key issues in LC–MS-based metabolomics is comprehensive and accurate analysis of enormous amount of data.Many free data preprocessing tools,such as XCMS,MZmine,MAVEN,and MetaboAnalyst,as well as commercial software,have been developed to facilitate data processing.However,researchers are challenged by the inevitable and unconquerable yields of numerous false-positive peaks,and human errors while manually removing such false peaks.Even with continuous improvements of data processing tools,there can still be many mistakes generated during data preprocessing.In addition,many data preprocessing software exist,and every tool has its own advantages and disadvantages.Thereby,a researcher needs to judge what kind of software or tools to choose that most suit their vendor proprietary formats and goal of downstream analysis.Here,we provided a brief introduction of the general steps of raw MS data processing,and properties of automated data processing tools.Then,characteristics of mainly free data preprocessing software were summarized for researchers’consideration in conducting metabolomics study.

An improved adaptive preprocessing method for TDI CCD images

In order to achieve high quality images with time-delayed integration(TDI) charge-coupled device(CCD) imaging system, an improved adaptive preprocessing method is proposed with functions of both denoising and edge enhancement. It is a weighted average filter integrating the average filter and the improved range filter. The weighted factors are deduced in terms of a cost function, which are adjustable to different images. To validate the proposed method, extensive tests are carried out on a developed TDI CCD imaging system. The experimental results confirm that this preprocessing method can fulfill the noise removal and edge sharpening simultaneously, which can play an important role in remote sensing field.