PAL-BERT:An Improved Question Answering Model

In the field of natural language processing(NLP),there have been various pre-training language models in recent years,with question answering systems gaining significant attention.However,as algorithms,data,and computing power advance,the issue of increasingly larger models and a growing number of parameters has surfaced.Consequently,model training has become more costly and less efficient.To enhance the efficiency and accuracy of the training process while reducing themodel volume,this paper proposes a first-order pruningmodel PAL-BERT based on the ALBERT model according to the characteristics of question-answering(QA)system and language model.Firstly,a first-order network pruning method based on the ALBERT model is designed,and the PAL-BERT model is formed.Then,the parameter optimization strategy of the PAL-BERT model is formulated,and the Mish function was used as an activation function instead of ReLU to improve the performance.Finally,after comparison experiments with traditional deep learning models TextCNN and BiLSTM,it is confirmed that PALBERT is a pruning model compression method that can significantly reduce training time and optimize training efficiency.Compared with traditional models,PAL-BERT significantly improves the NLP task’s performance.

Computer-aided diagnosis of retinopathy based on vision transformer

Age-related Macular Degeneration(AMD)and Diabetic Macular Edema(DME)are two com-mon retinal diseases for elder people that may ultimately cause irreversible blindness.Timely and accurate diagnosis is essential for the treatment of these diseases.In recent years,computer-aided diagnosis(CAD)has been deeply investigated and effectively used for rapid and early diagnosis.In this paper,we proposed a method of CAD using vision transformer to analyze optical co-herence tomography(OCT)images and to automatically discriminate AMD,DME,and normal eyes.A classification accuracy of 99.69%was achieved.After the model pruning,the recognition time reached 0.010 s and the classification accuracy did not drop.Compared with the Con-volutional Neural Network(CNN)image classification models(VGG16,Resnet50,Densenet121,and EfficientNet),vision transformer after pruning exhibited better recognition ability.Results show that vision transformer is an improved alternative to diagnose retinal diseases more accurately.