A Novel Tensor Decomposition-Based Efficient Detector for Low-Altitude Aerial Objects With Knowledge Distillation Scheme

Unmanned aerial vehicles(UAVs) have gained significant attention in practical applications, especially the low-altitude aerial(LAA) object detection imposes stringent requirements on recognition accuracy and computational resources. In this paper, the LAA images-oriented tensor decomposition and knowledge distillation-based network(TDKD-Net) is proposed,where the TT-format TD(tensor decomposition) and equalweighted response-based KD(knowledge distillation) methods are designed to minimize redundant parameters while ensuring comparable performance. Moreover, some robust network structures are developed, including the small object detection head and the dual-domain attention mechanism, which enable the model to leverage the learned knowledge from small-scale targets and selectively focus on salient features. Considering the imbalance of bounding box regression samples and the inaccuracy of regression geometric factors, the focal and efficient IoU(intersection of union) loss with optimal transport assignment(F-EIoU-OTA)mechanism is proposed to improve the detection accuracy. The proposed TDKD-Net is comprehensively evaluated through extensive experiments, and the results have demonstrated the effectiveness and superiority of the developed methods in comparison to other advanced detection algorithms, which also present high generalization and strong robustness. As a resource-efficient precise network, the complex detection of small and occluded LAA objects is also well addressed by TDKD-Net, which provides useful insights on handling imbalanced issues and realizing domain adaptation.

Digital Twin-Assisted Knowledge Distillation Framework for Heterogeneous Federated Learning

In this paper,to deal with the heterogeneity in federated learning(FL)systems,a knowledge distillation(KD)driven training framework for FL is proposed,where each user can select its neural network model on demand and distill knowledge from a big teacher model using its own private dataset.To overcome the challenge of train the big teacher model in resource limited user devices,the digital twin(DT)is exploit in the way that the teacher model can be trained at DT located in the server with enough computing resources.Then,during model distillation,each user can update the parameters of its model at either the physical entity or the digital agent.The joint problem of model selection and training offloading and resource allocation for users is formulated as a mixed integer programming(MIP)problem.To solve the problem,Q-learning and optimization are jointly used,where Q-learning selects models for users and determines whether to train locally or on the server,and optimization is used to allocate resources for users based on the output of Q-learning.Simulation results show the proposed DT-assisted KD framework and joint optimization method can significantly improve the average accuracy of users while reducing the total delay.

Two-Stage Edge-Side Fault Diagnosis Method Based on Double Knowledge Distillation

With the rapid development of the Internet of Things(IoT),the automation of edge-side equipment has emerged as a significant trend.The existing fault diagnosismethods have the characteristics of heavy computing and storage load,and most of them have computational redundancy,which is not suitable for deployment on edge devices with limited resources and capabilities.This paper proposes a novel two-stage edge-side fault diagnosis method based on double knowledge distillation.First,we offer a clustering-based self-knowledge distillation approach(Cluster KD),which takes the mean value of the sample diagnosis results,clusters them,and takes the clustering results as the terms of the loss function.It utilizes the correlations between faults of the same type to improve the accuracy of the teacher model,especially for fault categories with high similarity.Then,the double knowledge distillation framework uses ordinary knowledge distillation to build a lightweightmodel for edge-side deployment.We propose a two-stage edge-side fault diagnosismethod(TSM)that separates fault detection and fault diagnosis into different stages:in the first stage,a fault detection model based on a denoising auto-encoder(DAE)is adopted to achieve fast fault responses;in the second stage,a diverse convolutionmodel with variance weighting(DCMVW)is used to diagnose faults in detail,extracting features frommicro andmacro perspectives.Through comparison experiments conducted on two fault datasets,it is proven that the proposed method has high accuracy,low delays,and small computation,which is suitable for intelligent edge-side fault diagnosis.In addition,experiments show that our approach has a smooth training process and good balance.

Incremental Learning Based on Data Translation and Knowledge Distillation

Recently, deep convolutional neural networks (DCNNs) have achieved remarkable results in image classification tasks. Despite convolutional networks’ great successes, their training process relies on a large amount of data prepared in advance, which is often challenging in real-world applications, such as streaming data and concept drift. For this reason, incremental learning (continual learning) has attracted increasing attention from scholars. However, incremental learning is associated with the challenge of catastrophic forgetting: the performance on previous tasks drastically degrades after learning a new task. In this paper, we propose a new strategy to alleviate catastrophic forgetting when neural networks are trained in continual domains. Specifically, two components are applied: data translation based on transfer learning and knowledge distillation. The former translates a portion of new data to reconstruct the partial data distribution of the old domain. The latter uses an old model as a teacher to guide a new model. The experimental results on three datasets have shown that our work can effectively alleviate catastrophic forgetting by a combination of the two methods aforementioned.

Ultra-lightweight CNN design based on neural architecture search and knowledge distillation: A novel method to build the automatic recognition model of space target ISAR images

In this paper,a novel method of ultra-lightweight convolution neural network(CNN)design based on neural architecture search(NAS)and knowledge distillation(KD)is proposed.It can realize the automatic construction of the space target inverse synthetic aperture radar(ISAR)image recognition model with ultra-lightweight and high accuracy.This method introduces the NAS method into the radar image recognition for the first time,which solves the time-consuming and labor-consuming problems in the artificial design of the space target ISAR image automatic recognition model(STIIARM).On this basis,the NAS model’s knowledge is transferred to the student model with lower computational complexity by the flow of the solution procedure(FSP)distillation method.Thus,the decline of recognition accuracy caused by the direct compression of model structural parameters can be effectively avoided,and the ultralightweight STIIARM can be obtained.In the method,the Inverted Linear Bottleneck(ILB)and Inverted Residual Block(IRB)are firstly taken as each block’s basic structure in CNN.And the expansion ratio,output filter size,number of IRBs,and convolution kernel size are set as the search parameters to construct a hierarchical decomposition search space.Then,the recognition accuracy and computational complexity are taken as the objective function and constraint conditions,respectively,and the global optimization model of the CNN architecture search is established.Next,the simulated annealing(SA)algorithm is used as the search strategy to search out the lightweight and high accuracy STIIARM directly.After that,based on the three principles of similar block structure,the same corresponding channel number,and the minimum computational complexity,the more lightweight student model is designed,and the FSP matrix pairing between the NAS model and student model is completed.Finally,by minimizing the loss between the FSP matrix pairs of the NAS model and student model,the student model’s weight adjustment is completed.Thus the ultra-lightweight and high accuracy STIIARM is obtained.The proposed method’s effectiveness is verified by the simulation experiments on the ISAR image dataset of five types of space targets.

A Method Based on Knowledge Distillation for Fish School Stress State Recognition in Intensive Aquaculture

Fish behavior analysis for recognizing stress is very important for fish welfare and production management in aquaculture.Recent advances have been made in fish behavior analysis based on deep learning.However,most existing methods with top performance rely on considerable memory and computational resources,which is impractical in the real-world scenario.In order to overcome the limitations of these methods,a new method based on knowledge distillation is proposed to identify the stress states of fish schools.The knowledge distillation architecture transfers additional inter-class information via a mixed relative loss function,and it forces a lightweight network(GhostNet)to mimic the soft probabilities output of a well-trained fish stress state recognition network(ResNeXt101).The fish school stress state recognition model’s accuracy is improved from 94.17%to 98.12%benefiting from the method.The proposed model has about 5.18 M parameters and requires 0.15 G FLOPs(floating-point operations)to process an image of size 224×224.Furthermore,fish behavior images are collected in a land-based factory,and a dataset is constructed and extended through flip,rotation,and color jitter augmentation techniques.The proposed method is also compared with other state-of-the-art methods.The experimental results show that the proposed model is more suitable for deployment on resource-constrained devices or real-time applications,and it is conducive for real-time monitoring of fish behavior.

Soybean Leaf Morphology Classification Based on FPN-SSD and Knowledge Distillation

Soybean leaf morphology is one of the most important morphological and biological characteristics of soybean.The germplasm gene differences of soybeans can lead to different phenotypic traits,among which soybean leaf morphology is an important parameter that directly reflects the difference in soybean germplasm.To realize the morphological classification of soybean leaves,a method was proposed based on deep learning to automatically detect soybean leaves and classify leaf morphology.The morphology of soybean leaves included lanceolate,oval,ellipse and round.First,an image collection platform was designed to collect images of soybean leaves.Then,the feature pyramid networks–single shot multibox detector(FPN-SSD)model was proposed to detect the top leaflets of soybean leaves on the collected images.Finally,a classification model based on knowledge distillation was proposed to classify different morphologies of soybean leaves.The obtained results indicated an overall classification accuracy of 0.956 over a private dataset of 3200 soybean leaf images,and the accuracy of classification for each morphology was 1.00,0.97,0.93 and 0.94.The results showed that this method could effectively classify soybean leaf morphology and had great application potential in analyzing other phenotypic traits of soybean.

Knowledge Distillation for Mobile Edge Computation Offloading

Edge computation offloading allows mobile end devices to execute compute-inten?sive tasks on edge servers. End devices can decide whether the tasks are offloaded to edge servers, cloud servers or executed locally according to current network condition and devic?es'profiles in an online manner. In this paper, we propose an edge computation offloading framework based on deep imitation learning (DIL) and knowledge distillation (KD), which assists end devices to quickly make fine-grained decisions to optimize the delay of computa?tion tasks online. We formalize a computation offloading problem into a multi-label classifi?cation problem. Training samples for our DIL model are generated in an offline manner. Af?ter the model is trained, we leverage KD to obtain a lightweight DIL model, by which we fur?ther reduce the model's inference delay. Numerical experiment shows that the offloading de?cisions made by our model not only outperform those made by other related policies in laten?cy metric, but also have the shortest inference delay among all policies.

Teachers cooperation:team-knowledge distillation for multiple cross-domain few-shot learning

Although few-shot learning(FSL)has achieved great progress,it is still an enormous challenge especially when the source and target set are from different domains,which is also known as cross-domain few-shot learning(CD-FSL).Utilizing more source domain data is an effective way to improve the performance of CD-FSL.However,knowledge from different source domains may entangle and confuse with each other,which hurts the performance on the target domain.Therefore,we propose team-knowledge distllation networks(TKD-Net)to tackle this problem,which explores a strategy to help the cooperation of multiple teachers.Specifically,we distill knowledge from the cooperation of teacher networks to a single student network in a meta-learning framework.It incorporates task-oriented knowledge distillation and multiple cooperation among teachers to train an efficient student with better generalization ability on unseen tasks.Moreover,our TKD-Net employs both response-based knowledge and relation-based knowledge to transfer more comprehensive and effective knowledge.Extensive experimental results on four fine-grained datasets have demonstrated the effectiveness and superiority of our proposed TKD-Net approach.

A Federated Domain Adaptation Algorithm Based on Knowledge Distillation and Contrastive Learning

Smart manufacturing suffers from the heterogeneity of local data distribution across parties,mutual information silos and lack of privacy protection in the process of industry chain collaboration.To address these problems,we propose a federated domain adaptation algorithm based on knowledge distillation and contrastive learning.Knowledge distillation is used to extract transferable integration knowledge from the different source domains and the quality of the extracted integration knowledge is used to assign reasonable weights to each source domain.A more rational weighted average aggregation is used in the aggregation phase of the center server to optimize the global model,while the local model of the source domain is trained with the help of contrastive learning to constrain the local model optimum towards the global model optimum,mitigating the inherent heterogeneity between local data.Our experiments are conducted on the largest domain adaptation dataset,and the results show that compared with other traditional federated domain adaptation algorithms,the algorithm we proposed trains a more accurate model,requires fewer communication rounds,makes more effective use of imbalanced data in the industrial area,and protects data privacy.

A Weakly-Supervised Method for Named Entity Recognition of Agricultural Knowledge Graph

It is significant for agricultural intelligent knowledge services using knowledge graph technology to integrate multi-source heterogeneous crop and pest data and fully mine the knowledge hidden in the text.However,only some labeled data for agricultural knowledge graph domain training are available.Furthermore,labeling is costly due to the need for more data openness and standardization.This paper proposes a novel model using knowledge distillation for a weakly supervised entity recognition in ontology construction.Knowledge distillation between the target and source data domain is performed,where Bi-LSTM and CRF models are constructed for entity recognition.The experimental result is shown that we only need to label less than one-tenth of the data for model training.Furthermore,the agricultural domain ontology is constructed by BILSTM-CRF named entity recognition model and relationship extraction model.Moreover,there are a total of 13,983 entities and 26,498 relationships built in the neo4j graph database.

A Lightweight IoT Malware Detection and Family Classification Method

A lightweight malware detection and family classification system for the Internet of Things (IoT) was designed to solve the difficulty of deploying defense models caused by the limited computing and storage resources of IoT devices. By training complex models with IoT software gray-scale images and utilizing the gradient-weighted class-activated mapping technique, the system can identify key codes that influence model decisions. This allows for the reconstruction of gray-scale images to train a lightweight model called LMDNet for malware detection. Additionally, the multi-teacher knowledge distillation method is employed to train KD-LMDNet, which focuses on classifying malware families. The results indicate that the model’s identification speed surpasses that of traditional methods by 23.68%. Moreover, the accuracy achieved on the Malimg dataset for family classification is an impressive 99.07%. Furthermore, with a model size of only 0.45M, it appears to be well-suited for the IoT environment. By training complex models using IoT software gray-scale images and utilizing the gradient-weighted class-activated mapping technique, the system can identify key codes that influence model decisions. This allows for the reconstruction of gray-scale images to train a lightweight model called LMDNet for malware detection. Thus, the presented approach can address the challenges associated with malware detection and family classification in IoT devices.

Automated Classification of Inherited Retinal Diseases in Optical Coherence Tomography Images Using Few-shot Learning

Objective To develop a few-shot learning(FSL) approach for classifying optical coherence tomography(OCT) images in patients with inherited retinal disorders(IRDs).Methods In this study, an FSL model based on a student–teacher learning framework was designed to classify images. 2,317 images from 189 participants were included. Of these, 1,126 images revealed IRDs, 533 were normal samples, and 658 were control samples.Results The FSL model achieved a total accuracy of 0.974–0.983, total sensitivity of 0.934–0.957, total specificity of 0.984–0.990, and total F1 score of 0.935–0.957, which were superior to the total accuracy of the baseline model of 0.943–0.954, total sensitivity of 0.866–0.886, total specificity of 0.962–0.971,and total F1 score of 0.859–0.885. The performance of most subclassifications also exhibited advantages. Moreover, the FSL model had a higher area under curves(AUC) of the receiver operating characteristic(ROC) curves in most subclassifications.Conclusion This study demonstrates the effective use of the FSL model for the classification of OCT images from patients with IRDs, normal, and control participants with a smaller volume of data. The general principle and similar network architectures can also be applied to other retinal diseases with a low prevalence.

Eye Strain Detection During Online Learning

The recent outbreak of the coronavirus disease of 2019(Covid-19)has been causing many disruptions among the education systems worldwide,most of them due to the abrupt transition to online learning.The sudden upsurge in digital electronic devices usage,namely personal computers,laptops,tablets and smart-phones is unprecedented,which leads to a new wave of both mental and physical health problems among students,for example eye-related illnesses.The overexpo-sure to electronic devices,extended screen time usage and lack of outdoor sun-light have put a consequential strain on the student’s ophthalmic health because of their young age and a relative lack of responsibility on their own health.Failure to take appropriate external measures to mitigate the negative effects of this pro-cess could lead to common ophthalmic illnesses such as myopia or more serious conditions.To remedy this situation,we propose a software solution that is able to track and capture images of its users’eyes to detect symptoms of eye illnesses while simultaneously giving them warnings and even offering treatments.To meet the requirements of a small and light model that is operable on low-end devices without information loss,we optimized the original MobileNetV2 model with depth-wise separable convolutions by altering the parameters in the last layers with an aim to minimize the resizing of the input image and obtained a new model which we call EyeNet.Combined with applying the knowledge distillation tech-nique and ResNet-18 as a teacher model to train the student model,we have suc-cessfully increased the accuracy of the EyeNet model up to 87.16%and support the development of a model compatible with embedded systems with limited computing power,accessible to all students.

Multi-exit self-distillation with appropriate teachers

Multi-exit architecture allows early-stop inference to reduce computational cost,which can be used in resource-constrained circumstances.Recent works combine the multi-exit architecture with self-distillation to simultaneously achieve high efficiency and decent performance at different network depths.However,existing methods mainly transfer knowledge from deep exits or a single ensemble to guide all exits,without considering that inappropriate learning gaps between students and teachers may degrade the model performance,especially in shallow exits.To address this issue,we propose Multi-exit self-distillation with Appropriate TEachers(MATE)to provide diverse and appropriate teacher knowledge for each exit.In MATE,multiple ensemble teachers are obtained from all exits with different trainable weights.Each exit subsequently receives knowledge from all teachers,while focusing mainly on its primary teacher to keep an appropriate gap for efficient knowledge transfer.In this way,MATE achieves diversity in knowledge distillation while ensuring learning efficiency.Experimental results on CIFAR-100,TinyImageNet,and three fine-grained datasets demonstrate that MATE consistently outperforms state-of-the-art multi-exit self-distillation methods with various network architectures.

Motion Enhanced Model Based on High-Level Spatial Features

Action recognition has become a current research hotspot in computer vision.Compared to other deep learning methods,Two-stream convolutional network structure achieves better performance in action recognition,which divides the network into spatial and temporal streams,using video frame images as well as dense optical streams in the network,respectively,to obtain the category labels.However,the two-stream network has some drawbacks,i.e.,using dense optical flow as the input of the temporal stream,which is computationally expensive and extremely time-consuming for the current extraction algorithm and cannot meet the requirements of real-time tasks.In this paper,instead of the dense optical flow,the Motion Vectors(MVs)are used and extracted from the compressed domain as temporal features,which greatly reduces the extraction time.However,the motion pattern that MVs contain is coarser,which leads to low accuracy.In this paper,we propose two strategies to improve the accuracy:firstly,an accumulated strategy is used to enhance the motion information and continuity of MVs;secondly,knowledge distillation is used to fuse the spatial information into the temporal stream so that more information(e.g.,motion details,colors,etc.)is obtainable.Experimental results show that the accuracy of MV can be greatly improved by the strategies proposed in this paper and the final recognition for human actions accuracy is guaranteed without using optical flow.

Forget less,count better:a domain-incremental self-distillation learning benchmark for lifelong crowd counting

Crowd counting has important applications in public safety and pandemic control.A robust and practical crowd counting system has to be capable of continuously learning with the newly incoming domain data in real-world scenarios instead of fitting one domain only.Off-the-shelf methods have some drawbacks when handling multiple domains:(1)the models will achieve limited performance(even drop dramatically)among old domains after training images from new domains due to the discrepancies in intrinsic data distributions from various domains,which is called catastrophic forgetting;(2)the well-trained model in a specific domain achieves imperfect performance among other unseen domains because of domain shift;(3)it leads to linearly increasing storage overhead,either mixing all the data for training or simply training dozens of separate models for different domains when new ones are available.To overcome these issues,we investigate a new crowd counting task in incremental domain training setting called lifelong crowd counting.Its goal is to alleviate catastrophic forgetting and improve the generalization ability using a single model updated by the incremental domains.Specifically,we propose a self-distillation learning framework as a benchmark(forget less,count better,or FLCB)for lifelong crowd counting,which helps the model leverage previous meaningful knowledge in a sustainable manner for better crowd counting to mitigate the forgetting when new data arrive.A new quantitative metric,normalized Backward Transfer(nBwT),is developed to evaluate the forgetting degree of the model in the lifelong learning process.Extensive experimental results demonstrate the superiority of our proposed benchmark in achieving a low catastrophic forgetting degree and strong generalization ability.

NBA:defensive distillation for backdoor removal via neural behavior alignment

Recently,deep neural networks have been shown to be vulnerable to backdoor attacks.A backdoor is inserted into neural networks via this attack paradigm,thus compromising the integrity of the network.As soon as an attacker presents a trigger during the testing phase,the backdoor in the model is activated,allowing the network to make specific wrong predictions.It is extremely important to defend against backdoor attacks since they are very stealthy and dangerous.In this paper,we propose a novel defense mechanism,Neural Behavioral Alignment(NBA),for backdoor removal.NBA optimizes the distillation process in terms of knowledge form and distillation samples to improve defense performance according to the characteristics of backdoor defense.NBA builds high-level representations of neural behavior within networks in order to facilitate the transfer of knowledge.Additionally,NBA crafts pseudo samples to induce student models exhibit backdoor neural behavior.By aligning the backdoor neural behavior from the student network with the benign neural behavior from the teacher network,NBA enables the proactive removal of backdoors.Extensive experiments show that NBA can effectively defend against six different backdoor attacks and outperform five state-of-the-art defenses.

Federated learning on non-IID and long-tailed data viadual-decoupling

Federated learning(FL),a cutting-edge distributed machine learning training paradigm,aims to generate a global model by collaborating on the training of client models without revealing local private data.The co-occurrence of non-independent and identically distributed(non-IID)and long-tailed distribution in FL is one challenge that substantially degrades aggregate performance.In this paper,we present a corresponding solution called federated dual-decoupling via model and logit calibration(FedDDC)for non-IID and long-tailed distributions.The model is characterized by three aspects.First,we decouple the global model into the feature extractor and the classifier to fine-tune the components affected by the joint problem.For the biased feature extractor,we propose a client confidence re-weighting scheme to assist calibration,which assigns optimal weights to each client.For the biased classifier,we apply the classifier re-balancing method for fine-tuning.Then,we calibrate and integrate the client confidence re-weighted logits with the re-balanced logits to obtain the unbiased logits.Finally,we use decoupled knowledge distillation for the first time in the joint problem to enhance the accuracy of the global model by extracting the knowledge of the unbiased model.Numerous experiments demonstrate that on non-IID and long-tailed data in FL,our approach outperforms state-of-the-art methods.

Continual learning fault diagnosis:A dual-branch adaptive aggregation residual network for fault diagnosis with machine increments

As a data-driven approach, Deep Learning(DL)-based fault diagnosis methods need to collect the relatively comprehensive data on machine fault types to achieve satisfactory performance. A mechanical system may include multiple submachines in the real-world. During condition monitoring of a mechanical system, fault data are distributed in a continuous flow of constantly generated information and new faults will inevitably occur in unconsidered submachines, which are also called machine increments. Therefore, adequately collecting fault data in advance is difficult. Limited by the characteristics of DL, training existing models directly with new fault data of new submachines leads to catastrophic forgetting of old tasks, while the cost of collecting all known data to retrain the models is excessively high. DL-based fault diagnosis methods cannot learn continually and adaptively in dynamic environments. A new Continual Learning Fault Diagnosis method(CLFD) is proposed in this paper to solve a series of fault diagnosis tasks with machine increments. The stability–plasticity dilemma is an intrinsic issue in continual learning. The core of CLFD is the proposed Dual-branch Adaptive Aggregation Residual Network(DAARN).Two types of residual blocks are created in each block layer of DAARN: steady and dynamic blocks. The stability–plasticity dilemma is solved by assigning them with adaptive aggregation weights to balance stability and plasticity, and a bi-level optimization program is used to optimize adaptive aggregation weights and model parameters. In addition, a feature-level knowledge distillation loss function is proposed to further overcome catastrophic forgetting. CLFD is then applied to the fault diagnosis case with machine increments. Results demonstrate that CLFD outperforms other continual learning methods and has satisfactory robustness.