A Novel Quantization and Model Compression Approach for Hardware Accelerators in Edge Computing

Massive computational complexity and memory requirement of artificial intelligence models impede their deploy-ability on edge computing devices of the Internet of Things(IoT).While Power-of-Two(PoT)quantization is pro-posed to improve the efficiency for edge inference of Deep Neural Networks(DNNs),existing PoT schemes require a huge amount of bit-wise manipulation and have large memory overhead,and their efficiency is bounded by the bottleneck of computation latency and memory footprint.To tackle this challenge,we present an efficient inference approach on the basis of PoT quantization and model compression.An integer-only scalar PoT quantization(IOS-PoT)is designed jointly with a distribution loss regularizer,wherein the regularizer minimizes quantization errors and training disturbances.Additionally,two-stage model compression is developed to effectively reduce memory requirement,and alleviate bandwidth usage in communications of networked heterogenous learning systems.The product look-up table(P-LUT)inference scheme is leveraged to replace bit-shifting with only indexing and addition operations for achieving low-latency computation and implementing efficient edge accelerators.Finally,comprehensive experiments on Residual Networks(ResNets)and efficient architectures with Canadian Institute for Advanced Research(CIFAR),ImageNet,and Real-world Affective Faces Database(RAF-DB)datasets,indicate that our approach achieves 2×∼10×improvement in the reduction of both weight size and computation cost in comparison to state-of-the-art methods.A P-LUT accelerator prototype is implemented on the Xilinx KV260 Field Programmable Gate Array(FPGA)platform for accelerating convolution operations,with performance results showing that P-LUT reduces memory footprint by 1.45×,achieves more than 3×power efficiency and 2×resource efficiency,compared to the conventional bit-shifting scheme.

DPAL-BERT:A Faster and Lighter Question Answering Model

Recent advancements in natural language processing have given rise to numerous pre-training language models in question-answering systems.However,with the constant evolution of algorithms,data,and computing power,the increasing size and complexity of these models have led to increased training costs and reduced efficiency.This study aims to minimize the inference time of such models while maintaining computational performance.It also proposes a novel Distillation model for PAL-BERT(DPAL-BERT),specifically,employs knowledge distillation,using the PAL-BERT model as the teacher model to train two student models:DPAL-BERT-Bi and DPAL-BERTC.This research enhances the dataset through techniques such as masking,replacement,and n-gram sampling to optimize knowledge transfer.The experimental results showed that the distilled models greatly outperform models trained from scratch.In addition,although the distilled models exhibit a slight decrease in performance compared to PAL-BERT,they significantly reduce inference time to just 0.25%of the original.This demonstrates the effectiveness of the proposed approach in balancing model performance and efficiency.

Effect of warm acupuncture on nitric oxide synthase and calcitonin gene-related peptide in a rat model of lumbar nerve root compression

BACKGROUND： Varying degrees of inflammatory responses occur during lumbar nerve root compression. Studies have shown that nitric oxide synthase （NOS） and calcitonin gene-related peptide （CGRP） are involved in secondary disc inflammation. OBJECTIVE： To observe the effects of warm acupuncture on the ultrastructure of inflammatory mediators in a rat model of lumbar nerve root compression, including NOS and CGRP contents. DESIGN, TIME AND SETTING： Randomized, controlled study, with molecular biological analysis, was performed at the Experimental Center, Sixth People＇s Hospital Affiliated to Shanghai Jiao Tong University, between September 2006 and April 2007. MATERIALS： Acupuncture needles and refined Moxa grains were purchased from Shanghai Taicheng Technology Development Co., Ltd., China; Mobic tablets were purchased from Shanghai Boehringer Ingelheim Pharmaceuticals Co., Ltd., China; enzyme linked immunosorbent assay （ELISA） kits for NOS and CGRP were purchased from ADL Biotechnology, Inc., USA. METHODS： A total of 50, healthy, adult Sprague-Dawley rats, were randomly divided into five groups normal, model, warm acupuncture, acupuncture, and drug, with 10 rats in each group. Rats in the four groups, excluding the normal group, were used to establish models of lumbar nerve root compression. After 3 days, Jiaji points were set using reinforcing-reducing manipulation in the warm acupuncture group. Moxa grains were burned on each needle, with 2 grains each daily. The acupuncture group was the same as the warm acupuncture group, with the exception of non-moxibustion. Mobic suspension （3.75 mg/kg） was used in the oral drug group, once a day. Treatment of each group lasted for 14 consecutive days. Modeling and medication were not performed in the normal group. MAIN OUTCOME MEASURES： The ultrastructure of damaged nerve roots was observed with transmission electron microscopy; NOS and CGRP contents were measured using ELISA. RESULTS： The changes of the radicular ultramicrostructure were characterized by Wallerian degeneration; nerve fibers were clearly demyelinated; axons collapsed or degenerated; outer Schwann cell cytoplasm was swollen and its nucleus was compacted. Compared with the normal group, NOS and CGRP contents in the nerve root compression zone in the model group were significantly increased （P 〈 0.01）. Nerve root edema was improved in the drug, acupuncture and the warm acupuncture groups over the model group. NOS and CGRP expressions were also decreased with the warm acupuncture group having the lowest concentration （P 〈 0.01）. CONCLUSION： In comparison to the known effects of Mobic drug and acupuncture treatments, the warm acupuncture significantly decreased NOS and CGRP expression which helped improve the ultrastructure of the compressed nerve root.

Experimental Study on the Repair Effect of Xianlinggubao Capsule on Osteoporotic Vertebral Compression Fracture in Rabbits

[Objectives]To observe the effect of Xianlinggubao Capsule on osteoporotic vertebral compression fracture(OVCF)in rabbits and the influence mechanism of the repair of fractures.[Methods]Female June age 30 rabbits were randomly divided into control group,model control group and Xianlinggubao group.After bilateral ovariectomy,the model control group and Xianlinggubao group were injected with dexamethasone continuously for 4 weeks,and then the OVCF compound model was established by surgery.The Xianlinggubao group was treated with Xianlinggubao at a dose of 300 mg/(kg·d)for 60 d,while the blank control group and the model control group were treated with the same amount of normal saline for 60 d.The number of blood vessels and the expression of bone morphogenetic protein-2(BMP-2)were detected by immunohistochemical staining and the bone mineral density(BMD)in the callus of the third lumbar fracture area of rabbits was measured.The content of serum phosphorus(P),alkaline phosphatase(ALP)and total calcium(TCa)in rabbit venous blood were measured by automatic biochemical analyzer.The content of vascular endothelial growth factor(VEGF)and platelet-derived growth factor(PDGF)in rabbit venous blood were measured by ELISA kit.[Results]The number of blood vessels and the expression of BMP-2 in the callus of the third lumbar fracture area of rabbits was high in Xianlinggubao group,the content of serum P,ALP,TCa,VEGF and PDGF was obviously increased,BMD was obviously increased,the bone microstructure of the third lumbar vertebrae fracture area of rabbits was basically restored.Compared with the model control group(P<0.05),the difference was statistically significant.[Conclusions]Xianlinggubao Capsule can increase calcium and phosphorus deposition,promote the formation of blood vessels in the fracture area of OVCF in rabbits,and have a strong repair effect on OVCF in rabbits.

An Efficient Approach to Escalate the Speed of Training Convolution Neural Networks

Deep neural networks excel at image identification and computer vision applications such as visual product search, facial recognition, medical image analysis, object detection, semantic segmentation,instance segmentation, and many others. In image and video recognition applications, convolutional neural networks(CNNs) are widely employed. These networks provide better performance but at a higher cost of computation. With the advent of big data, the growing scale of datasets has made processing and model training a time-consuming operation, resulting in longer training times. Moreover, these large scale datasets contain redundant data points that have minimum impact on the final outcome of the model. To address these issues, an accelerated CNN system is proposed for speeding up training by eliminating the noncritical data points during training alongwith a model compression method. Furthermore, the identification of the critical input data is performed by aggregating the data points at two levels of granularity which are used for evaluating the impact on the model output.Extensive experiments are conducted using the proposed method on CIFAR-10 dataset on ResNet models giving a 40% reduction in number of FLOPs with a degradation of just 0.11% accuracy.

Low rank optimization for efficient deep learning:making a balance between compact architecture and fast training

Deep neural networks(DNNs)have achieved great success in many data processing applications.However,high computational complexity and storage cost make deep learning difficult to be used on resource-constrained devices,and it is not environmental-friendly with much power cost.In this paper,we focus on low-rank optimization for efficient deep learning techniques.In the space domain,DNNs are compressed by low rank approximation of the network parameters,which directly reduces the storage requirement with a smaller number of network parameters.In the time domain,the network parameters can be trained in a few subspaces,which enables efficient training for fast convergence.The model compression in the spatial domain is summarized into three categories as pre-train,pre-set,and compression-aware methods,respectively.With a series of integrable techniques discussed,such as sparse pruning,quantization,and entropy coding,we can ensemble them in an integration framework with lower computational complexity and storage.In addition to summary of recent technical advances,we have two findings for motivating future works.One is that the effective rank,derived from the Shannon entropy of the normalized singular values,outperforms other conventional sparse measures such as the?_1 norm for network compression.The other is a spatial and temporal balance for tensorized neural networks.For accelerating the training of tensorized neural networks,it is crucial to leverage redundancy for both model compression and subspace training.

BLOWUP CRITERION FOR THE COMPRESSIBLE FLUID-PARTICLE INTERACTION MODEL IN 3D WITH VACUUM

In this article, we consider the blowup criterion for the local strong solution to the compressible fluid-particle interaction model in dimension three with vacuum. We establish a BKM type criterion for possible breakdown of such solutions at critical time in terms of both the L^∞ （0, T; L^6）-norm of the density of particles and the ^L1（0, T; L^∞）-norm of the deformation tensor of velocity gradient.

REVIEW ON MATHEMATICAL ANALYSIS OF SOME TWO-PHASE FLOW MODELS

The two-phase flow models are commonly used in industrial applications, such as nuclear, power, chemical-process, oil-and-gas, cryogenics, bio-medical, micro-technology and so on. This is a survey paper on the study of compressible nonconservative two-fluid model, drift-flux model and viscous liquid-gas two-phase flow model. We give the research developments of these three two-phase flow models, respectively. In the last part, we give some open problems about the above models.

A BLOW-UP CRITERION OF STRONG SOLUTIONS TO THE QUANTUM HYDRODYNAMIC MODEL

In this article,we focus on the short time strong solution to a compressible quantum hydrodynamic model.We establish a blow-up criterion about the solutions of the compressible quantum hydrodynamic model in terms of the gradient of the velocity,the second spacial derivative of the square root of the density,and the first order time derivative and first order spacial derivative of the square root of the density.

Using structure restoration maps to comprehensively identify potential faults and fractures in compressional structures

Faults and fractures of multiple scales are frequently induced and generated in compressional structural system. Comprehensive identification of these potential faults and fractures that cannot be distinguished directly from seismic profile of the complex structures is still an unanswered problem. Based on the compressional structural geometry and kinematics theories as well as the structural interpretation from seismic data, a set of techniques is established for the identification of potential faults and fractures in compressional structures. Firstly, three-dimensional(3D) patterns and characteristics of the faults directly interpreted from seismic profile were illustrated by 3D structural model. Then, the unfolding index maps, the principal structural curvature maps, and tectonic stress field maps were obtained from structural restoration. Moreover, potential faults and fractures in compressional structures were quantitatively identified relying on comprehensive analysis of these three maps. Successful identification of the potential faults and fractures in Mishrif limestone formation and in Asmari dolomite formation of Buzurgan anticline in Iraq demonstrates the applicability and reliability of these techniques.

Selection of regression models for predicting strength and deformability properties of rocks using GA

Recently,many regression models have been presented for prediction of mechanical parameters of rocks regarding to rock index properties.Although statistical analysis is a common method for developing regression models,but still selection of suitable transformation of the independent variables in a regression model is diffcult.In this paper,a genetic algorithm(GA)has been employed as a heuristic search method for selection of best transformation of the independent variables(some index properties of rocks)in regression models for prediction of uniaxial compressive strength(UCS)and modulus of elasticity(E).Firstly,multiple linear regression(MLR)analysis was performed on a data set to establish predictive models.Then,two GA models were developed in which root mean squared error(RMSE)was defned as ftness function.Results have shown that GA models are more precise than MLR models and are able to explain the relation between the intrinsic strength/elasticity properties and index properties of rocks by simple formulation and accepted accuracy.

MoTransFrame:Model Transfer Framework for CNNs on Low-Resource Edge Computing Node

Deep learning technology has been widely used in computer vision,speech recognition,natural language processing,and other related fields.The deep learning algorithm has high precision and high reliability.However,the lack of resources in the edge terminal equipment makes it difficult to run deep learning algorithms that require more memory and computing power.In this paper,we propose MoTransFrame,a general model processing framework for deep learning models.Instead of designing a model compression algorithm with a high compression ratio,MoTransFrame can transplant popular convolutional neural networks models to resources-starved edge devices promptly and accurately.By the integration method,Deep learning models can be converted into portable projects for Arduino,a typical edge device with limited resources.Our experiments show that MoTransFrame has good adaptability in edge devices with limited memories.It is more flexible than other model transplantation methods.It can keep a small loss of model accuracy when the number of parameters is compressed by tens of times.At the same time,the computational resources needed in the reasoning process are less than what the edge node could handle.

Comparative Study on Constitutive Models to Predict Flow Stress of Fe-Cr-Ni Preform Reinforced Al-Si-Cu-Ni-Mg Composite

The isothermal compression tests were carried out on Gleeble-3500 thermal-mechanical simulation machine in a temperature range of 298-473 K and strain rate range of 0.001-10 s^-1. The experimental results show that the flow stress data are negatively correlated with temperature for temperature softening, and the strain rates sensitivity of this composite increases with elevating temperature. Based on the experimental data, Johnson-Cook, modified Johnson-Cook and Arrhenius constitutive models were established. The accuracy of these three constitutive models was analyzed and compared. The results show that the values predicted by Johnson-Cook model could not agree well with the experimental values. The prediction accuracy of Arrhenius model is higher than that of Johnson-Cook model but lower than that of the Modified Johnson-Cook model.

Real-time modular dynamic modeling for compression system of altitude ground test facilities

Modeling of a centrifugal compressor is of great significance to surge characteristics and fluid dynamics in the Altitude Ground Test Facilities(AGTF).Real-time Modular Dynamic System Greitzer(MDSG)modeling for dynamic response and simulation of the compression system is introduced.The centrifugal compressor,pipeline network,and valve are divided into pressure output type and mass flow output type for module modeling,and the two types of components alternate when the system is established.The pressure loss and thermodynamics of the system are considered.An air supply compression system of AGTF is modeled and simulated by the MDSG model.The simulation results of mass flow,pressure,and temperature are compared with the experimental results,and the error is less than 5%,which demonstrates the reliability,practicability,and universality of the MDSG model.

A Secure and Effective Energy-Aware Fixed-Point Quantization Scheme for Asynchronous Federated Learning

Asynchronous federated learning(AsynFL)can effectivelymitigate the impact of heterogeneity of edge nodes on joint training while satisfying participant user privacy protection and data security.However,the frequent exchange of massive data can lead to excess communication overhead between edge and central nodes regardless of whether the federated learning(FL)algorithm uses synchronous or asynchronous aggregation.Therefore,there is an urgent need for a method that can simultaneously take into account device heterogeneity and edge node energy consumption reduction.This paper proposes a novel Fixed-point Asynchronous Federated Learning(FixedAsynFL)algorithm,which could mitigate the resource consumption caused by frequent data communication while alleviating the effect of device heterogeneity.FixedAsynFL uses fixed-point quantization to compress the local and global models in AsynFL.In order to balance energy consumption and learning accuracy,this paper proposed a quantization scale selection mechanism.This paper examines the mathematical relationship between the quantization scale and energy consumption of the computation/communication process in the FixedAsynFL.Based on considering the upper bound of quantization noise,this paper optimizes the quantization scale by minimizing communication and computation consumption.This paper performs pertinent experiments on the MNIST dataset with several edge nodes of different computing efficiency.The results show that the FixedAsynFL algorithm with an 8-bit quantization can significantly reduce the communication data size by 81.3%and save the computation energy in the training phase by 74.9%without significant loss of accuracy.According to the experimental results,we can see that the proposed AsynFixedFL algorithm can effectively solve the problem of device heterogeneity and energy consumption limitation of edge nodes.

Effective Model Compression via Stage-wise Pruning

Automated machine learning(AutoML)pruning methods aim at searching for a pruning strategy automatically to reduce the computational complexity of deep convolutional neural networks(deep CNNs).However,some previous work found that the results of many Auto-ML pruning methods cannot even surpass the results of the uniformly pruning method.In this paper,the ineffectiveness of Auto-ML pruning,which is caused by unfull and unfair training of the supernet,is shown.A deep supernet suffers from unfull training because it contains too many candidates.To overcome the unfull training,a stage-wise pruning(SWP)method is proposed,which splits a deep supernet into several stage-wise supernets to reduce the candidate number and utilize inplace distillation to supervise the stage training.Besides,a wide supernet is hit by unfair training since the sampling probability of each channel is unequal.Therefore,the fullnet and the tinynet are sampled in each training iteration to ensure that each channel can be overtrained.Remarkably,the proxy performance of the subnets trained with SWP is closer to the actual performance than that of most of the previous AutoML pruning work.Furthermore,experiments show that SWP achieves the state-of-the-art in both CIFAR-10 and ImageNet under the mobile setting.

A pruning-then-quantization model compression framework for facial emotion recognition

Facial emotion recognition achieves great success with the help of large neural models but also fails to be applied in practical situations due to the large model size of neural methods.To bridge this gap,in this paper,we combine two mainstream model compression methods(pruning and quantization)together,and propose a pruningthen-quantization framework to compress the neural models for facial emotion recognition tasks.Experiments on three datasets show that our model could achieve a high model compression ratio and maintain the model’s high performance well.Besides,We analyze the layer-wise compression performance of our proposed framework to explore its effect and adaptability in fine-grained modules.

基底收缩对挤压构造变形特征影响——来自砂箱实验的启示

进行了 4个砂箱模型的模拟实验。其中基底收缩单侧挤压剖面模型以形成后冲叠瓦式断层为特征 ,基底无收缩单侧挤压剖面模型则以发育前冲叠瓦断层为特征。基底收缩斜向双侧挤压平面模型形成两组不同走向的逆断层 ,一组为与挤压带边界基本平行的主控断层 ,另一组为与挤压方向近于垂直的雁列断层。而基底无收缩斜向双侧挤压平面模型仅形成一组与挤压带边界基本平行的逆断层。这些模型的实验结果表明 ,基底收缩对挤压构造变形特征具有重要的影响。后冲 (或向陆冲向 )叠瓦式这种较为罕见挤压构造、以及与挤压方向近垂直的雁列断层的形成与基底的收缩作用有关。对于类似于焉耆盆地这种通常认为是压剪形成的挤压构造组合 ,笔者认为用“基底收缩条件下的斜向挤压作用”可更加贴切地加以解释。

Failure behavior and strength model of blocky rock mass with and without rockbolts

To better understand the failure behaviours and strength of bolt-reinforced blocky rocks,large scale extensive laboratory experiments are carried out on blocky rock-like specimens with and without rockbolt reinforcement.The results show that both shear failure and tensile failure along joint surfaces are observed but the shear failure is a main controlling factor for the peak strength of the rock mass with and without rockbolts.The rockbolts are necked and shear deformation simultaneously happens in bolt reinforced rock specimens.As the joint dip angle increases,the joint shear failure becomes more dominant.The number of rockbolts has a significant impact on the peak strain and uniaxial compressive strength(UCS),but little influence on the deformation modulus of the rock mass.Using the Winkler beam model to represent the rockbolt behaviours,an analytical model for the prediction of the strength of boltreinforced blocky rocks is proposed.Good agreement between the UCS values predicted by proposed model and obtained from experiments suggest an encouraging performance of the proposed model.In addition,the performance of the proposed model is further assessed using published results in the literature,indicating the proposed model can be used effectively in the prediction of UCS of bolt-reinforced blocky rocks.

沉积物中天然气水合物微观分布模式及其声学响应特征

天然气水合物(以下简称水合物)的微观分布与其形成条件、流体运移通道等密切相关,对天然气水合物资源勘探与评价具有重要意义。为了解水合物在固结沉积物和松散沉积物中的微观分布及其声学响应特征,采用超声和时域反射联合探测技术实时测量了沉积物中水合物生成、分解过程中声速等参数的变化情况。结果表明,在固结沉积物中,水合物先在孔隙流体中形成,随后逐渐向骨架靠拢,当饱和度大于30%后水合物开始胶结沉积物颗粒生成,这种胶结模式会圈闭部分孔隙流体,使之因得不到气源的补充而难以形成水合物,因此固结沉积物中水合物饱和度最终为65.5%左右;在松散沉积物中,少量的水合物(饱和度1%左右)胶结沉积物颗粒生成,当饱和度大于1%后水合物开始在孔隙流体中以悬浮状形态生成,由于水合物与沉积物颗粒间尚有流体运移通道,水合物能进一步生成,最终几乎完全充填沉积物孔隙。不同的水合物微观分布特征对沉积物的声速具有不同影响:水合物在孔隙流体中生成时,10%饱和度的水合物对固结沉积物的声速影响不明显;当水合物胶结沉积物颗粒生成时,约1%饱和度的水合物可使松散沉积物的纵波速度增长200～300m/s。