Activation Functions Effect on Fractal Coding Using Neural Networks

Activation functions play an essential role in converting the output of the artificial neural network into nonlinear results,since without this nonlinearity,the results of the network will be less accurate.Nonlinearity is the mission of all nonlinear functions,except for polynomials.The activation function must be dif-ferentiable for backpropagation learning.This study’s objective is to determine the best activation functions for the approximation of each fractal image.Different results have been attained using Matlab and Visual Basic programs,which indi-cate that the bounded function is more helpful than other functions.The non-lin-earity of the activation function is important when using neural networks for coding fractal images because the coefficients of the Iterated Function System are different according to the different types of fractals.The most commonly cho-sen activation function is the sigmoidal function,which produces a positive value.Other functions,such as tansh or arctan,whose values can be positive or negative depending on the network input,tend to train neural networks faster.The coding speed of the fractal image is different depending on the appropriate activation function chosen for each fractal shape.In this paper,we have provided the appro-priate activation functions for each type of system of iterated functions that help the network to identify the transactions of the system.

Multistability of delayed complex-valued recurrent neural networks with discontinuous real-imaginarytype activation functions

In this paper, the multistability issue is discussed for delayed complex-valued recurrent neural networks with discontinuous real-imaginary-type activation functions. Based on a fixed theorem and stability definition, sufficient criteria are established for the existence and stability of multiple equilibria of complex-valued recurrent neural networks. The number of stable equilibria is larger than that of real-valued recurrent neural networks, which can be used to achieve high-capacity associative memories. One numerical example is provided to show the effectiveness and superiority of the presented results.

Neural Networks on an FPGA and Hardware-Friendly Activation Functions

This paper describes our implementation of several neural networks built on a field programmable gate array (FPGA) and used to recognize a handwritten digit dataset—the Modified National Institute of Standards and Technology (MNIST) database. We also propose a novel hardware-friendly activation function called the dynamic Rectifid Linear Unit (ReLU)—D-ReLU function that achieves higher performance than traditional activation functions at no cost to accuracy. We built a 2-layer online training multilayer perceptron (MLP) neural network on an FPGA with varying data width. Reducing the data width from 8 to 4 bits only reduces prediction accuracy by 11%, but the FPGA area decreases by 41%. Compared to networks that use the sigmoid functions, our proposed D-ReLU function uses 24% - 41% less area with no loss to prediction accuracy. Further reducing the data width of the 3-layer networks from 8 to 4 bits, the prediction accuracies only decrease by 3% - 5%, with area being reduced by 9% - 28%. Moreover, FPGA solutions have 29 times faster execution time, even despite running at a 60× lower clock rate. Thus, FPGA implementations of neural networks offer a high-performance, low power alternative to traditional software methods, and our novel D-ReLU activation function offers additional improvements to performance and power saving.

Learning Specialized Activation Functions for Physics-Informed Neural Networks

Physics-informed neural networks(PINNs)are known to suffer from optimization difficulty.In this work,we reveal the connection between the optimization difficulty of PINNs and activation functions.Specifically,we show that PINNs exhibit high sensitivity to activation functions when solving PDEs with distinct properties.Existing works usually choose activation functions by inefficient trial-and-error.To avoid the inefficient manual selection and to alleviate the optimization difficulty of PINNs,we introduce adaptive activation functions to search for the optimal function when solving different problems.We compare different adaptive activation functions and discuss their limitations in the context of PINNs.Furthermore,we propose to tailor the idea of learning combinations of candidate activation functions to the PINNs optimization,which has a higher requirement for the smoothness and diversity on learned functions.This is achieved by removing activation functions which cannot provide higher-order derivatives from the candidate set and incorporating elementary functions with different properties according to our prior knowledge about the PDE at hand.We further enhance the search space with adaptive slopes.The proposed adaptive activation function can be used to solve different PDE systems in an interpretable way.Its effectiveness is demonstrated on a series of benchmarks.Code is available at https://github.com/LeapLabTHU/AdaAFforPINNs.

GAAF:Searching Activation Functions for Binary Neural Networks Through Genetic Algorithm

Binary neural networks(BNNs)show promising utilization in cost and power-restricted domains such as edge devices and mobile systems.This is due to its significantly less computation and storage demand,but at the cost of degraded performance.To close the accuracy gap,in this paper we propose to add a complementary activation function(AF)ahead of the sign based binarization,and rely on the genetic algorithm(GA)to automatically search for the ideal AFs.These AFs can help extract extra information from the input data in the forward pass,while allowing improved gradient approximation in the backward pass.Fifteen novel AFs are identified through our GA-based search,while most of them show improved performance(up to 2.54%on ImageNet)when testing on different datasets and network models.Interestingly,periodic functions are identified as a key component for most of the discovered AFs,which rarely exist in human designed AFs.Our method offers a novel approach for designing general and application-specific BNN architecture.GAAF will be released on GitHub.

Global Exponential Periodicity of a Class of Recurrent Neural Networks with Non-Monotone Activation Functions and Time-Varying Delays

The stability of a periodic oscillation and the global exponential class of recurrent neural networks with non-monotone activation functions and time-varying delays are analyzed. For two sets of activation functions, some algebraic criteria for ascertaining global exponential periodicity and global exponential stability of the class of recurrent neural networks are derived by using the comparison principle and the theory of monotone operator. These conditions are easy to check in terms of system parameters. In addition, we provide a new and efficacious method for the qualitative analysis of various neural networks.

Robust stability of mixed Cohen–Grossberg neural networks with discontinuous activation functions

Purpose–The purpose of this paper is to develop a method for the existence,uniqueness and globally robust stability of the equilibrium point for Cohen–Grossberg neural networks with time-varying delays,continuous distributed delays and a kind of discontinuous activation functions.Design/methodology/approach–Basedonthe Leray–Schauderalternativetheoremand chainrule,by using a novel integral inequality dealing with monotone non-decreasing function,the authors obtain a delay-dependent sufficient condition with less conservativeness for robust stability of considered neural networks.Findings–Itturns out thattheauthors’delay-dependent sufficientcondition canbeformed intermsof linear matrix inequalities conditions.Two examples show the effectiveness of the obtained results.Originality/value–The novelty of the proposed approach lies in dealing with a new kind of discontinuous activation functions by using the Leray–Schauder alternative theorem,chain rule and a novel integral inequality on monotone non-decreasing function.

A Universal Activation Function for Deep Learning

Recently,deep learning has achieved remarkable results in fields that require human cognitive ability,learning ability,and reasoning ability.Activation functions are very important because they provide the ability of artificial neural networks to learn complex patterns through nonlinearity.Various activation functions are being studied to solve problems such as vanishing gradients and dying nodes that may occur in the deep learning process.However,it takes a lot of time and effort for researchers to use the existing activation function in their research.Therefore,in this paper,we propose a universal activation function(UA)so that researchers can easily create and apply various activation functions and improve the performance of neural networks.UA can generate new types of activation functions as well as functions like traditional activation functions by properly adjusting three hyperparameters.The famous Convolutional Neural Network(CNN)and benchmark datasetwere used to evaluate the experimental performance of the UA proposed in this study.We compared the performance of the artificial neural network to which the traditional activation function is applied and the artificial neural network to which theUA is applied.In addition,we evaluated the performance of the new activation function generated by adjusting the hyperparameters of theUA.The experimental performance evaluation results showed that the classification performance of CNNs improved by up to 5%through the UA,although most of them showed similar performance to the traditional activation function.

Convolution-Based Heterogeneous Activation Facility for Effective Machine Learning of ECG Signals

Machine Learning(ML)and Deep Learning(DL)technologies are revolutionizing the medical domain,especially with Electrocardiogram(ECG),by providing new tools and techniques for diagnosing,treating,and preventing diseases.However,DL architectures are computationally more demanding.In recent years,researchers have focused on combining the computationally less intensive portion of the DL architectures with ML approaches,say for example,combining the convolutional layer blocks of Convolution Neural Networks(CNNs)into ML algorithms such as Extreme Gradient Boosting(XGBoost)and K-Nearest Neighbor(KNN)resulting in CNN-XGBoost and CNN-KNN,respectively.However,these approaches are homogenous in the sense that they use a fixed Activation Function(AFs)in the sequence of convolution and pooling layers,thereby limiting the ability to capture unique features.Since various AFs are readily available and each could capture unique features,we propose a Convolutionbased Heterogeneous Activation Facility(CHAF)which uses multiple AFs in the convolution layer blocks,one for each block,with a motivation of extracting features in a better manner to improve the accuracy.The proposed CHAF approach is validated on PTB and shown to outperform the homogeneous approaches such as CNN-KNN and CNN-XGBoost.For PTB dataset,proposed CHAF-KNN has an accuracy of 99.55%and an F1 score of 99.68%in just 0.008 s,outperforming the state-of-the-art CNN-XGBoost which has an accuracy of 99.38%and an F1 score of 99.32%in 1.23 s.To validate the generality of the proposed CHAF,experiments were repeated on MIT-BIH dataset,and the proposed CHAF-KNN is shown to outperform CNN-KNN and CNN-XGBoost.

Interpretation and characterization of rate of penetration intelligent prediction model

Accurate prediction of the rate of penetration(ROP)is significant for drilling optimization.While the intelligent ROP prediction model based on fully connected neural networks(FNN)outperforms traditional ROP equations and machine learning algorithms,its lack of interpretability undermines its credibility.This study proposes a novel interpretation and characterization method for the FNN ROP prediction model using the Rectified Linear Unit(ReLU)activation function.By leveraging the derivative of the ReLU function,the FNN function calculation process is transformed into vector operations.The FNN model is linearly characterized through further simplification,enabling its interpretation and analysis.The proposed method is applied in ROP prediction scenarios using drilling data from three vertical wells in the Tarim Oilfield.The results demonstrate that the FNN ROP prediction model with ReLU as the activation function performs exceptionally well.The relative activation frequency curve of hidden layer neurons aids in analyzing the overfitting of the FNN ROP model and determining drilling data similarity.In the well sections with similar drilling data,averaging the weight parameters enables linear characterization of the FNN ROP prediction model,leading to the establishment of a corresponding linear representation equation.Furthermore,the quantitative analysis of each feature's influence on ROP facilitates the proposal of drilling parameter optimization schemes for the current well section.The established linear characterization equation exhibits high precision,strong stability,and adaptability through the application and validation across multiple well sections.

Uncertainty-Aware Physical Simulation of Neural Radiance Fields for Fluids

This paper presents a novel framework aimed at quantifying uncertainties associated with the 3D reconstruction of smoke from2Dimages.This approach reconstructs color and density fields from 2D images using Neural Radiance Field(NeRF)and improves image quality using frequency regularization.The NeRF model is obtained via joint training ofmultiple artificial neural networks,whereby the expectation and standard deviation of density fields and RGB values can be evaluated for each pixel.In addition,customized physics-informed neural network(PINN)with residual blocks and two-layer activation functions are utilized to input the density fields of the NeRF into Navier-Stokes equations and convection-diffusion equations to reconstruct the velocity field.The velocity uncertainties are also evaluated through ensemble learning.The effectiveness of the proposed algorithm is demonstrated through numerical examples.The presentmethod is an important step towards downstream tasks such as reliability analysis and robust optimization in engineering design.

Nonparametric Statistical Feature Scaling Based Quadratic Regressive Convolution Deep Neural Network for Software Fault Prediction

The development of defect prediction plays a significant role in improving software quality. Such predictions are used to identify defective modules before the testing and to minimize the time and cost. The software with defects negatively impacts operational costs and finally affects customer satisfaction. Numerous approaches exist to predict software defects. However, the timely and accurate software bugs are the major challenging issues. To improve the timely and accurate software defect prediction, a novel technique called Nonparametric Statistical feature scaled QuAdratic regressive convolution Deep nEural Network (SQADEN) is introduced. The proposed SQADEN technique mainly includes two major processes namely metric or feature selection and classification. First, the SQADEN uses the nonparametric statistical Torgerson–Gower scaling technique for identifying the relevant software metrics by measuring the similarity using the dice coefficient. The feature selection process is used to minimize the time complexity of software fault prediction. With the selected metrics, software fault perdition with the help of the Quadratic Censored regressive convolution deep neural network-based classification. The deep learning classifier analyzes the training and testing samples using the contingency correlation coefficient. The softstep activation function is used to provide the final fault prediction results. To minimize the error, the Nelder–Mead method is applied to solve non-linear least-squares problems. Finally, accurate classification results with a minimum error are obtained at the output layer. Experimental evaluation is carried out with different quantitative metrics such as accuracy, precision, recall, F-measure, and time complexity. The analyzed results demonstrate the superior performance of our proposed SQADEN technique with maximum accuracy, sensitivity and specificity by 3%, 3%, 2% and 3% and minimum time and space by 13% and 15% when compared with the two state-of-the-art methods.

Elevated brain temperature under severe heat exposure impairs cortical motor activity and executive function

Background:Excessive heat exposure can lead to hyperthermia in humans,which impairs physical performance and disrupts cognitive function.While heat is a known physiological stressor,it is unclear how severe heat stress affects brain physiology and function.Methods:Eleven healthy participants were subjected to heat stress from prolonged exercise or warm water immersion until their rectal temperatures(T_(re))attained 39.5℃,inducing exertional or passive hyperthermia,respectively.In a separate trial,blended ice was ingested before and during exercise as a cooling strategy.Data were compared to a control condition with seated rest(normothermic).Brain temperature(T_(br)),cerebral perfusion,and task-based brain activity were assessed using magnetic resonance imaging techniques.Results:T_(br)in motor cortex was found to be tightly regulated at rest(37.3℃±0.4℃(mean±SD))despite fluctuations in T_(re).With the development of hyperthermia,T_(br)increases and dovetails with the rising T_(re).Bilateral motor cortical activity was suppressed during high-intensity plantarflexion tasks,implying a reduced central motor drive in hyperthermic participants(T_(re)=38.5℃±0.1℃).Global gray matter perfusion and regional perfusion in sensorimotor cortex were reduced with passive hyperthermia.Executive function was poorer under a passive hyperthermic state,and this could relate to compromised visual processing as indicated by the reduced activation of left lateral-occipital cortex.Conversely,ingestion of blended ice before and during exercise alleviated the rise in both T_(re)and T_(bc)and mitigated heat-related neural perturbations.Conclusion:Severe heat exposure elevates T_(br),disrupts motor cortical activity and executive function,and this can lead to impairment of physical and cognitive performance.

Large-scale self-normalizing neural networks

Self-normalizing neural networks(SNN)regulate the activation and gradient flows through activation functions with the self-normalization property.As SNNs do not rely on norms computed from minibatches,they are more friendly to data parallelism,kernel fusion,and emerging architectures such as ReRAM-based accelerators.However,existing SNNs have mainly demonstrated their effectiveness on toy datasets and fall short in accuracy when dealing with large-scale tasks like ImageNet.They lack the strong normalization,regularization,and expression power required for wider,deeper models and larger-scale tasks.To enhance the normalization strength,this paper introduces a comprehensive and practical definition of the self-normalization property in terms of the stability and attractiveness of the statistical fixed points.It is comprehensive as it jointly considers all the fixed points used by existing studies:the first and second moment of forward activation and the expected Frobenius norm of backward gradient.The practicality comes from the analytical equations provided by our paper to assess the stability and attractiveness of each fixed point,which are derived from theoretical analysis of the forward and backward signals.The proposed definition is applied to a meta activation function inspired by prior research,leading to a stronger self-normalizing activation function named‘‘bi-scaled exponential linear unit with backward standardized’’(bSELU-BSTD).We provide both theoretical and empirical evidence to show that it is superior to existing studies.To enhance the regularization and expression power,we further propose scaled-Mixup and channel-wise scale&shift.With these three techniques,our approach achieves 75.23%top-1 accuracy on the ImageNet with Conv MobileNet V1,surpassing the performance of existing self-normalizing activation functions.To the best of our knowledge,this is the first SNN that achieves comparable accuracy to batch normalization on ImageNet.

Progress on the melanoidins produced by the Maillard reaction of fermented food and traditional Chinese medicine

FMs(Food-borne melanoidins)are brown high molecular weight polymers formed by the Mailiard reaction between carbohydrates and nitrogen-containing compounds during the processing of food or Traditional Chinese Medicine(TCM),and are widely found in food-borne products such as TCM concoctions,bakery,brewing,soy sauce,ferment and other food-borne products.FMs not only have a variety of biological activities,such as antioxidant,antibacterial,immunomodulation,regulation of intestinal flora,etc.,and can change the color,aroma and taste of food.The diversity of its components has become a research hotspot at home and abroad in recent years,with a wide range of application prospects.Therefore,this paper summarizes the existing information on FMs at home and abroad,mainly describes their preparation process,physicochemical properties,structural characteristics and functional activity research progress.Typical FMs,such as coffee,biscuits,wine and soy sauce in daily food,and Polygonatum,Perilla oil,Black ginseng,and Red jujube in T,were highlighted.Summarising the current status of research between the chemistry and pharmacodynamics of relevant FMs and presenting challenges and future recommendations for melanoidin research.In future research on FMs,one should pay more attention to basic research,especially isolation and purification and generation mechanisms,to further demonstrate the biological activity of FMs in vivo and in clinical trials.Thus,the potential value of its existence is deeply exploited to meet the needs of technology,production and health.

The cortical activation pattern during bilateral arm raising movements

Bilateral arm raising movements have been used in brain rehabilitation for a long time. However, no study has been reported on the effect of these movements on the cerebral cortex. In this study, using functional near infrared spectroscopy（f NIRS）, we attempted to investigate cortical activation generated during bilateral arm raising movements. Ten normal subjects were recruited for this study. f NIRS was performed using an f NIRS system with 49 channels. Bilateral arm raising movements were performed in sitting position at the rate of 0.5 Hz. We measured values of oxyhemoglobin and total hemoglobin in five regions of interest： the primary sensorimotor cortex, premotor cortex, supplementary motor area, prefrontal cortex, and posterior parietal cortex. During performance of bilateral arm raising movements, oxyhemoglobin and total hemoglobin values in the primary sensorimotor cortex, premotor cortex, supplementary motor area, and prefrontal cortex were similar, but higher in these regions than those in the prefrontal cortex. We observed activation of the arm somatotopic areas of the primary sensorimotor cortex and premotor cortex in both hemispheres during bilateral arm raising movements. According to this result, bilateral arm raising movements appeared to induce large-scale neuronal activation and therefore arm raising movements would be good exercise for recovery of brain functions.

Theoretical Study on the C-H Activation in Decarbonylation of Acetaldehyde by NiL_2(L=SO_3CH_3) Using Density Functional Theory

Density functional theory calculations were carried out to explore the potential energy surface（PES） associated with the gas-phase reaction of Ni L2（L=SO3CH3） with acetone. The geometries and energies of the reactants, intermediates, products and transition states of the triplet ground potential energy surfaces of [Ni, O, C2, H4] were obtained at the B3LYP/6-311＋＋G（d,p） levels in C,H,O atoms and B3LYP/ Lanl2 dz in Ni atom. It was found through our calculations that the decabonylation of acetaldehyde contains four steps including encounter complexation, C-C activation, aldehyde H-shift and nonreactive dissociation. The results revealed that C-C activation induced by Ni L2（L=SO3CH3） led to the decarbonylation of acetaldehyde.

A Novel Approach to Heart Failure Prediction and Classification through Advanced Deep Learning Model

In this study, the author will investigate and utilize advanced machine learning models related to two different methodologies to determine the best and most effective way to predict individuals with heart failure and cardiovascular diseases. The first methodology involves a list of classification machine learning algorithms, and the second methodology involves the use of a deep learning algorithm known as MLP or Multilayer Perceptrons. Globally, hospitals are dealing with cases related to cardiovascular diseases and heart failure as they are major causes of death, not only for overweight individuals but also for those who do not adopt a healthy diet and lifestyle. Often, heart failures and cardiovascular diseases can be caused by many factors, including cardiomyopathy, high blood pressure, coronary heart disease, and heart inflammation [1]. Other factors, such as irregular shocks or stress, can also contribute to heart failure or a heart attack. While these events cannot be predicted, continuous data from patients’ health can help doctors predict heart failure. Therefore, this data-driven research utilizes advanced machine learning and deep learning techniques to better analyze and manipulate the data, providing doctors with informative decision-making tools regarding a person’s likelihood of experiencing heart failure. In this paper, the author employed advanced data preprocessing and cleaning techniques. Additionally, the dataset underwent testing using two different methodologies to determine the most effective machine-learning technique for producing optimal predictions. The first methodology involved employing a list of supervised classification machine learning algorithms, including Naïve Bayes (NB), KNN, logistic regression, and the SVM algorithm. The second methodology utilized a deep learning (DL) algorithm known as Multilayer Perceptrons (MLPs). This algorithm provided the author with the flexibility to experiment with different layer sizes and activation functions, such as ReLU, logistic (sigmoid), and Tanh. Both methodologies produced optimal models with high-level accuracy rates. The first methodology involves a list of supervised machine learning algorithms, including KNN, SVM, Adaboost, Logistic Regression, Naive Bayes, and Decision Tree algorithms. They achieved accuracy rates of 86%, 89%, 89%, 81%, 79%, and 99%, respectively. The author clearly explained that Decision Tree algorithm is not suitable for the dataset at hand due to overfitting issues. Therefore, it was discarded as an optimal model to be used. However, the latter methodology (Neural Network) demonstrated the most stable and optimal accuracy, achieving over 87% accuracy while adapting well to real-life situations and requiring low computing power overall. A performance assessment and evaluation were carried out based on a confusion matrix report to demonstrate feasibility and performance. The author concluded that the performance of the model in real-life situations can advance not only the medical field of science but also mathematical concepts. Additionally, the advanced preprocessing approach behind the model can provide value to the Data Science community. The model can be further developed by employing various optimization techniques to handle even larger datasets related to heart failures. Furthermore, different neural network algorithms can be tested to explore alternative approaches and yield different results.

Impact point prediction guidance of ballistic missile in high maneuver penetration condition

An impact point prediction(IPP) guidance based on supervised learning is proposed to address the problem of precise guidance for the ballistic missile in high maneuver penetration condition.An accurate ballistic trajectory model is applied to generate training samples,and ablation experiments are conducted to determine the mapping relationship between the flight state and the impact point.At the same time,the impact point coordinates are decoupled to improve the prediction accuracy,and the sigmoid activation function is improved to ameliorate the prediction efficiency.Therefore,an IPP neural network model,which solves the contradiction between the accuracy and the speed of the IPP,is established.In view of the performance deviation of the divert control system,the mapping relationship between the guidance parameters and the impact deviation is analysed based on the variational principle.In addition,a fast iterative model of guidance parameters is designed for reference to the Newton iteration method,which solves the nonlinear strong coupling problem of the guidance parameter solution.Monte Carlo simulation results show that the prediction accuracy of the impact point is high,with a 3 σ prediction error of 4.5 m,and the guidance method is robust,with a 3 σ error of 7.5 m.On the STM32F407 singlechip microcomputer,a single IPP takes about 2.374 ms,and a single guidance solution takes about9.936 ms,which has a good real-time performance and a certain engineering application value.

Machine Learning Controller for DFIG Based Wind Conversion System

Renewable energy production plays a major role in satisfying electricity demand.Wind power conversion is one of the most popular renewable energy sources compared to other sources.Wind energy conversion has two major types of generators such as the Permanent Magnet Synchronous Generator(PMSG)and the Doubly Fed Induction Generator(DFIG).The maximum power tracking algo-rithm is a crucial controller,a wind energy conversion system for generating maximum power in different wind speed conditions.In this article,the DFIG wind energy conversion system was developed in Matrix Laboratory(MATLAB)and designed a machine learning(ML)algorithm for the rotor and grid side converter.The ML algorithm has been developed and trained in a MATLAB environment.There are two types of learning algorithms such as supervised and unsupervised learning.In this research supervised learning is used to power the neural networks and analysis is made for various hidden layers and activation functions.Simulation results are assessed to demonstrate the efficiency of the proposed system.