Self-normalizing neural networks(SNN)regulate the activation and gradient flows through activation functions with the self-normalization property.As SNNs do not rely on norms computed from minibatches,they are more fr...Self-normalizing neural networks(SNN)regulate the activation and gradient flows through activation functions with the self-normalization property.As SNNs do not rely on norms computed from minibatches,they are more friendly to data parallelism,kernel fusion,and emerging architectures such as ReRAM-based accelerators.However,existing SNNs have mainly demonstrated their effectiveness on toy datasets and fall short in accuracy when dealing with large-scale tasks like ImageNet.They lack the strong normalization,regularization,and expression power required for wider,deeper models and larger-scale tasks.To enhance the normalization strength,this paper introduces a comprehensive and practical definition of the self-normalization property in terms of the stability and attractiveness of the statistical fixed points.It is comprehensive as it jointly considers all the fixed points used by existing studies:the first and second moment of forward activation and the expected Frobenius norm of backward gradient.The practicality comes from the analytical equations provided by our paper to assess the stability and attractiveness of each fixed point,which are derived from theoretical analysis of the forward and backward signals.The proposed definition is applied to a meta activation function inspired by prior research,leading to a stronger self-normalizing activation function named‘‘bi-scaled exponential linear unit with backward standardized’’(bSELU-BSTD).We provide both theoretical and empirical evidence to show that it is superior to existing studies.To enhance the regularization and expression power,we further propose scaled-Mixup and channel-wise scale&shift.With these three techniques,our approach achieves 75.23%top-1 accuracy on the ImageNet with Conv MobileNet V1,surpassing the performance of existing self-normalizing activation functions.To the best of our knowledge,this is the first SNN that achieves comparable accuracy to batch normalization on ImageNet.展开更多
With the continuous growth of online news articles,there arises the necessity for an efficient abstractive summarization technique for the problem of information overloading.Abstractive summarization is highly complex...With the continuous growth of online news articles,there arises the necessity for an efficient abstractive summarization technique for the problem of information overloading.Abstractive summarization is highly complex and requires a deeper understanding and proper reasoning to come up with its own summary outline.Abstractive summarization task is framed as seq2seq modeling.Existing seq2seq methods perform better on short sequences;however,for long sequences,the performance degrades due to high computation and hence a two-phase self-normalized deep neural document summarization model consisting of improvised extractive cosine normalization and seq2seq abstractive phases has been proposed in this paper.The novelty is to parallelize the sequence computation training by incorporating feed-forward,the self-normalized neural network in the Extractive phase using Intra Cosine Attention Similarity(Ext-ICAS)with sentence dependency position.Also,it does not require any normalization technique explicitly.Our proposed abstractive Bidirectional Long Short Term Memory(Bi-LSTM)encoder sequence model performs better than the Bidirectional Gated Recurrent Unit(Bi-GRU)encoder with minimum training loss and with fast convergence.The proposed model was evaluated on the Cable News Network(CNN)/Daily Mail dataset and an average rouge score of 0.435 was achieved also computational training in the extractive phase was reduced by 59%with an average number of similarity computations.展开更多
In the case of Z＋^d（d ≥ 2）-the positive d-dimensional lattice points with partial ordering ≤, {Xk,k∈ Z＋^d} i.i.d, random variables with mean 0, Sn =∑k≤nXk and Vn^2 = ∑j≤nXj^2, the precise asymptotics for ∑...In the case of Z＋^d（d ≥ 2）-the positive d-dimensional lattice points with partial ordering ≤, {Xk,k∈ Z＋^d} i.i.d, random variables with mean 0, Sn =∑k≤nXk and Vn^2 = ∑j≤nXj^2, the precise asymptotics for ∑n1/｜n｜（log｜n｜dP（｜Sn/Vn｜≥ε√log log｜n｜） and ∑n（logn｜）b/｜n｜（log｜n｜）^d-1P（｜Sn/Vn｜≥ε√log n）,as ε↓0,is established.展开更多
The past two decades have witnessed the active development of a rich probability theory of Studentized statistics or self-normalized processes, typified by Student’s t-statistic as introduced by W. S. Gosset more tha...The past two decades have witnessed the active development of a rich probability theory of Studentized statistics or self-normalized processes, typified by Student’s t-statistic as introduced by W. S. Gosset more than a century ago, and their applications to statistical problems in high dimensions, including feature selection and ranking, large-scale multiple testing and sparse, high dimensional signal detection. Many of these applications rely on the robustness property of Studentization/self-normalization against heavy-tailed sampling distributions. This paper gives an overview of the salient progress of self-normalized limit theory, from Student’s t-statistic to more general Studentized nonlinear statistics. Prototypical examples include Studentized one- and two-sample U-statistics. Furthermore, we go beyond independence and glimpse some very recent advances in self-normalized moderate deviations under dependence.展开更多
In this article, the unit root test for AR（p） model with GARCH errors is considered. The Dickey-Fuller test statistics are rewritten in the form of self-normalized sums, and the asymptotic distribution of the test s...In this article, the unit root test for AR（p） model with GARCH errors is considered. The Dickey-Fuller test statistics are rewritten in the form of self-normalized sums, and the asymptotic distribution of the test statistics is derived under the weak conditions.展开更多
In this case study, we would like to illustrate the utility of characteristic functions, using an example of a sample statistic defined for samples from Cauchy distribution. The derivation of the corresponding asympto...In this case study, we would like to illustrate the utility of characteristic functions, using an example of a sample statistic defined for samples from Cauchy distribution. The derivation of the corresponding asymptotic probability density function is based on [1], elaborating and expanding the individual steps of their presentation, and including a small extension;our reason for such a plagiarism is to make the technique, its mathematical tools and ingenious arguments available to the widest possible audience.展开更多
Let variables in the {X, Xn, n ≥ 1} be a sequence of strictly stationary φ-mixing positive random domain of attraction of the normal law. Under some suitable conditions the principle for self-normalized products of ...Let variables in the {X, Xn, n ≥ 1} be a sequence of strictly stationary φ-mixing positive random domain of attraction of the normal law. Under some suitable conditions the principle for self-normalized products of partial sums is obtained.展开更多
Recently,fuzzy multi-sets have come to the forefront of scientists’interest and have been used in algebraic structures such asmulti-groups,multirings,anti-fuzzy multigroup and(α,γ)-anti-fuzzy subgroups.In this pape...Recently,fuzzy multi-sets have come to the forefront of scientists’interest and have been used in algebraic structures such asmulti-groups,multirings,anti-fuzzy multigroup and(α,γ)-anti-fuzzy subgroups.In this paper,we first summarize the knowledge about the algebraic structure of fuzzy multi-sets such as(α,γ)-anti-multi-fuzzy subgroups.In a way,the notion of anti-fuzzy multigroup is an application of anti-fuzzy multi sets to the theory of group.The concept of anti-fuzzy multigroup is a complement of an algebraic structure of a fuzzy multi set that generalizes both the theories of classical group and fuzzy group.The aim of this paper is to highlight the connection between fuzzy multi-sets and algebraic structures from an anti-fuzzification point of view.Therefore,in this paper,we define(α,γ)-antimulti-fuzzy subgroups,(α,γ)-anti-multi-fuzzy normal subgroups,(α,γ)-antimulti-fuzzy homomorphism on(α,γ)-anti-multi-fuzzy subgroups and these been explicated some algebraic structures.Then,we introduce the concept(α,γ)-anti-multi-fuzzy subgroups and(α,γ)-anti-multi-fuzzy normal subgroups and of their properties.This new concept of homomorphism as a bridge among set theory,fuzzy set theory,anti-fuzzy multi sets theory and group theory and also shows the effect of anti-fuzzy multi sets on a group structure.Certain results that discuss the(α,γ)cuts of anti-fuzzy multigroup are explored.展开更多
基金National Key R&D Program of China(2018AAA0102600)National Natural Science Foundation of China(No.61876215,62106119)+1 种基金Beijing Academy of Artificial Intelligence(BAAI),ChinaChinese Institute for Brain Research,Beijing,and the Science and Technology Major Project of Guangzhou,China(202007030006).
文摘Self-normalizing neural networks(SNN)regulate the activation and gradient flows through activation functions with the self-normalization property.As SNNs do not rely on norms computed from minibatches,they are more friendly to data parallelism,kernel fusion,and emerging architectures such as ReRAM-based accelerators.However,existing SNNs have mainly demonstrated their effectiveness on toy datasets and fall short in accuracy when dealing with large-scale tasks like ImageNet.They lack the strong normalization,regularization,and expression power required for wider,deeper models and larger-scale tasks.To enhance the normalization strength,this paper introduces a comprehensive and practical definition of the self-normalization property in terms of the stability and attractiveness of the statistical fixed points.It is comprehensive as it jointly considers all the fixed points used by existing studies:the first and second moment of forward activation and the expected Frobenius norm of backward gradient.The practicality comes from the analytical equations provided by our paper to assess the stability and attractiveness of each fixed point,which are derived from theoretical analysis of the forward and backward signals.The proposed definition is applied to a meta activation function inspired by prior research,leading to a stronger self-normalizing activation function named‘‘bi-scaled exponential linear unit with backward standardized’’(bSELU-BSTD).We provide both theoretical and empirical evidence to show that it is superior to existing studies.To enhance the regularization and expression power,we further propose scaled-Mixup and channel-wise scale&shift.With these three techniques,our approach achieves 75.23%top-1 accuracy on the ImageNet with Conv MobileNet V1,surpassing the performance of existing self-normalizing activation functions.To the best of our knowledge,this is the first SNN that achieves comparable accuracy to batch normalization on ImageNet.
文摘With the continuous growth of online news articles,there arises the necessity for an efficient abstractive summarization technique for the problem of information overloading.Abstractive summarization is highly complex and requires a deeper understanding and proper reasoning to come up with its own summary outline.Abstractive summarization task is framed as seq2seq modeling.Existing seq2seq methods perform better on short sequences;however,for long sequences,the performance degrades due to high computation and hence a two-phase self-normalized deep neural document summarization model consisting of improvised extractive cosine normalization and seq2seq abstractive phases has been proposed in this paper.The novelty is to parallelize the sequence computation training by incorporating feed-forward,the self-normalized neural network in the Extractive phase using Intra Cosine Attention Similarity(Ext-ICAS)with sentence dependency position.Also,it does not require any normalization technique explicitly.Our proposed abstractive Bidirectional Long Short Term Memory(Bi-LSTM)encoder sequence model performs better than the Bidirectional Gated Recurrent Unit(Bi-GRU)encoder with minimum training loss and with fast convergence.The proposed model was evaluated on the Cable News Network(CNN)/Daily Mail dataset and an average rouge score of 0.435 was achieved also computational training in the extractive phase was reduced by 59%with an average number of similarity computations.
文摘In the case of Z＋^d（d ≥ 2）-the positive d-dimensional lattice points with partial ordering ≤, {Xk,k∈ Z＋^d} i.i.d, random variables with mean 0, Sn =∑k≤nXk and Vn^2 = ∑j≤nXj^2, the precise asymptotics for ∑n1/｜n｜（log｜n｜dP（｜Sn/Vn｜≥ε√log log｜n｜） and ∑n（logn｜）b/｜n｜（log｜n｜）^d-1P（｜Sn/Vn｜≥ε√log n）,as ε↓0,is established.
文摘The past two decades have witnessed the active development of a rich probability theory of Studentized statistics or self-normalized processes, typified by Student’s t-statistic as introduced by W. S. Gosset more than a century ago, and their applications to statistical problems in high dimensions, including feature selection and ranking, large-scale multiple testing and sparse, high dimensional signal detection. Many of these applications rely on the robustness property of Studentization/self-normalization against heavy-tailed sampling distributions. This paper gives an overview of the salient progress of self-normalized limit theory, from Student’s t-statistic to more general Studentized nonlinear statistics. Prototypical examples include Studentized one- and two-sample U-statistics. Furthermore, we go beyond independence and glimpse some very recent advances in self-normalized moderate deviations under dependence.
基金National Natural Science Foundation of China(1047112610671176).
文摘In this article, the unit root test for AR（p） model with GARCH errors is considered. The Dickey-Fuller test statistics are rewritten in the form of self-normalized sums, and the asymptotic distribution of the test statistics is derived under the weak conditions.
文摘In this case study, we would like to illustrate the utility of characteristic functions, using an example of a sample statistic defined for samples from Cauchy distribution. The derivation of the corresponding asymptotic probability density function is based on [1], elaborating and expanding the individual steps of their presentation, and including a small extension;our reason for such a plagiarism is to make the technique, its mathematical tools and ingenious arguments available to the widest possible audience.
基金National Natural Science Foundation of China(1067117610771192).
文摘Let variables in the {X, Xn, n ≥ 1} be a sequence of strictly stationary φ-mixing positive random domain of attraction of the normal law. Under some suitable conditions the principle for self-normalized products of partial sums is obtained.
基金Yibin University Pre-research Project,Research on the coupling and coordinated development ofmanufacturing and logistics industry under the background of intelligentmanufacturing,(2022YY001)Sichuan ProvincialDepartment of EducationWater Transport EconomicResearch Center,Research on the Development Path and Countermeasures of the Advanced Manufacturing Industry in the Sanjiang New District of Yibin under a“dual circulation”development pattern(SYJJ2020A06).
文摘Recently,fuzzy multi-sets have come to the forefront of scientists’interest and have been used in algebraic structures such asmulti-groups,multirings,anti-fuzzy multigroup and(α,γ)-anti-fuzzy subgroups.In this paper,we first summarize the knowledge about the algebraic structure of fuzzy multi-sets such as(α,γ)-anti-multi-fuzzy subgroups.In a way,the notion of anti-fuzzy multigroup is an application of anti-fuzzy multi sets to the theory of group.The concept of anti-fuzzy multigroup is a complement of an algebraic structure of a fuzzy multi set that generalizes both the theories of classical group and fuzzy group.The aim of this paper is to highlight the connection between fuzzy multi-sets and algebraic structures from an anti-fuzzification point of view.Therefore,in this paper,we define(α,γ)-antimulti-fuzzy subgroups,(α,γ)-anti-multi-fuzzy normal subgroups,(α,γ)-antimulti-fuzzy homomorphism on(α,γ)-anti-multi-fuzzy subgroups and these been explicated some algebraic structures.Then,we introduce the concept(α,γ)-anti-multi-fuzzy subgroups and(α,γ)-anti-multi-fuzzy normal subgroups and of their properties.This new concept of homomorphism as a bridge among set theory,fuzzy set theory,anti-fuzzy multi sets theory and group theory and also shows the effect of anti-fuzzy multi sets on a group structure.Certain results that discuss the(α,γ)cuts of anti-fuzzy multigroup are explored.
