基于概念器的深度神经网络模型

Conceptor-based deep neural networks

近年来,深度神经网络,亦被称为深度学习,在机器学习方法主导的各个领域都取得了重大的突破.虽然经过训练的深度神经网络具有卓越的性能,但是整个训练过程却十分耗时,即使借助高性能计算设备,也需要数日甚至数周的训练时间.概念器作为回音状态网络的延续和发展,可以理解为描述神经动态活动模式的过滤器,是一个强大的时序数据处理工具.为了解决上述问题,基于对原始概念器模型的改进,本文在深度神经网络的非迭代方法和迁移学习两个方面分别做了一些工作.具体来说,(1)提出了针对非时序数据的概念器分类器,并在此基础上提出了一种非迭代方法前馈卷积概念器神经网络,通过在MNIST变集数据集上的实验测试了前馈卷积概念器神经网络的分类性能,不仅达到了同类方法的最高水平,而且极大地降低了训练时间;(2)提出了一种基于概念器的快速概念器分类器,在数据集Caltech-101和Caltech-256上,测试了快速概念器分类器结合预训练且不再微调的深度神经网络的表现,不仅在性能上超越了同类方法的最高水平,而且训练时间平均减少到原有的1/60.

In recent years, deep neural networks, also known as deep learning, have achieved several breakthroughs in different fields that were previously dominated by machine learning. Even when using highperformance computing devices, it takes days or weeks to train a deep neural network. Conceptor, as an extension of echo state networks, can be understood as certain neural filters that characterize dynamical neural activation patterns. In this study, based on some improvements to the original conceptor model, we have conducted several studies from the perspectives of non-iterative methods and transfer learning to address the issues mentioned above,which can be summarized as follows：（1） A conceptor-based classifier for non-temporal data and a non-iterative approach feedforward convolutional conceptor neural network are proposed. This classifier achieves classifying accuracy comparable to that of the state-of-the-art methods while requiring significantly less training time. Through experiments on MNIST variation datasets, we evaluate the classifying quality of the feedforward convolutional conceptor neural network.（2） A classifier called fast conceptor classifier is proposed based on conceptors and it achieves state-of-the-art results with the training time reduced by a factor of 60 on average. Its evaluations with pre-trained rather than fine-tuned neural networks have been investigated on Caltech-101 and Caltech-256 datasets.