Session 11: Focus Session - Memory Technology Modelling Challenges for Neuromorphic Computing

Tuesday, December 5
Grand Ballroom A
Co-Chairs: Denis Rideau, STMicroelectronics
Meng-Fan Chang, National Tsing Hua University

9:05 AM
11.1 Stochastic Synapses as Resource for Efficient Deep Learning Machines (Invited), E. Neftci, University of California, Irvine

Synaptic unreliability was shown to be a robust and sufficient mechanism for inducing the stochasticity in biological and artificial neural network models. Previous work demonstrated multiplicative noise (also called dropconnect) as a powerful regularizer during training. Here, we show that always-on stochasticity at networks connections is a sufficient resource for deep learning machines when combined with simple threshold non-linearities. Furthermore, the resulting activity function exhibits a self-normalizing property that reflects a recently proposed “Weight Normalization” technique, itself fulfilling many of the features of batch normalization in an online fashion. Normalization of activities during training can speed up convergence by preventing so-called internal covariate shift caused by changes in the distribution of inputs as the parameters of the previous layers are trained. Collectively, our findings can improve performance of deep learning machines with fixed point representations and argue in favor of stochastic nanodevices as primitives for efficient deep learning machines with online and embedded learning capabilities.

9:30 AM
11.2 Attractor networks and associative memories with STDP learning in RRAM synapses (Invited), V. Milo, D. Ielmini and E. Chicca*, Politecnico di Milano and IU.NET, *Bielefeld University

Attractor networks can realistically describe neurophysiological processes while providing useful computational modules for pattern recognition, signal restoration, and feature extraction. To implement attractor networks in small-area integrated circuits, the development of a hybrid technology including CMOS transistors and resistive switching memory (RRAM) is essential. This work presents a summary of recent results toward implementing RRAM-based attractor networks. Based on realistic models of HfO2 RRAM devices, we design and simulate recurrent networks showing the capability to train, recall and sustain attractors. The results support the feasibility of RRAM-based bio-realistic attractor networks.

9:55 AM
11.3 Energy use constrains brain information processing (Invited), M. Conrad, E. Engl* and R. B. Jolivet, University of Geneva, *Lodestar Insights

The brain is an energetically expensive organ to build and operate, and a large body of literature links the evolutionary development of many of the human brain’s components to the need to save energy. We, and others, have shown experimentally and through computational modelling that synapses in the brain do not maximise information transfer, but instead transfer information in an energetically efficient manner. Strikingly, this optimum implies a high failure rate in the transmission of individual information-carrying signals (action potentials or spikes). This design principle may be important when considering trade-offs between energy use and information transfer in man-made devices.

10:20 AM
11.4 Understanding the Trade-offs of Device, Circuit and Application in ReRAM-based Neuromorphic Computing Systems (Invited), B. Yan, C. Liu*, X. Liu**, Y. Chen and H. Li, Duke University, *Clarkson University, **AMD, Sunnyvale

ReRAM technology demonstrates great potential in the development of neuromorphic computing systems. This paper discusses the importance of the comprehensive understanding across the device, circuit, and application levels in ReRAM-based neuromorphic system, through the discussion of three major problems—weight mapping, reliability, and system integration.

10:45 AM Coffee Break

11:10 AM
11.5 Device and circuit optimization of RRAM for Neuromorphic computing (Invited), H. Wu, P. Yao, B. Gao, W. Wu, Q. Zhang, W. Zhang, N. Deng, D. Wu, H.-S. P. Wong*, S. Yu**, and H. Qian, Tsinghua University, *Stanford University, **Arizona State University

RRAM is a promising electrical synaptic device for efficient neuromorphic computing. The device structure and materials stack were optimized to achieve reliable bidirectional analog switching behavior. A human face recognition task was demonstrated on a 1k-bit 1T1R RRAM array using an online training perceptron network. A binarized-hidden-layer (BHL) circuit architecture is proposed to minimize the needs of A/D and D/A converters required between RRAM crossbars. Several non-ideal RRAM characteristics were carefully evaluated for handwritten digits’ recognition task with proposed BHL architecture and modified neural network algorithm.

11:35 AM
11.6 Challenges and Opportunities toward Online Training Acceleration using RRAM-based Hardware Neural Network (Invited), C.-C. Chang, J.-C. Liu, Y.-L. Shen, T. Chou, P.-C. Chen, I-T. Wang, C.-C. Su, M.-H. Wu, B. Hudec, C.-C. Chang, C.-M. Tsai, T.-S. Chang, H.-S. P. Wong*, and T.-H. Hou, National Chiao Tung University, *Stanford University

This paper highlights the feasible routes of using resistive memory (RRAM) for accelerating online training of deep neural networks (DNNs). A high degree of asymmetric nonlinearity in analog RRAMs could be tolerated when weight update algorithms are optimized with reduced training noise. Hybrid- weight Net (HW-Net), a modified multilayer perceptron (MLP) algorithm that utilizes hybrid internal analog and external binary weights is also proposed. Highly accurate online training could be realized using simple binary RRAMs that have already been widely developed as digital memory.

12:00 PM
11.7 Multiscale modeling of neuromorphic computing: from materials to device operations (Invited), L. Larcher, A. Padovani* and V. Di Lecce*, University of ModenaReggio Emilia, *MDLSoft Inc.

In this paper, a multiscale modeling platform for neuromorphic computing devices connecting the atomic material properties to the electrical device performances is presented. The main ingredients of the modeling platform are discussed in view of the different technologies (e.g. RRAM, PCM, FTJ) proposed for 3D integrated neuromorphic computing.