Toward Training Recurrent Neural Networks for Lifelong Learning.

07:00 EST 8th November 2019 | BioPortfolio

Summary of "Toward Training Recurrent Neural Networks for Lifelong Learning."

Catastrophic forgetting and capacity saturation are the central challenges of any parametric lifelong learning system. In this work, we study these challenges in the context of sequential supervised learning with an emphasis on recurrent neural networks. To evaluate the models in the lifelong learning setting, we propose a curriculum-based, simple, and intuitive benchmark where the models are trained on tasks with increasing levels of difficulty. To measure the impact of catastrophic forgetting, the model is tested on all the previous tasks as it completes any task. As a step toward developing true lifelong learning systems, we unify gradient episodic memory (a catastrophic forgetting alleviation approach) and Net2Net (a capacity expansion approach). Both models are proposed in the context of feedforward networks, and we evaluate the feasibility of using them for recurrent networks. Evaluation on the proposed benchmark shows that the unified model is more suitable than the constituent models for lifelong learning setting.


Journal Details

This article was published in the following journal.

Name: Neural computation
ISSN: 1530-888X
Pages: 1-34


DeepDyve research library

PubMed Articles [12632 Associated PubMed Articles listed on BioPortfolio]

GRUU-Net: Integrated convolutional and gated recurrent neural network for cell segmentation.

Cell segmentation in microscopy images is a common and challenging task. In recent years, deep neural networks achieved remarkable improvements in the field of computer vision. The dominant paradigm i...

Recurrent Neural Networks With External Addressable Long-Term and Working Memory for Learning Long-Term Dependences.

Learning long-term dependences (LTDs) with recurrent neural networks (RNNs) is challenging due to their limited internal memories. In this paper, we propose a new external memory architecture for RNNs...

Multi-source sequential knowledge regression by using transfer RNN units.

Transfer learning has achieved a lot of success in deep neural networks to reuse useful knowledge from source domains. However, most of the existing transfer learning strategies on neural networks are...

Putting a bug in ML: The moth olfactory network learns to read MNIST.

We seek to (i) characterize the learning architectures exploited in biological neural networks for training on very few samples, and (ii) port these algorithmic structures to a machine learning contex...

A Taxonomy for Neural Memory Networks.

An increasing number of neural memory networks have been developed, leading to the need for a systematic approach to analyze and compare their underlying memory structures. Thus, in this paper, we fir...

Clinical Trials [10116 Associated Clinical Trials listed on BioPortfolio]

Consolidation of Motor Learning of Writing Skills and Its Related Brain Activity Changes in Parkinson&Apos;s Disease

The basal ganglia play an important role in motor learning, especially during the consolidation phase of motor learning. This raises the question whether it is possible to sustain learning...

Early Discrimination of Periprosthetic Hip Infections Using Neural Networks (SEPTIC-ANNR)

The study is about the role of cellular neural networks-genetic algorithm in the diagnosis of periprosthetic hip infections. A retrospective case series of septic and aseptic loosening of ...

Renal Cancer Detection Using Convolutional Neural Networks

We aim to experiment and implement various deep learning architectures in order to achieve human-level accuracy in Computer-aided diagnosis (CAD) systems. In particular, we are interested ...

Functional MRI Study in Children With a Developmental Coordination Disorder (DCD) and/or Developmental Dyslexia (DD)

All the studies underlined the high frequency of co-morbid associations in specific learning disorders. Understanding the reasons for these associations could enable us to determine the ce...

Positron Emission Tomography (PET) Images Using Deep Neural Networks

PET images are based on detecting two annihilation 511 KeV photons that are produced by positron emitting isotopes. The longer the acquisition time, the more photons are detected and proce...

Medical and Biotech [MESH] Definitions

A computer architecture, implementable in either hardware or software, modeled after biological neural networks. Like the biological system in which the processing capability is a result of the interconnection strengths between arrays of nonlinear processing nodes, computerized neural networks, often called perceptrons or multilayer connectionist models, consist of neuron-like units. A homogeneous group of units makes up a layer. These networks are good at pattern recognition. They are adaptive, performing tasks by example, and thus are better for decision-making than are linear learning machines or cluster analysis. They do not require explicit programming.

Diagnosed when there are specific deficits in an individual’s ability to perceive or process information efficiently and accurately. This disorder first manifests during the years of formal schooling and is characterized by persistent and impairing difficulties with learning foundational academic skills in reading, writing, and/or math. The individual’s performance of the affected academic skills is well below average for age, or acceptable performance levels are achieved only with extraordinary effort. Specific learning disorder may occur in individuals identified as intellectually gifted and manifest only when the learning demands or assessment procedures (e.g., timed tests) pose barriers that cannot be overcome by their innate intelligence and compensatory strategies. For all individuals, specific learning disorder can produce lifelong impairments in activities dependent on the skills, including occupational performance. (from DSM-V)

Process in which individuals take the initiative, in diagnosing their learning needs, formulating learning goals, identifying resources for learning, choosing and implementing learning strategies and evaluating learning outcomes (Knowles, 1975)

A MACHINE LEARNING paradigm used to make predictions about future instances based on a given set of labeled paired input-output training (sample) data.

A MACHINE LEARNING paradigm used to make predictions about future instances based on a given set of unlabeled paired input-output training (sample) data.

Quick Search

DeepDyve research library

Searches Linking to this Article