Properties of the Geometry of Solutions and Capacity of Multilayer Neural Networks with Rectified Linear Unit Activations.

08:00 EDT 25th October 2019 | BioPortfolio

Summary of "Properties of the Geometry of Solutions and Capacity of Multilayer Neural Networks with Rectified Linear Unit Activations."

Rectified linear units (ReLUs) have become the main model for the neural units in current deep learning systems. This choice was originally suggested as a way to compensate for the so-called vanishing gradient problem which can undercut stochastic gradient descent learning in networks composed of multiple layers. Here we provide analytical results on the effects of ReLUs on the capacity and on the geometrical landscape of the solution space in two-layer neural networks with either binary or real-valued weights. We study the problem of storing an extensive number of random patterns and find that, quite unexpectedly, the capacity of the network remains finite as the number of neurons in the hidden layer increases, at odds with the case of threshold units in which the capacity diverges. Possibly more important, a large deviation approach allows us to find that the geometrical landscape of the solution space has a peculiar structure: While the majority of solutions are close in distance but still isolated, there exist rare regions of solutions which are much more dense than the similar ones in the case of threshold units. These solutions are robust to perturbations of the weights and can tolerate large perturbations of the inputs. The analytical results are corroborated by numerical findings.


Journal Details

This article was published in the following journal.

Name: Physical review letters
ISSN: 1079-7114
Pages: 170602


DeepDyve research library

PubMed Articles [17259 Associated PubMed Articles listed on BioPortfolio]

Application of hyperbolic geometry in link prediction of multiplex networks.

Recently multilayer networks are introduced to model real systems. In these models the individuals make connection in multiple layers. Transportation networks, biological systems and social networks a...

An argument for hyperbolic geometry in neural circuits.

This review connects several lines of research to argue that hyperbolic geometry should be broadly applicable to neural circuits as well as other biological circuits. The reason for this is that netwo...

Utilizing multilayer structures to enhance terahertz characterization of thin films ranging from aqueous solutions to histology slides.

We propose a multilayer geometry to characterize thin-film samples in reflection terahertz time domain spectroscopy. Theory indicates that this geometry has higher sensitivity compared to ordinary tra...

Sex estimation from sacrum and coccyx with discriminant analyses and neural networks in an equally distributed population by age and sex.

Sex estimation is an essential step in the process of the identification of the skeletal remains in forensic anthropology since it reduces the number of possible matches by half. In this study, sex es...

Iterative Retrieval and Block Coding in Autoassociative and Heteroassociative Memory.

Neural associative memories (NAM) are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Gripon and Be...

Clinical Trials [4476 Associated Clinical Trials listed on BioPortfolio]

Early Discrimination of Periprosthetic Hip Infections Using Neural Networks (SEPTIC-ANNR)

The study is about the role of cellular neural networks-genetic algorithm in the diagnosis of periprosthetic hip infections. A retrospective case series of septic and aseptic loosening of ...

Contributions From the Analysis of Graphs for Identification of Neural Cliques

The aim of the study is to demonstrate that our semantic knowledge (elements of our long-term memory and the process we use them) respond to a graphic organisation and gather together foll...

Effectiveness of Two Compression Methods in the Treatment of Upper Limb Lymphedema

Treatment of lymphedema is based on a combined program called complex decongestive therapy. The components of the complex decongestive therapy are skin care, kinesiotherapy, manual lymphat...

Effects of Teriparatide (PTH) on Bone in Men and Women With Osteoporosis

Teriparatide is the only bone formation therapy that has been approved for the treatment of postmenopausal osteoporosis in Canada. Osteoporosis is currently diagnosed using a bone mineral...

Computerized Intervention Targeting Cognitive Control Deficits in Depressed Adults

Project:EVO is a video-game based intervention that targets neural networks associated with cognitive control. The same networks have been implicated in poor treatment response in middle a...

Medical and Biotech [MESH] Definitions

A computer architecture, implementable in either hardware or software, modeled after biological neural networks. Like the biological system in which the processing capability is a result of the interconnection strengths between arrays of nonlinear processing nodes, computerized neural networks, often called perceptrons or multilayer connectionist models, consist of neuron-like units. A homogeneous group of units makes up a layer. These networks are good at pattern recognition. They are adaptive, performing tasks by example, and thus are better for decision-making than are linear learning machines or cluster analysis. They do not require explicit programming.

Sterile solutions, essentially free from foreign particles and suitably compounded and dispensed, for instillation into the eye. It does not include solutions for cleaning eyeglasses or CONTACT LENS SOLUTIONS. (From Dorland, 27th ed)

Cells with the capacity to take up and decarboxylate the amine precursors DIHYDROXYPHENYLALANINE or 5-HYDROXYTRYPTOPHAN. This is a property of endocrine cells of neural and non-neural origin. APUDOMA is a general term collectively applied to tumors associated with APUD cells.

An early embryonic developmental process of CHORDATES that is characterized by morphogenic movements of ECTODERM resulting in the formation of the NEURAL PLATE; the NEURAL CREST; and the NEURAL TUBE. Improper closure of the NEURAL GROOVE results in congenital NEURAL TUBE DEFECTS.

Capacity that enables an individual to cope with and/or recover from the impact of a neural injury or a psychotic episode.

Quick Search

DeepDyve research library

Searches Linking to this Article