Track topics on Twitter Track topics that are important to you
Rectified linear units (ReLUs) have become the main model for the neural units in current deep learning systems. This choice was originally suggested as a way to compensate for the so-called vanishing gradient problem which can undercut stochastic gradient descent learning in networks composed of multiple layers. Here we provide analytical results on the effects of ReLUs on the capacity and on the geometrical landscape of the solution space in two-layer neural networks with either binary or real-valued weights. We study the problem of storing an extensive number of random patterns and find that, quite unexpectedly, the capacity of the network remains finite as the number of neurons in the hidden layer increases, at odds with the case of threshold units in which the capacity diverges. Possibly more important, a large deviation approach allows us to find that the geometrical landscape of the solution space has a peculiar structure: While the majority of solutions are close in distance but still isolated, there exist rare regions of solutions which are much more dense than the similar ones in the case of threshold units. These solutions are robust to perturbations of the weights and can tolerate large perturbations of the inputs. The analytical results are corroborated by numerical findings.
This article was published in the following journal.
Name: Physical review letters
Recently multilayer networks are introduced to model real systems. In these models the individuals make connection in multiple layers. Transportation networks, biological systems and social networks a...
This review connects several lines of research to argue that hyperbolic geometry should be broadly applicable to neural circuits as well as other biological circuits. The reason for this is that netwo...
We propose a multilayer geometry to characterize thin-film samples in reflection terahertz time domain spectroscopy. Theory indicates that this geometry has higher sensitivity compared to ordinary tra...
Sex estimation is an essential step in the process of the identification of the skeletal remains in forensic anthropology since it reduces the number of possible matches by half. In this study, sex es...
Neural associative memories (NAM) are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Gripon and Be...
The study is about the role of cellular neural networks-genetic algorithm in the diagnosis of periprosthetic hip infections. A retrospective case series of septic and aseptic loosening of ...
The aim of the study is to demonstrate that our semantic knowledge (elements of our long-term memory and the process we use them) respond to a graphic organisation and gather together foll...
Treatment of lymphedema is based on a combined program called complex decongestive therapy. The components of the complex decongestive therapy are skin care, kinesiotherapy, manual lymphat...
Teriparatide is the only bone formation therapy that has been approved for the treatment of postmenopausal osteoporosis in Canada. Osteoporosis is currently diagnosed using a bone mineral...
Project:EVO is a video-game based intervention that targets neural networks associated with cognitive control. The same networks have been implicated in poor treatment response in middle a...
A computer architecture, implementable in either hardware or software, modeled after biological neural networks. Like the biological system in which the processing capability is a result of the interconnection strengths between arrays of nonlinear processing nodes, computerized neural networks, often called perceptrons or multilayer connectionist models, consist of neuron-like units. A homogeneous group of units makes up a layer. These networks are good at pattern recognition. They are adaptive, performing tasks by example, and thus are better for decision-making than are linear learning machines or cluster analysis. They do not require explicit programming.
Sterile solutions, essentially free from foreign particles and suitably compounded and dispensed, for instillation into the eye. It does not include solutions for cleaning eyeglasses or CONTACT LENS SOLUTIONS. (From Dorland, 27th ed)
Cells with the capacity to take up and decarboxylate the amine precursors DIHYDROXYPHENYLALANINE or 5-HYDROXYTRYPTOPHAN. This is a property of endocrine cells of neural and non-neural origin. APUDOMA is a general term collectively applied to tumors associated with APUD cells.
An early embryonic developmental process of CHORDATES that is characterized by morphogenic movements of ECTODERM resulting in the formation of the NEURAL PLATE; the NEURAL CREST; and the NEURAL TUBE. Improper closure of the NEURAL GROOVE results in congenital NEURAL TUBE DEFECTS.
Capacity that enables an individual to cope with and/or recover from the impact of a neural injury or a psychotic episode.