A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips.

08:00 EDT 2nd July 2019 | BioPortfolio

Summary of "A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips."

Multimodal emotion understanding enables AI systems to interpret human emotions. With accelerated video surge, emotion understanding remains challenging due to inherent data ambiguity and diversity of video content. Although deep learning has made a considerable progress in big data feature learning, they are viewed as deterministic models used in a "black-box" manner which does not have capabilities to represent inherent ambiguities with data. Since the possibility theory of fuzzy logic focuses on knowledge representation and reasoning under uncertainty, we intend to incorporate the concepts of fuzzy logic into deep learning framework. This paper presents a novel convolutional neuro-fuzzy network, which is an integration of convolutional neural networks in fuzzy logic domain to extract high-level emotion features from text, audio, and visual modalities. The feature sets extracted by fuzzy convolutional layers are compared with those of convolutional neural networks at the same level using t-distributed Stochastic Neighbor Embedding. This paper demonstrates a multimodal emotion understanding framework with an adaptive neural fuzzy inference system that can generate new rules to classify emotions. For emotion understanding of movie clips, we concatenate audio, visual, and text features extracted using the proposed convolutional neuro-fuzzy network to train adaptive neural fuzzy inference system. In this paper, we go one step further to explain how deep learning arrives at a conclusion that can guide us to an interpretable AI. To identify which visual/text/audio aspects are important for emotion understanding, we use direct linear non-Gaussian additive model to explain the relevance in terms of causal relationships between features of deep hidden layers. The critical features extracted are input to the proposed multimodal framework to achieve higher accuracy.


Journal Details

This article was published in the following journal.

Name: Neural networks : the official journal of the International Neural Network Society
ISSN: 1879-2782
Pages: 208-219


DeepDyve research library

PubMed Articles [11409 Associated PubMed Articles listed on BioPortfolio]

Automatic Detection and Classification of Lung Nodules in CT Image Using Optimized Neuro Fuzzy Classifier with Cuckoo Search Algorithm.

The Lung nodules are very important to indicate the lung cancer, and its early detection enables timely treatment and increases the survival rate of patient. Even though lots of works are done in this...

Oxidative stability of virgin olive oil: Evaluation and prediction by adaptive neuro-fuzzy inference system (ANFIS).

Adaptive neuro-fuzzy inference system (ANFIS) was employed for the prediction of oxidative stability of virgin olive oil (VOO) during storage as a function of time, storage temperature, total polyphen...

Application of adaptive-network-based fuzzy inference systems to the parameter optimization of a biochemical rule-based model.

In this study, the binding of allergens to antibody-receptor complexes was investigated. This process is important in understanding the allergic response. A BioNetGen model that simulates this process...

Fuzzy Petri nets for modelling of uncertain biological systems.

The modelling of biological systems is accompanied with epistemic uncertainties that range from structural uncertainty to parametric uncertainty due to such limitations as insufficient understanding o...

Large social cognitive impairments characterize homicide offenders with schizophrenia.

Schizophrenia is associated with an increased violence risk, particularly homicide. One possible, but scarcely explored, contributor to the increased violence risk is social cognitive impairment. Indi...

Clinical Trials [3492 Associated Clinical Trials listed on BioPortfolio]

Diagnostic Efficacy of CNN in Differentiation of Visual Field

Glaucoma is currently the leading cause of irreversible blindness in the world. The multi-center study is designed to evaluate the efficacy of the convolutional neural network based algori...

Neuro 1 vs. Neuro 2 Sound Processors

This study evaluates the benefit of Neuro 1 sound processor upgrade in speech perfomance in adults. Half of participants will be tested with Neuro 1 first and Neuro 2, while the other half...

Emotion Regulation in Binge Eating and Purging Among Adolescents

The study will examine the neural and behavioral correlates of emotion regulation in adolescents engaging in binge eating and/or purging and healthy adolescents. Furthermore, it will look ...

The Fear F8ctor Study - Does Fear Induce a Blood Curdling State?

For centuries the term "blood curling" has been used to describe extreme frightening situations. However, the origin of this ancient theory has never been studied and it remains unknown if...

Inform and Reduce Pre-surgical Anxiety of the Child and His Parents: Interest of an Explanatory Movie.

The purpose of the study is to measure the interest of a movie explaining the path of the children in surgery, in order to reduce the anxiety of the children and his parents. This study i...

Medical and Biotech [MESH] Definitions

Approximate, quantitative reasoning that is concerned with the linguistic ambiguity which exists in natural or synthetic language. At its core are variables such as good, bad, and young as well as modifiers such as more, less, and very. These ordinary terms represent fuzzy sets in a particular problem. Fuzzy logic plays a key role in many medical expert systems.

The act or fact of grasping the meaning, nature, or importance of; understanding. (American Heritage Dictionary, 4th ed) Includes understanding by a patient or research subject of information disclosed orally or in writing.

The process of helping patients to effectively and efficiently use the health care system when faced with one or more of these challenges: (1) choosing, understanding, and using health coverage or applying for assistance when uninsured; (2) choosing, using, and understanding different types of health providers and services; (3) making treatment decisions; and (4) managing care received by multiple providers.

The use of combination of imaging techniques or platforms (e.g., MRI SCAN and PET SCAN) encompassing aspects of anatomical, functional, or molecular imaging methods.

The circulation of the BLOOD through the MICROVASCULAR NETWORK.

Quick Search


DeepDyve research library

Relevant Topic

Neurology - Central Nervous System (CNS)
Alzheimer's Disease Anesthesia Anxiety Disorders Autism Bipolar Disorders Dementia Epilepsy Multiple Sclerosis (MS) Neurology Pain Parkinson's Disease Sleep Disorders Neurology is the branch of me...

Searches Linking to this Article