A multimodal parallel architecture: A cognitive framework for multimodal interactions.

08:00 EDT 19th October 2015 | BioPortfolio

Summary of "A multimodal parallel architecture: A cognitive framework for multimodal interactions."

Human communication is naturally multimodal, and substantial focus has examined the semantic correspondences in speech-gesture and text-image relationships. However, visual narratives, like those in comics, provide an interesting challenge to multimodal communication because the words and/or images can guide the overall meaning, and both modalities can appear in complicated "grammatical" sequences: sentences use a syntactic structure and sequential images use a narrative structure. These dual structures create complexity beyond those typically addressed by theories of multimodality where only a single form uses combinatorial structure, and also poses challenges for models of the linguistic system that focus on single modalities. This paper outlines a broad theoretical framework for multimodal interactions by expanding on Jackendoff's (2002) parallel architecture for language. Multimodal interactions are characterized in terms of their component cognitive structures: whether a particular modality (verbal, bodily, visual) is present, whether it uses a grammatical structure (syntax, narrative), and whether it "dominates" the semantics of the overall expression. Altogether, this approach integrates multimodal interactions into an existing framework of language and cognition, and characterizes interactions between varying complexity in the verbal, bodily, and graphic domains. The resulting theoretical model presents an expanded consideration of the boundaries of the "linguistic" system and its involvement in multimodal interactions, with a framework that can benefit research on corpus analyses, experimentation, and the educational benefits of multimodality.


Journal Details

This article was published in the following journal.

Name: Cognition
ISSN: 1873-7838
Pages: 304-323


DeepDyve research library

PubMed Articles [14115 Associated PubMed Articles listed on BioPortfolio]

A New Multi-Atlas Registration Framework for Multimodal Pathological Images Using Conventional Monomodal Normal Atlases.

Using multi-atlas registration (MAR), information carried by atlases can be transferred onto a new input image for the tasks of region of interest (ROI) segmentation, anatomical landmark detection, an...

Multimodal Postoperative Pain Control Is Effective and Reduces Opioid Use After Laparoscopic Roux-en-Y Gastric Bypass.

Opioids have been the mainstay for postoperative pain relief for many decades. Recently, opioid-related adverse events and death have been linked to postoperative dependency. Multimodal approaches to ...

A qualitative exploration of post-acute stroke participants' experiences of a multimodal intervention incorporating horseback riding.

Multimodal rehabilitation interventions delivered in late phase of stroke recovery involve physical (motor and sensory), social, and cognitively challenging activities. Horseback riding can be incorpo...

A multimodal biomarker for concussion identification, prognosis and management.

Prompt, accurate, objective assessment of concussion is crucial, particularly for children/adolescents and young adults. While there is currently no gold standard for the diagnosis of concussion, the ...

Multimodal Analgesia Decreases Opioid Use in Critically Ill Trauma Patients.

Opioids are the mainstay of pain management in critically ill trauma patients. However, the risks of opioid use mandate a different approach. Multimodal analgesia employs a combination of opioid and n...

Clinical Trials [7346 Associated Clinical Trials listed on BioPortfolio]

The COMUNICARE Study: the Effectiveness of a Multimodal Intervention Based on Person-centered Communication

Pragmatic cluster-randomized clinical trial. An open, multicentric, parallel-group trial with a control group, and with a follow-up period of 1 year.

Neural and Cognitive Mechanisms of Depression and Anxiety: a Multimodal MRI Study

To explore the neural and cognitive mechanisms of depression (including unipolar depression and Bipolar disorder) and anxiety using multimodal MRI techniques.

Multimodal Resonance Imaging for Outcome Prediction on Coma Patients

Stroke, traumatic head injury, subarachnoid hemorrhage and cerebral anoxia are main causes of a coma condition implying severe brain damage and thus, poor prognosis. Clinicians are often i...

FRench Acute Cerebral Multimodal Imaging to Select Patient for MEchanical Thrombectomy

FRench Acute cerebral multimodal imaging to select patient for MEchanical thrombectomy is a prospective multi-center study to determine if multimodal imaging could identify patients who ma...

Multimodal Analgesia After Shoulder Arthroscopy

Pain management after shoulder surgery remains to be a major challenge. Reliance on a single agent (most commonly opioids) results in increased incidence of side effects. Multimodal pain m...

Medical and Biotech [MESH] Definitions

The use of combination of imaging techniques or platforms (e.g., MRI SCAN and PET SCAN) encompassing aspects of anatomical, functional, or molecular imaging methods.

A disease marked by repeated episodes of increased bone resorption followed by excessive attempts at repair, resulting in weakened, deformed bones of increased mass. The resultant architecture of the bone assumes a mosaic pattern in which the fibers take on a haphazard pattern instead of the normal parallel symmetry.

The art and science of designing buildings and structures. More generally, it is the design of the total built environment, including town planning, urban design, and landscape architecture.

Works consisting of drawings of architecture and architectural projects, whether the project was executed or not. (Art & Architecture Thesaurus, 1990, v.2)

Works on the art and science of designing buildings and structures. More generally, works on the design of the total built environment, including town planning, urban design, and landscape architecture.

Quick Search


DeepDyve research library

Searches Linking to this Article