Lau, E. F., Gramfort, A., Hämäläinen, M. S., & Kuperberg, G. R. (2013). Automatic semantic facilitation in anterior temporal cortex revealed through multimodal neuroimaging. J Neurosci , 33 (43), 17174-81. Full TextAbstract
A core property of human semantic processing is the rapid, facilitatory influence of prior input on extracting the meaning of what comes next, even under conditions of minimal awareness. Previous work has shown a number of neurophysiological indices of this facilitation, but the mapping between time course and localization-critical for separating automatic semantic facilitation from other mechanisms-has thus far been unclear. In the current study, we used a multimodal imaging approach to isolate early, bottom-up effects of context on semantic memory, acquiring a combination of electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) measurements in the same individuals with a masked semantic priming paradigm. Across techniques, the results provide a strikingly convergent picture of early automatic semantic facilitation. Event-related potentials demonstrated early sensitivity to semantic association between 300 and 500 ms; MEG localized the differential neural response within this time window to the left anterior temporal cortex, and fMRI localized the effect more precisely to the left anterior superior temporal gyrus, a region previously implicated in semantic associative processing. However, fMRI diverged from early EEG/MEG measures in revealing semantic enhancement effects within frontal and parietal regions, perhaps reflecting downstream attempts to consciously access the semantic features of the masked prime. Together, these results provide strong evidence that automatic associative semantic facilitation is realized as reduced activity within the left anterior superior temporal cortex between 300 and 500 ms after a word is presented, and emphasize the importance of multimodal neuroimaging approaches in distinguishing the contributions of multiple regions to semantic processing.
Temereanca, S., Hämäläinen, M. S., Kuperberg, G. R., Stufflebeam, S. M., Halgren, E., & Brown, E. N. (2012). Eye movements modulate the spatiotemporal dynamics of word processing. J Neurosci , 32 (13), 4482-94. Full TextAbstract
Active reading requires coordination between frequent eye movements (saccades) and short fixations in text. Yet, the impact of saccades on word processing remains unknown, as neuroimaging studies typically employ constant eye fixation. Here we investigate eye-movement effects on word recognition processes in healthy human subjects using anatomically constrained magnetoencephalography, psychophysical measurements, and saccade detection in real time. Word recognition was slower and brain responses were reduced to words presented early versus late after saccades, suggesting an overall transient impairment of word processing after eye movements. Response reductions occurred early in visual cortices and later in language regions, where they colocalized with repetition priming effects. Qualitatively similar effects occurred when words appeared early versus late after background movement that mimicked saccades, suggesting that retinal motion contributes to postsaccadic inhibition. Further, differences in postsaccadic and background-movement effects suggest that central mechanisms also contribute to postsaccadic modulation. Together, these results suggest a complex interplay between visual and central saccadic mechanisms during reading.
Blackford, T., Holcomb, P. J., Grainger, J., & Kuperberg, G. R. (2012). A funny thing happened on the way to articulation: N400 attenuation despite behavioral interference in picture naming. Cognition , 123 (1), 84-99. Full TextAbstract
We measured Event-Related Potentials (ERPs) and naming times to picture targets preceded by masked words (stimulus onset asynchrony: 80 ms) that shared one of three different types of relationship with the names of the pictures: (1) Identity related, in which the prime was the name of the picture ("socks" - ), (2) Phonemic Onset related, in which the initial segment of the prime was the same as the name of the picture ("log" - ), and (3) Semantically related in which the prime was a co-category exemplar and associated with the name of the picture ("cake" - ). Each type of related picture target was contrasted with an Unrelated picture target, resulting in a 3×2 design that crossed Relationship Type between the word and the target picture (Identity, Phonemic Onset and Semantic) with Relatedness (Related and Unrelated). Modulation of the N400 component to related (versus unrelated) pictures was taken to reflect semantic processing at the interface between the picture's conceptual features and its lemma, while naming times reflected the end product of all stages of processing. Both attenuation of the N400 and shorter naming times were observed to pictures preceded by Identity related (versus Unrelated) words. No ERP effects within 600 ms, but shorter naming times, were observed to pictures preceded by Phonemic Onset related (versus Unrelated) words. An attenuated N400 (electrophysiological semantic priming) but longer naming times (behavioral semantic interference) were observed to pictures preceded by Semantically related (versus Unrelated) words. These dissociations between ERP modulation and naming times suggest that (a) phonemic onset priming occurred late, during encoding of the articulatory response, and (b) semantic behavioral interference was not driven by competition at the lemma level of representation, but rather occurred at a later stage of production.
Supplementary Materials Supplementary Figures
Fields, E. C., & Kuperberg, G. R. (2012). It's All About You: an ERP study of emotion and self-relevance in discourse. Neuroimage , 62 (1), 562-74. Full TextAbstract
Accurately communicating self-relevant and emotional information is a vital function of language, but we have little idea about how these factors impact normal discourse comprehension. In an event-related potential (ERP) study, we fully crossed self-relevance and emotion in a discourse context. Two-sentence social vignettes were presented either in the third or the second person (previous work has shown that this influences the perspective from which mental models are built). ERPs were time-locked to a critical word toward the end of the second sentence which was pleasant, neutral, or unpleasant (e.g., A man knocks on Sandra's/your hotel room door. She/You see(s) that he has agift/tray/gunin his hand.). We saw modulation of early components (P1, N1, and P2) by self-relevance, suggesting that a self-relevant context can lead to top-down attentional effects during early stages of visual processing. Unpleasant words evoked a larger late positivity than pleasant words, which evoked a larger positivity than neutral words, indicating that, regardless of self-relevance, emotional words are assessed as motivationally significant, triggering additional or deeper processing at post-lexical stages. Finally, self-relevance and emotion interacted on the late positivity: a larger late positivity was evoked by neutral words in self-relevant, but not in non-self-relevant, contexts. This may reflect prolonged attempts to disambiguate the emotional valence of ambiguous stimuli that are relevant to the self. More broadly, our findings suggest that the assessment of emotion and self-relevance are not independent, but rather that they interactively influence one another during word-by-word language comprehension.
Supplementary Materials
Paczynski, M., & Kuperberg, G. R. (2012). Multiple Influences of Semantic Memory on Sentence Processing: Distinct Effects of Semantic Relatedness on Violations of Real-World Event/State Knowledge and Animacy Selection Restrictions. J Mem Lang , 67 (4), 426-448. Full TextAbstract
We aimed to determine whether semantic relatedness between an incoming word and its preceding context can override expectations based on two types of stored knowledge: real-world knowledge about the specific events and states conveyed by a verb, and the verb's broader selection restrictions on the animacy of its argument. We recorded event-related potentials on post-verbal Agent arguments as participants read and made plausibility judgments about passive English sentences. The N400 evoked by incoming animate Agent arguments that violated expectations based on real-world event/state knowledge, was strongly attenuated when they were semantically related to the context. In contrast, semantic relatedness did not modulate the N400 evoked by inanimate Agent arguments that violated the preceding verb's animacy selection restrictions. These findings suggest that, under these task and experimental conditions, semantic relatedness can facilitate processing of post-verbal animate arguments that violate specific expectations based on real-world event/state knowledge, but only when the semantic features of these arguments match the coarser-grained animacy restrictions of the verb. Animacy selection restriction violations also evoked a P600 effect, which was not modulated by semantic relatedness, suggesting that it was triggered by propositional impossibility. Together, these data indicate that the brain distinguishes between real-world event/state knowledge and animacy-based selection restrictions during online processing.
Supplementary Materials
Cohn, N., Paczynski, M., Jackendoff, R., Holcomb, P. J., & Kuperberg, G. R. (2012). (Pea)nuts and bolts of visual narrative: structure and meaning in sequential image comprehension. Cogn Psychol , 65 (1), 1-38. Full TextAbstract
Just as syntax differentiates coherent sentences from scrambled word strings, the comprehension of sequential images must also use a cognitive system to distinguish coherent narrative sequences from random strings of images. We conducted experiments analogous to two classic studies of language processing to examine the contributions of narrative structure and semantic relatedness to processing sequential images. We compared four types of comic strips: (1) Normal sequences with both structure and meaning, (2) Semantic Only sequences (in which the panels were related to a common semantic theme, but had no narrative structure), (3) Structural Only sequences (narrative structure but no semantic relatedness), and (4) Scrambled sequences of randomly-ordered panels. In Experiment 1, participants monitored for target panels in sequences presented panel-by-panel. Reaction times were slowest to panels in Scrambled sequences, intermediate in both Structural Only and Semantic Only sequences, and fastest in Normal sequences. This suggests that both semantic relatedness and narrative structure offer advantages to processing. Experiment 2 measured ERPs to all panels across the whole sequence. The N300/N400 was largest to panels in both the Scrambled and Structural Only sequences, intermediate in Semantic Only sequences and smallest in the Normal sequences. This implies that a combination of narrative structure and semantic relatedness can facilitate semantic processing of upcoming panels (as reflected by the N300/N400). Also, panels in the Scrambled sequences evoked a larger left-lateralized anterior negativity than panels in the Structural Only sequences. This localized effect was distinct from the N300/N400, and appeared despite the fact that these two sequence types were matched on local semantic relatedness between individual panels. These findings suggest that sequential image comprehension uses a narrative structure that may be independent of semantic relatedness. Altogether, we argue that the comprehension of visual narrative is guided by an interaction between structure and meaning.
Supplementary Materials
Kuperberg, G. R., Kreher, D. A., Swain, A., Goff, D. C., & Holt, D. J. (2011). Selective emotional processing deficits to social vignettes in schizophrenia: an ERP study. Schizophr Bull , 37 (1), 148-63. Full TextAbstract
Schizophrenia is associated with abnormalities in emotional processing and social cognition. However, it remains unclear whether patients show abnormal neurophysiological responses during fast, online appraisals of the emotional meaning of social information. To examine this question, event-related potentials (ERPs) were collected while 18 schizophrenia patients and 18 demographically matched controls evaluated 2-sentence vignettes describing negative, positive, or neutral social situations. ERPs were time locked to a critical word (CW) in the second sentence that conferred emotional valence. A late positivity effect to emotional (vs neutral) CWs was seen in both groups (in controls, to negative and positive CWs; in patients, to negative CWs only). However, the controls showed a greater late positivity effect to the negative and positive (vs neutral) CWs than the schizophrenia patients at mid-posterior (negative vs neutral) and at right posterior peripheral (positive vs neutral) sites. These between-group differences arose from reduced amplitudes of the late positivity to the negative and positive CWs in the patients relative to the controls; there was no difference between the 2 groups in the amplitude of the late positivity to the neutral CWs. These findings suggest that schizophrenia is associated with a specific neural deficit during the online evaluation of emotionally valent, socially relevant information.
Supplementary Materials
Holt, D. J., Lakshmanan, B., Freudenreich, O., Goff, D. C., Rauch, S. L., & Kuperberg, G. R. (2011). Dysfunction of a cortical midline network during emotional appraisals in schizophrenia. Schizophr Bull , 37 (1), 164-76. Full TextAbstract
A cardinal feature of schizophrenia is the poor comprehension, or misinterpretation, of the emotional meaning of social interactions and events, which can sometimes take the form of a persecutory delusion. It has been shown that the comprehension of the emotional meaning of the social world involves a midline paralimbic cortical network. However, the function of this network during emotional appraisals in patients with schizophrenia is not well understood. In this study, hemodynamic responses were measured in 14 patients with schizophrenia and 18 healthy subjects during the evaluation of descriptions of social situations with negative, positive, and neutral affective valence. The healthy and schizophrenia groups displayed opposite patterns of responses to emotional and neutral social situations within the medial prefrontal and posterior cingulate cortices--healthy participants showed greater activity to the emotional compared to the neutral situations, while patients exhibited greater responses to the neutral compared to the emotional situations. Moreover, the magnitude of the response within bilateral cingulate gyri to the neutral social stimuli predicted delusion severity in the patients with schizophrenia. These findings suggest that impaired functioning of cortical midline structures in schizophrenia may underlie faulty interpretations of social events, contributing to delusion formation.
Supplementary Materials
Paczynski, M., & Kuperberg, G. R. (2011). Electrophysiological Evidence for Use of the Animacy Hierarchy, but not Thematic Role Assignment, During Verb Argument Processing. Lang Cogn Process , 26 (9), 1402-1456. Full TextAbstract
Animacy is known to play an important role in language processing and production, but debate remains as to how it exerts its effects: 1) through links to syntactic ordering, 2) through inherent differences between animate and inanimate entities in their salience/lexico-semantic accessibility, 3) through links to specific thematic roles. We contrasted these three accounts in two event related potential (ERP) experiments examining the processing of direct object arguments in simple English sentences. In Experiment 1, we found a larger N400 to animate than inanimate direct object arguments assigned the Patient role, ruling out the second account. In Experiment 2 we found no difference in the N400 evoked by animate direct object arguments assigned the Patient role (prototypically inanimate) and those assigned the Experiencer role (prototypically animate), ruling out the third account. We therefore suggest that animacy may impact processing through a direct link to syntactic linear ordering, at least on post-verbal arguments in English. We also examined processing on direct object arguments that violated the animacy-based selection restriction constraints of their preceding verbs. These violations evoked a robust P600, which was not modulated by thematic role assignment or reversibility, suggesting that the so-called semantic P600 is driven by overall propositional impossibility, rather than thematic role reanalysis.
Supplementary Figures Supplementary Materials
Kuperberg, G. R., Paczynski, M., & Ditman, T. (2011). Establishing causal coherence across sentences: an ERP study. J Cogn Neurosci , 23 (5), 1230-46. Full TextAbstract
This study examined neural activity associated with establishing causal relationships across sentences during on-line comprehension. ERPs were measured while participants read and judged the relatedness of three-sentence scenarios in which the final sentence was highly causally related, intermediately related, and causally unrelated to its context. Lexico-semantic co-occurrence was matched across the three conditions using a Latent Semantic Analysis. Critical words in causally unrelated scenarios evoked a larger N400 than words in both highly causally related and intermediately related scenarios, regardless of whether they appeared before or at the sentence-final position. At midline sites, the N400 to intermediately related sentence-final words was attenuated to the same degree as to highly causally related words, but otherwise the N400 to intermediately related words fell in between that evoked by highly causally related and intermediately related words. No modulation of the late positivity/P600 component was observed across conditions. These results indicate that both simple and complex causal inferences can influence the earliest stages of semantically processing an incoming word. Further, they suggest that causal coherence, at the situation level, can influence incremental word-by-word discourse comprehension, even when semantic relationships between individual words are matched.
Ditman, T., Goff, D., & Kuperberg, G. R. (2011). Slow and steady: sustained effects of lexico-semantic associations can mediate referential impairments in schizophrenia. Cogn Affect Behav Neurosci , 11 (2), 245-58. Full TextAbstract
The present study investigated the contribution of lexico-semantic associations to impairments in establishing reference in schizophrenia. We examined event-related potentials as schizophrenia patients and healthy, demographically matched controls read five-sentence scenarios. Sentence 4 introduced a noun that referred back to three possible referents introduced in Sentences 1-3. These referents were contextually appropriate, contextually inappropriate but lexico-semantically associated, and contextually inappropriate and lexico-semantically nonassociated. In order to determine whether participants had correctly linked the anaphor to its referent, the final sentence reintroduced each referent, and participants indicated whether the last two sentences referred to the same entity. Results indicated that between 300 and 400 ms, patients, like healthy controls, used discourse context to link the noun with its preceding referent. However, between 400 and 500 ms, neural activity in patients was modulated only by lexico-semantic associations, rather than by discourse context. Moreover, patients were also more likely than controls to incorrectly link the noun with contextually inappropriate but lexico-semantically associated referents. These results suggest that at least some types of referential impairments may be driven by sustained activation of contextually inappropriate lexico-semantic associations.
Ditman, T., & Kuperberg, G. R. (2010). Building coherence: A framework for exploring the breakdown of links across clause boundaries in schizophrenia. J Neurolinguistics , 23 (3), 254-269. Full TextAbstract
Clinically, patients with schizophrenia show prominent abnormalities at the discourse level, with production characterized by tangential and illogical relationships between ideas and unclear references. Despite these clinical manifestations, most studies of language in schizophrenia have focused on semantic relationships between single words and the build-up of meaning within single-clause sentences. The present paper discusses the few studies that have gone beyond clause boundaries to fully understand language impairments in schizophrenia. We also give an overview of a relevant literature that considers the neurocognitive mechanisms by which coherence links are established across clauses in healthy adults, providing a framework that may guide future research in this area.
Kuperberg, G. R., Choi, A., Cohn, N., Paczynski, M., & Jackendoff, R. (2010). Electrophysiological correlates of complement coercion. J Cogn Neurosci , 22 (12), 2685-701. Full TextAbstract
This study examined the electrophysiological correlates of complement coercion. ERPs were measured as participants read and made acceptability judgments about plausible coerced sentences, plausible noncoerced sentences, and highly implausible animacy-violated sentences ("The journalist began/wrote/astonished the article before his coffee break"). Relative to noncoerced complement nouns, the coerced nouns evoked an N400 effect. This effect was not modulated by the number of possible activities implied by the coerced nouns (e.g., began reading the article; began writing the article) and did not differ either in magnitude or scalp distribution from the N400 effect evoked by the animacy-violated complement nouns. We suggest that the N400 modulation to both coerced and animacy-violated complement nouns reflected different types of mismatches between the semantic restrictions of the verb and the semantic properties of the incoming complement noun. This is consistent with models holding that a verb's semantic argument structure is represented and stored at a distinct level from its syntactic argument structure. Unlike the coerced complement noun, the animacy-violated nouns also evoked a robust P600 effect, which may have been triggered by the judgments of the highly implausible (syntactically determined) meanings of the animacy-violated propositions. No additional ERP effects were seen in the coerced sentences until the sentence-final word that, relative to sentence-final words in the noncoerced sentences, evoked a sustained anteriorly distributed positivity. We suggest that this effect reflected delayed attempts to retrieve the specific event(s) implied by coerced complement nouns.
Supplementary Materials
De Grauwe, S., Swain, A., Holcomb, P. J., Ditman, T., & Kuperberg, G. R. (2010). Electrophysiological insights into the processing of nominal metaphors. Neuropsychologia , 48 (7), 1965-84. Full TextAbstract
We used event-related potentials (ERPs) to examine the time-course of processing metaphorical and literal sentences in the brain. ERPs were measured to sentence-final (Experiment 1) and mid-sentence (Experiment 2) critical words (CWs) as participants read and made plausibility judgments about familiar nominal metaphors ("A is a B") as well as literal and semantically anomalous sentences of the same form. Unlike the anomalous words, which evoked a robust N400 effect (on the CW in experiments 1 and 2 as well as on the sentence-final word in experiment 2), CWs in the metaphorical, relative to the literal, sentences only evoked an early, localized N400 effect that was over by 400ms after CW onset, suggesting that, by this time, their metaphorical meaning had been accessed. CWs in the metaphorical sentences also evoked a significantly larger LPC (Late Positive Component) than in the literal sentences. We suggest that this LPC reflected additional analysis that resolved a conflict between the implausibility of the literal sentence interpretation and the match between the metaphorical meaning of the CW, the context and stored information within semantic memory, resulting from early access to both literal and figurative meanings of the CWs.
Supplementary Materials
Nieuwland, M. S., Ditman, T., & Kuperberg, G. R. (2010). On the incrementality of pragmatic processing: An ERP investigation of informativeness and pragmatic abilities. J Mem Lang , 63 (3), 324-346. Full TextAbstract
In two event-related potential (ERP) experiments, we determined to what extent Grice's maxim of informativeness as well as pragmatic ability contributes to the incremental build-up of sentence meaning, by examining the impact of underinformative versus informative scalar statements (e.g. "Some people have lungs/pets, and…") on the N400 event-related potential (ERP), an electrophysiological index of semantic processing. In Experiment 1, only pragmatically skilled participants (as indexed by the Autism Quotient Communication subscale) showed a larger N400 to underinformative statements. In Experiment 2, this effect disappeared when the critical words were unfocused so that the local underinformativeness went unnoticed (e.g., "Some people have lungs that…"). Our results suggest that, while pragmatic scalar meaning can incrementally contribute to sentence comprehension, this contribution is dependent on contextual factors, whether these are derived from individual pragmatic abilities or the overall experimental context.
Supplementary Materials
Kuperberg, G. R. (2010). Language in schizophrenia Part 1: an Introduction. Lang Linguist Compass , 4 (8), 576-589. Full TextAbstract
This is the first of two articles that discuss higher-order language and semantic processing in schizophrenia. This article reviews clinical characterizations of language output and the phenomenon of positive thought disorder, as well as more principled characterizations of language output in schizophrenia. It also gives an overview of evidence for the predominant theory of language dysfunction in schizophrenia: that it arises from abnormalities in (a) semantic memory and/or (b) working memory and executive function. The companion article (Part 2) focuses on the study of language in schizophrenia using online psycholinguistic methods and considers how the study of schizophrenia may inform our understanding of normal language processing.
Kuperberg, G. R. (2010). Language in schizophrenia Part 2: What can psycholinguistics bring to the study of schizophrenia..and vice versa? Lang Linguist Compass , 4 (8), 590-604. Full TextAbstract
This is the second of two articles that discuss higher-order language and semantic processing in schizophrenia. The companion article (Part 1) gives an introduction to language dysfunction in schizophrenia patients. This article reviews a selection of psycholinguistic studies which suggest that sentence-level abnormalities in schizophrenia may stem from a relative overdependence on semantic associative relationships at the expense of building higher-order meaning. Language disturbances in schizophrenia may be best conceptualized as arising from an imbalance of activity across two streams of processing, one drawing upon semantic relationships within semantic memory and the other involving the use of combinatorial mechanisms to build propositional meaning. I will also discuss some of the ways in which the study of schizophrenia may offer new insights into the cognitive and neural architecture of the normal language system.
Sitnikova, T., Perrone, C., Goff, D., & Kuperberg, G. R. (2010). Neurocognitive mechanisms of conceptual processing in healthy adults and patients with schizophrenia. Int J Psychophysiol , 75 (2), 86-99. Full TextAbstract
This overview outlines findings of cognitive and neurocognitive studies on comprehension of verbal, pictorial, and video stimuli in healthy participants and patients with schizophrenia. We present evidence for a distinction between two complementary neurocognitive streams of conceptual analysis during comprehension. In familiar situations, adequate understanding of events may be achieved by mapping the perceived information on the associative and similarity-based connections between concepts in semantic memory - a process reflected by an N400 waveform of event-related electrophysiological potentials (ERPs). However, in less conventional contexts, a more flexible mechanism may be needed. We suggest that this alternative processing stream, reflected by a P600 ERP waveform, may use discrete, rule-like goal-related requirements of real-world actions to comprehend relationships between perceived people, objects, and actions. This neurocognitive model of comprehension is used as a basis in discussing studies in schizophrenia. These studies suggest an imbalanced engagement of the two conceptual streams in schizophrenia, whereby patients may rely on the associative and similarity-based networks in semantic memory even when it would be more adaptive to recruit mechanisms that draw upon goal-related requirements. Finally, we consider the roles that these conceptual mechanisms may play in real-life behavior, and the consequences that their dysfunction may have for disorganized behavior and inability to plan actions to achieve behavioral goals in schizophrenia.
Kuperberg, G. R., Kreher, D. A., & Ditman, T. (2010). What can Event-related Potentials tell us about language, and perhaps even thought, in schizophrenia? Int J Psychophysiol , 75 (2), 66-76. Full TextAbstract
Disturbances of thought and language are fundamental to schizophrenia. Cognitive behavioral and electrophysiological research has implicated problems in two different neurocognitive mechanisms: abnormalities in the structure and function of semantic memory, and abnormalities in combining and integrating words together to build up sentence and discourse context. This review discusses recent electrophysiological evidence suggesting that these two deficits are not completely distinct, but rather that language impairment in schizophrenia results from a dysfunctional interaction between these systems in an effort to build up higher-order meaning. Moreover, although language abnormalities are more pronounced in patients with positive thought disorder, they manifest themselves in all patients when increased demands are placed on the comprehension system. Further investigation of language dysfunction may also provide insights into other aspects of psychotic thought.
Kuperberg, G., Ditman, T., Kreher, D., & Goldberg, T. (2009). Behavioral and electrophysiological approaches to understanding language dysfunction in neuropsychiatric disorders: Insights from the study of schizophrenia. In S. Wood, N. Allen and C. Pantelis (Eds): Handbook of Neuropsychology of Mental Illness (pp. 67-95) . Cambridge University Press. Full Text