The Neural Network for Sign Language Comprehension

2.1 |. From the Eyes to the Brain

The brain first receives and encodes visual information from the retinas as a retinotopic map (i.e., a ‘map’ of neurons whose arrangement matches the arrangement of receptors in the retina) in the most posterior part of the occipital lobe, known as primary visual cortex (V1). Neurons in these regions encode information about very simple visual features and feed this information to higher-level retinotopic regions—secondary visual cortex—which encode more complex visual patterns or characteristics in the input. These early visual processes occur very quickly: bilateral posterior occipital cortex responds to the perceptual properties of signs about 80–120 ms after the onset of the sign (Leonard et al. 2012). The timing of early visual processing of the signed signal appears to be analogous to early auditory processing of spoken words in the bilateral superior temporal gyrus (STG).

The activation of occipital cortex for sign perception is primarily driven by domain-general visual processes which are not specific to signers. For instance, both signers and nonsigners synchronise their brain waves (measured by EEG or MEG) to the oscillating movements of the signer’s hands, a phenomenon which is called entrainment (Brookshire et al. 2017). Neural oscillations synchronised with sign movements are argued to be analogous to neural oscillations that entrain to acoustic oscillations in speech (i.e., fluctuations in volume). However, neural entrainment in frontal cortex (Brookshire et al. 2017) and in right parietal cortex (Rivolta 2023) is only observed in signers, indicating that linguistic knowledge can strengthen temporal predictions in these regions. Similarly, occipital activation in response to sign input can be influenced by higher-level linguistic processes, such as internal predictive models of upcoming sign input, which may be generated by frontal regions within the language network. When these predictions are violated, early visual processing effort appears to increase to resolve the mismatch between linguistic prediction and visual input (Stroh et al. 2019; Almeida et al. 2016).

Sign-specific occipital activity can also be observed without relying on linguistic violations, which are presumed to elicit top-down linguistic reanalysis. Evans et al. (2019) used a representational similarity analysis to assess which neural areas were activated similarly when signers perceived the same signs produced by two different signers (a man and a woman). The results identified bilateral regions in primary and secondary visual cortex. Because activation patterns within these regions were consistent across different sign models, these regions seem able to encode information about the abstract structure of signs in a consistent manner even when the retinal ‘picture’ is wholly different.

2.2 |. Recognising Linguistic Articulators: The Hands and Face

Spoken languages use one set of articulators: the vocal tract consisting of the lips, tongue, and larynx, which work together to create speech sounds. In contrast, sign languages use several articulators that can move independently of each other: the two hands and the face. The movements of the speech articulators are perceived auditorily and indirectly (e.g., some movements of the tongue and larynx are not visible). In contrast, the sign articulators are all perceived directly through vision. The brain analyzes the visual signal by decomposing it into distinct processing streams, which eventually diverge into separate but interdependent regions of the brain. One stream, known as the ventral stream (see Figure 2), is responsible for shape and object recognition (e.g., Milner and Goodale 2008). The ventral stream primarily travels through occipital visual regions to inferior temporal (IT) cortex, which contains neural regions that are functionally specialised for identifying bodies, body parts, and faces (see Peelen and Downing 2007, for a review).

In sign language comprehension, posterior IT cortex is often activated for simple tasks which are meant to identify regions involved in sublexical or phonetic processes (Matchin et al. 2022; P. C. Trettenbrein et al. 2021; Mayberry et al. 2011). However, posterior IT has not been reliably implicated in language-specific processes, and IT-specific activation may be simply due to the activation of hand and face representations during sign perception without specific contributions to linguistic processing. Corina et al. (2007) found IT activation in deaf signers when they viewed meaningless gestures, but not ASL signs. They suggested that the lack of IT activation for signs was due to top-down mechanisms, possibly in higher-level occipital regions, which quickly filter out linguistic from nonlinguistic hand movements to be processed by regions specialised for linguistic processing, rather than posterior IT cortex.

While known for processing static facial structure, face-selective regions in posterior IT are also implicated in processing dynamic facial expressions through their interaction with more dorsal regions in superior temporal cortex (see Lander and Butcher 2015, for a review). Facial expressions carry linguistic content in sign languages (e.g., marking adverbials, topics, conditional clauses—see Figure 1C), and linguistic facial expressions are distinct from emotional facial expressions (Reilly et al. 1990). Linguistic and emotional facial expressions in the signed signal activate IT regions for both signers and nonsigners, but activation is left-lateralised in signers and bilateral in nonsigners (McCullough et al. 2005). The perception of mouth patterns (phonologically-specified mouth movements that are unrelated to speech) in British Sign Language (BSL) also activate left IT more than lipreading speech for deaf signers (Capek et al. 2008), indicating sensitivity to linguistic facial expressions in left IT.

Therefore, for faces (and possibly for other body parts), left posterior IT may serve a role in dynamic feature processing during sign language perception through its resonance with regions in the superior temporal sulcus (STS) implicated in the perception of linguistic facial expressions (McCullough et al. 2005). Just as in nonsigners, the right STS of signers is involved in perceiving emotional facial expressions, but the left STS is uniquely recruited for processing linguistic information in facial expressions and emotional facial expressions in a linguistic context (McCullough et al. 2005; Emmorey and McCullough 2009).

2.3 |. Processing Linguistic Movement

Lifelong signers have extensive experience with the online extraction of linguistic information from rapid and complex manual and facial movements, and studies have shown that movement alone is a powerful carrier of linguistic information about signs (Poizner 1983; Poizner et al. 1981; E. Malaia et al. 2018). With point-light displays, in which only moving points of light representing the signer’s joints are visible, signers can identify individual signs, recognise which movements are purposeful or transitional, and identify familiar signers (Poizner et al. 1981; Klima et al. 1999; Bigand et al. 2020).

Recent research has suggested that the network involved in processing visual movement can be decomposed into two distinct pathways: the dorsal stream and the lateral stream (see Figure 2; Pitcher and Ungerleider 2021; McMahon et al. 2023). The dorsal stream constitutes a network that processes information about the spatial location and movement of objects and their related actions (e.g., reaching and grasping). Regions in the dorsal stream, such as the superior parietal lobule (SPL), seem to be involved in sign language comprehension only in limited contexts (see Section 4.3). In contrast, regions in the lateral stream are critically implicated in sign language comprehension.

The lateral stream is argued to be primarily engaged in dynamic social perception, and regions within this stream seem to be robustly activated for sign language comprehension. The lateral stream travels from primary visual cortex through higher-lever visual regions such as the middle temporal complex (area MT+) (see Figure 2). MT+ is implicated in the perception of simple and complex motion (McMahon et al. 2023; Bavelier et al. 2001). As would be expected for a linguistic signal with rich visual motion, MT+ is robustly activated during sign language perception for both signers and nonsigners (e.g., Levänen et al. 2001; MacSweeney, Woll, Campbell, McGuire, et al. 2002). The lateral stream terminates in the posterior STS, a functionally heterogeneous region which responds to a broad set of visual stimuli, including biological motion and social actions (McMahon et al. 2023).

MT+ is also more activated for signed sentences which describe visual motion (‘the deer walked along the hillside’) as compared to near-identical sentences which instead describe static scenes (‘the deer slept along the hillside’). This result suggests that MT+ is also engaged by linguistic motion semantics as the two sentence types were matched for amount of visual motion (McCullough et al. 2012). Just as for early visual areas, area MT+ seems to contribute to abstract linguistic processes over and above the perception of motion itself. However, studies have not yet investigated whether regional connectivity and/or activation within MT+differs for linguistic versus domain-general movement patterns.

2.4 |. The Action Observation Network

Research on nonlinguistic action observation has identified a network of frontoparietal regions which are engaged when one observes other people’s actions. This network, called the Action Observation Network (AON; Condy et al. 2021), is proposed to be critical to recognising others’ actions by mentally imitating or simulating the observed actions using one’s own motor system. This network would reasonably be expected to be engaged in sign language comprehension. Surprisingly, this prediction has not been borne out by studies looking at the AON in sign language comprehension. Studies comparing nonsigners and signers have found that nonsigners activate the AON as they view signs or communicative gestures, whereas signers do not (Corina et al. 2007; Emmorey et al. 2010). EEG studies which measure the mu rhythm tell a similar story. The mu rhythm is a synchronised pattern of electrical activity over sensorimotor cortex that has been correlated with action synchronisation in the AON. By measuring the mu rhythm while participants watched signs, Kubicek and Quandt (2019) found that nonsigners recruited the AON during sign perception, whereas signers did not.

The AON might not be critical to sign language comprehension because after a lifetime of experience decoding the signed signal, signers might rapidly dissociate between linguistic and nonlinguistic actions to send them down distinct processing streams (Corina et al. 2007; Kubicek and Quandt 2019). Instead of being processed by the AON, the signed signal is sent to more canonical language areas. However, the AON might not be fully disengaged for sign language comprehension. Kubicek and Quandt (2019) reported differential AON engagement for one-handed versus two-handed signs in signers, and this effect was larger in a later study which asked signers to repeat the signs after viewing them (Quandt and Willis 2021). Therefore, the AON might be only or especially recruited for linguistic processing when task demands surpass passive perception. It also may be that the AON is activated only when internal motor simulation of an observed action would help to perceive or produce that action. In everyday sign comprehension (without additional tasks), a signer would not need to continually simulate the signed signal and is able to rely on a more efficient visual-linguistic network through occipital and temporal regions to comprehend the linguistic input (for behavioural evidence that signers do not automatically engage in motor simulation during sign perception see Brozdowski and Emmorey 2020).

3.1 |. From Perception to Phonology

Phonologically, signs are instantiated as a structured combination of phonological parameters, and the perceiver must segment the signing stream into distinct signs using word-level phonotactics (e.g., using the Possible Word Constraint; Orfanidou et al. 2010). A shift from perceptual representations to phonological structure implies several important processing steps, most importantly the normalisation of variable perceptual representations into fixed phonological representations and the fusion of these representations into a unified lexical representation.

Of the major phonological parameters (handshape, location, and movement; see Figure 1A), the movement parameter instantiates the prosodic structure of signs by acting as the nucleus of sign syllables (somewhat analogous to vowels in spoken languages; see Brentari 2019, for review). Movement may serve an important role in segmenting the signed stream into phonological representations. As reviewed in Section 2.2, sign language perception engages regions involved in motion perception; this movement-based information may then be analysed to parse signing into syllables to facilitate linguistic comprehension. E. A. Malaia and Wilbur (2020) propose that during both signed and spoken language comprehension, the brain entrains to a modality-independent syllabic rate (~4 Hz; but see Rivolta 2023). For sign languages, this syllabic rate is perceptually marked by rises and falls in visual entropy, allowing neural entrainment (as measured by EEG) to this rate, which may enable the brain to parse syllables for linguistic analysis and comprehension.

Unlike spoken languages, the phonology of signs is not primarily based on sequential structure; rather, signs are composed of both simultaneous and sequentially organised phonological units, with simultaneous units (e.g., handshape) remaining salient throughout the sign syllable. The simultaneous perception of many aspects of phonological form is a unique feature of sign languages that is enabled by the visuospatial modality. Handshape and location (place of articulation on the body), as simultaneous phonological units, remain the same or change only in constrained ways within each sign. To extract stable phonological representations of handshape and location from a dynamic and variable perceptual stream, the brain may engage in perceptual normalisation. Perceptual normalisation occurs when phonetic variation within phonemic categories is ignored, allowing for consistent categorisation of perceptual input into the appropriate phonological representation.

Such categorical perception (CP) effects have been found for signed stimuli: While both signers and nonsigners similarly identify and categorise handshapes that fall along a physical continuum, only signers exhibit better discrimination for handshapes that straddle a category boundary, as opposed to handshapes that fall within a category (Baker et al. 2005; Emmorey et al. 2003; see also, Eccarius 2008). However, CP effects have not been observed for location (Emmorey et al. 2003) and for some handshapes (Best et al. 2010). Sign perception is also impacted by viewing angle (Watkins et al. 2024; Corina et al. 2011), indicating that the brain does not extract angle-invariant representations of signs—a result that parallels the visual perception of objects (Edelman and Bülthoff 1992). Nonetheless, event-related potential (ERP) evidence from phonological priming paradigms indicates that (for signers only) the brain categorises phonological representations of handshapes and locations across signs (Meade et al. 2022).

There is also evidence that perceptual representations of signs and abstract phonological representations of signs activate distinct functional networks. In Cardin et al. (2016), deaf BSL signers, deaf nonsigners, and hearing nonsigners were asked to detect a target phonological feature (handshapes or locations) while watching a series of BSL signs, foreign signs, and phonologically illegal nonsigns. They found that for all groups, monitoring for a specific handshape versus for a specific location activated distinct perceptual networks encompassing inferior temporal and parietal regions implicated in complex visual perception. Critically, both monitoring tasks recruited bilateral superior temporal cortex (STC) only in the deaf signers, suggesting that linguistic phonological structure was processed by this region regardless of the specific perceptual processing demands. However, the meta-linguistic nature of the monitoring task provides only indirect evidence that phonological representations are encoded in bilateral STC during naturalistic comprehension.

3.2 |. Neural Regions for Sign Phonology

The task of converting perceptual representations of signing into stable phonological representations, and storing these representations seems to be subserved by a cortical network encompassing the posterior STC and the supramarginal gyrus (SMG; see Figure 2). The STC is a large neural region which encompasses the superior temporal gyrus (STG) and sulcus (STS). These regions are often subdivided into anterior (aSTG/aSTS) and posterior (pSTG/pSTS) regions.

Bilateral pSTS is reliably and robustly activated for sign language perception in deaf signers (Cardin et al. 2016; Emmorey, Xu et al. 2011; Emmorey, McCullough et al. 2011; Capek et al. 2008; MacSweeney, Woll, Campbell, McGuire, et al. 2002; Levänen et al., 2001; Neville et al. 1998). Emmorey, Xu et al. (2011); Emmorey, McCullough et al. (2011) scanned deaf signers and hearing nonsigners while they viewed ASL pseudosigns (possible but non-existing signs) and non-iconic signs, aiming to assess what neural regions respond to signed phonological structure without lexical meaning. They found that left STS seems to become tuned to linguistically structured body movements in signers, as signers activated left STS more than nonsigners when passively perceiving ASL pseudosigns, which contain phonological structure but no lexical meaning. This replicated results from a positron emission tomography (PET) study by Petitto et al. (2000), which found bilateral STS activation for signers but not nonsigners in response to pseudosigns and real signs. Emmorey, Xu et al. (2011) and Emmorey, McCullough et al. (2011) speculated that right STS, which was activated to a lesser extent, might subserve more gradient phonological processes whereas left STS might subserve ‘categorical and combinatorial processing of sublexical sign structure’.

The pSTG may also be involved in processing sign phonology. Emmorey, Xu et al. (2011) and Emmorey, McCullough et al. (2011) found that the left pSTG was activated more during pseudosign than sign perception in deaf signers only. Because left pSTG is known to process the sound structure of speech, the authors speculated that left pSTG may be involved in sublexical sign processing by identifying linguistic phonetic units within a dynamic signal, either auditory or visual. Malaia and Wilbur (2020) further suggest that the right STG is involved in segmenting sign syllables based on movement. However, the precise nature of the structural and/or segmental processing carried out by pSTG is not currently understood. For spoken languages, Binder (2017) argued that the left pSTG is not critical for speech comprehension but rather supports the storage of phonological codes which are retrieved for language production or for tasks which require holding these forms in short-term memory. If the linguistic functions of the pSTG are supramodal, this might be true for sign languages as well.

Cardin et al. (2016) also found that nonsigns activated bilateral SMG more than signs in deaf signers, and they interpreted this finding as an increase in phonological processing due to the nonsigns’ violation of phonotactic rules. The SMG is often implicated in phonological processing of signs (Corina et al. 1999; Emmorey et al. 2015; MacSweeney et al. 2004).

Corina et al. (1999) found that disrupting the function of the left SMG in a deaf signer (via cortical stimulation) caused phonological substitution errors in handshape and movement which were sometimes also semantically related, suggesting disruption at the lexical level. These findings implicate the left SMG in the assembly of phonological representations for lexical production (see also Emmorey et al. 2016). An electrocorticographic (eCoG) study analysing the spontaneous production of signs by a deaf patient found that activity of electrodes over the left SMG were correlated with the types of handshapes produced (Leonard et al. 2020). Meanwhile, electrodes over hand and face areas in sensorimotor cortex were correlated with the location of produced signs. Although Corina et al. and Leonard et al. investigated sign production, these studies provide some evidence that sign sublexical structure is systematically organised within the SMG.

The precise functional organisation and computational roles of the components within the phonological network in pSTS, pSTG, and SMG are still unclear and present important topics for future research. The STS is a core region for sign language comprehension, implicated in processing biological motion, linguistic facial expressions, sign phonology, syntactic structure, and semantics (McCullough et al. 2005; Emmorey, Xu et al., 2011; Emmorey, McCullough et al. 2011; MacSweeney et al. 2006; Matchin et al. 2022; Leonard et al. 2012). Future studies could employ fMRI techniques such as multivariate pattern and voxelwise analyses to identify how distinct types of linguistic information are organised within the STS at a finer scale, as has been done for social perception within this region (McMahon et al. 2023; Deen et al. 2015).

4.1 |. Lexical Processing of Signs

When the phonological units of a sign are bound together, the brain is presumed to activate a coherent abstract lexical representation, which includes the semantic and syntactic properties of the sign. To isolate the neural processes underlying the comprehension of lexical signs, neuroimaging studies have typically used lists of unconnected signs, with the expectation that such stimuli will not elicit combinatorial processing due to the lack of grammatical structure. Lists of unconnected signs are contrasted with a low-level baseline (e.g., viewing the sign model at rest). These studies have identified a bilateral network for lexical processing which encompasses posterior temporal cortex and the inferior and middle frontal gyri (IFG and MFG; see Figure 2) (e.g., MacSweeney et al. 2006; Li et al. 2014; Emmorey et al. 2015; Banaszkiewicz et al. 2021; Matchin et al. 2022).

Superior temporal regions within this network are also activated for sublexical processes, which can make it difficult to disentangle lexical from sublexical representations and processes. However, an MEG study by Leonard et al. (2012) found that activation in bilateral pSTS and pSTG is modulated by the semantic congruence between a picture and a sign for ASL signers. This neural modulation occurred in the same time window as the picture/word semantic congruence effect in the same regions for English speakers (200–400 ms after sign/word onset). The sensitivity of pSTS and pSTG to semantic content suggests that bilateral superior temporal cortex is involved in encoding semantic lexical representations for both sign and speech.

Two neural regions that are implicated in lexical processing for sign language are the posterior middle temporal gyri (pMTG) and inferior frontal gyri (IFG), which exhibit bilateral engagement but with greater activation in the left hemisphere (e.g., Emmorey et al. 2015; MacSweeney et al. 2006). The IFG and pMTG are often coactivated and are thought to be part of a lexicosemantic network that builds phonological representations into semantic and syntactic representations (Matchin and Hickok 2020). This lexicosemantic network is activated for single signs, but it also shares many commonalities with patterns of activation in response to sign sentences (see below). Therefore, this network may be functionally integrated with neural circuits underlying semantic-syntactic processing at the phrase level, and there may be no clear computational or regional demarcation between the comprehension of single signs and sentences.

In addition, neural regions outside the traditional network for spoken languages have sometimes been implicated in lexical processing for sign languages. One such area is the left intraparietal sulcus (IPS; see Figure 2), which has been found to be engaged during semantic processing of single signs when making picture-sign matching judgements (Leonard et al. 2012) or concreteness judgements (Emmorey et al. 2015). This region is not typically activated during the comprehension of single words (Leonard et al. 2012), and it might play a modality-specific role related to sign production. i.e., left IPS is often engaged during single sign production (e.g., Emmorey et al. 2016), and signers might recruit this motor-related region when making overt decisions about signs (rather than passive viewing).

A lexical feature that is arguably more common in signed than spoken languages is iconicity, the resemblance between a form and its meaning. For example, the form of the ASL sign DRINK depicts the act of drinking from a glass. However, it turns out that this lexical feature has little impact on neural processing. The comprehension and production of iconic and non-iconic signs are equally impaired with sign language aphasia (Atkinson et al. 2005), and evidence from fMRI (Evans et al. 2019), PET (Emmorey, Xu et al., 2011; Emmorey, McCullough et al. 2011), and EEG (Emmorey et al., 2020) indicates that the iconicity of signs does not influence neural activation patterns. Although recent ERP studies have found effects of iconicity on neural responses during picture-naming (Baus and Costa 2015; McGarry et al. 2024), these effects appear to be task-specific and may be related to the mapping between visual features of the picture and iconic features of the signs (Gimeno-Martinez and Baus, 2022; McGarry et al., 2023).

4.2 |. Sentence Processing in Sign Languages

To understand the thoughts being expressed by a signer, the perceiver must use their knowledge of the language’s grammar to interpret the order of lexical items in the signing stream and combine them into phrases with a cohesive meaning. Studies which assess sentence-level comprehension in sign languages have identified the same frontotemporal (also called the perisylvian) language network i.e. activated for lexical-level comprehension across several sign languages (e.g., MacSweeney et al. 2006; Mayberry et al. 2011; Inubushi and Sakai 2013; Emmorey et al. 2014). Language activation for sentence comprehension is often bilateral, but signers with lesions in left posterior temporal lobe perform worse on single-sign and sentence comprehension tasks compared to signers with lesions in the right temporal lobe (Hickok et al. 2002). This finding highlights the importance of the left posterior temporal cortex in sign language comprehension.

A question of interest is whether certain regions within this frontotemporal network specifically represent sentence-level (supralexical) processes, that is syntactic and semantic combinatorial processes. Studies on sentence-level processing must find a way to subtract neural activation associated with the lexical items themselves. One method is to contrast two or more conditions which only differ along the syntactic/semantic dimension to isolate neural activation associated with this dimension. MacSweeney et al. (2006) compared sentences with unconnected lists of signs and found that sentences recruited the left IFG and posterior temporal regions more than sign lists. More recently, Matchin et al. (2022) compared activation for three conditions that gradually increased in grammatical complexity: sign lists, simple two-sign sentences, and complex sentences. They found that increasing syntactic and semantic complexity elicited activation in left-lateralised anterior and posterior STS. Activation in this network peaked in the anterior temporal lobe (ATL), a region known to be involved in combinatorial semantics (see Blanco-Elorrieta et al. 2018, for a sign production study implicating the ATL in phrase-building). The regions activated by syntactic/semantic complexity only minimally overlapped with inferior temporal and occipital regions activated by the sign lists alone. Thus, Matchin et al. (2022) argue for a network centred around left STS which is sensitive to sentential structure and is distinct from a more posterior and bilateral network for lexical comprehension.

Other studies have used the same sign stimuli but changed the cognitive task to isolate sentence-level processes. Mayberry et al. (2011), Inubushi and Sakai (2013), and Stroh et al. (2019) all found increased or more widespread left IFG activation when signers were asked to make decisions requiring the grammatical analysis of sentences rather than phonological or lexical decisions. While the left IFG is activated for lexical comprehension along with the pMTG, these two regions seem especially involved in sentential processing. A meta-analysis by Trettenbrein et al. (2021) concluded that within a bilateral frontal-temporal-occipital sign language network, the left IFG is a hub for integrating abstract linguistic information in a similar manner as spoken and written languages. A network analysis by Liu et al. (2017) with hearing signers and non-signers reached a similar conclusion: a region in the left IFG was identified as a network hub only within signers.

The large-scale frontotemporal overlap between lexical and syntactic processes is consistent with recent accounts of the language system which postulate an integrated and distributed language system which does not abstract syntax away from lexical and compositional semantics (Shain et al. 2024). In this account, temporal and inferior frontal regions that have been hypothesised to be specialised for syntax process both lexical and phrase-level information dynamically and in parallel as the network builds meaning from linguistic input.

4.3 |. The Supramodal Language Network

Whereas the visuospatial nature of sign language comprehension leads to very different neural computations involved in decoding phonological structure, the neural substrates of signed and spoken comprehension above the lexical level are largely identical: a bilateral, but more left-lateralised, network of frontal and temporal regions. This frontotemporal network is similarly activated by sign, speech, and text comprehension within individual signers (Emmorey et al. 2014; Emmorey et al. 2015; Waters et al. 2007) and has been described as the supramodal language network (Matchin et al. 2022; Liu et al. 2020). Liu et al. (2020) found that hearing signers’ individual patterns of neural activation while comprehending spoken, written, and signed sentences within the frontotemporal network were similar across modalities. This finding suggests that language representations within these regions are supramodal, rather than unimodal representations which occupy common regions.

While regions across the frontotemporal language network may be similarly activated for all language modalities, there may be differences in the linguistic computations and representations subserved by these regions. For example, Evans et al. (2019) found that the semantic representations of individual spoken and signed lexical items (e.g., ‘train’, ‘bus’, ‘apple’, ‘grapes’) were not shared across languages (BSL and English) within an individual, but the neural representation of semantic categories was shared across languages. That is, while the concepts of modes of transportation and fruit might be supramodal in the brain of a bimodal bilingual, the neural representation of the spoken word train may be distinct from that for the sign TRAIN.

There are also some differences in the higher-level language network that are associated with linguistic constructions unique to sign languages. For example, classifier constructions are complex expressions in which the location and movement of the hands can depict the location and movement of referents, and handshape can represent the type of referent, such as a person or flat object (see papers in Emmorey 2003, and Figure 1B). Both lesion evidence (Hickok et al. 1996; Atkinson et al. 2005) and neuroimaging evidence (Emmorey et al. 2013, 2021) indicate that the right hemisphere is critical for comprehending and producing this type of spatial language (see Corina et al. 2013, for a review). These constructions may particularly necessitate the involvement of the right superior parietal lobule (SPL), which is not typically implicated in language processing. The SPL is a core region in the dorsal stream (see Section 2.3 and Figure 2), a network specialised for action perception which requires precise information about spatial locations (e.g., for reaching and grasping). In signers, it may contribute to the linguistic processing of classifier constructions.

Similarly, co-reference in sign languages involves the association and maintenance of referents with distinct locations in signing space (see Figure 1C), and again both lesion and neuroimaging data suggest a role for the right hemisphere, with neuroimaging data implicating the right SMG (Hickok et al. 1999; Stroh et al. 2019). The SPL along with the inferior parietal lobule (IPL) are also implicated, although MacSweeney et al. (2002) found involvement of left, not right, SPL and IPL in spatial reference processing. Interestingly, EEG evidence indicates that these types of constructions nonetheless exhibit similar neural responses (an increased N400) when they appear in non-canonical sentence structures, indicating a linguistic (rather than gestural) processing strategy (Krebs et al. 2021).

Beyond linguistic constructions that specifically require spatial processing, increased bilateral activation for signed as compared to spoken languages has been observed since the earliest ERP and fMRI studies on sign language comprehension (see Trettenbrein et al. 2021 for a meta-analysis). However, as noted above, lesion studies indicate that only left hemisphere damage results in frank sign language aphasias (e.g., Atkinson et al. 2005; Poizner et al. 1987). Laterality differences in the neuroimaging data have been argued to be stimulus-specific: the addition of a visual display of the interlocutor (required for sign, but not for speech) recruits right-hemispheric regions (MacSweeney, Woll, Campbell, McGuire, et al. 2002; Capek et al. 2004). Occipital and parietal regions are bilaterally activated by necessity, as each hemisphere processes visual and spatial information from the contralateral side of the body. Frontotemporal language areas, by contrast, still display a bias to the left hemisphere, supporting the importance of this hemisphere for supramodal language processes (see Trettenbrein et al. 2023, for a review). Nonetheless, some right-hemispheric regions (particularly in posterior temporal and inferior parietal cortex) seem to engage in linguistic processes, but the functional asymmetries between left and right regions remain underspecified.