Research

Dynamic representations of motion events in sentence processing (Dr Yuki Kamide & Dr Shane Lindsay)

In the environment that surrounds us, objects constantly undergo changes, for example, regarding their relative spatial locations over time. It is fundamentally important that we adapt and respond to those changes appropriately. To humans, language is one of the most readily available means of exchanging information about such changes. The present research investigates how people are capable of establishing and updating mental representations of object locations ‘dynamically’ in response to unfolding linguistic descriptions of events.

When we hear or read a sentence that describes an object moving from one place to another, we have to keep in memory where the object was initially, then where it is after the movement, in order to understand the event described in the sentence. For example, upon hearing that a ball will be thrown from the floor into a pond, our understanding of the sentence involves representing the starting location of the ball (on the floor) and its final destination (into the pond). This is a fundamental aspect of our understanding of the events and changes that occur around us in the world within which we live.

Our research closely investigates how we construct such representations as the sentence unfolds. For example, let us suppose we are hearing a sentence ‘the ball will be thrown into the pond.’ while looking a scene picture containing a ball on the floor and also a pond somewhere nearby. Much research in the past 10 years or so has suggested that we, as listeners, shift our attention to the referred objects as soon as we receive referring expressions (e.g., we move eye eyes towards the ball as soon as we hear ‘ball’), or even anticipate what will be mentioned later in the sentence (e.g., we look at the pond as soon as we hear ‘into the’, when the pond is the most plausible object in the scene to which the ball is thrown). However, what is not clear yet is how precisely we shift our attention between the ball and the pond. One possibility is that we never pay attention to anywhere between the ball (as it is) and the pond – after all, the ‘trajectory’ of the ball is not so relevant to ‘who did what to whom’ of the event described in the sentence. Alternatively, we can hypothesise that we actually ‘simulate’ the movement of the ball by making representations of a possible trajectory of the ball in mind.

Numerous experiments are planned in an attempt to undercover the mental simulation processes in understanding linguistic input that describes motion events. In some experiments, for example, we will ask if listeners’ attention shifts can be modulated by the shape of the trajectory of the object in motion (e.g., the ball will thrown/rolled into the pond). To do so, we use various methods to track attention shifts: we use standard eye-movement tracking techniques to explore listeners’ overt attention shifts, and other methods to capture more ‘covert’ attention shifts. We also attempt to explore how closely the linguistic input and such attention shifts can be mapped in terms of their timing. The information about the time-course is very useful to know how automatic the processes are. In other experiments, we aim to compare the processes by which people represent such motion events in spoken language and written language. Simulation of motion events in written language will provide a very important dimension to our research, as constructing spatial representations is likely to be more costly when there is no scene accompanying the linguistic input.

Our exploration of these phenomena have implications for theories of cognitive representation (how we represent the external world in our minds) and cognitive development in childhood (including children's ability to maintain multiple representations of the same object from different perspectives). It also has implications for how we conceive of the relationship between language (as used to describe events in the external world) and vision (through which we directly perceived those events), and the time-course of such processes.

Collaborators:
Dr Christoph Scheepers (Glasgow), Scott Gilmour (Glasgow)

Funding Source:
Economic and Social Research Council (ESRC), 2011-14
(452,449 (FEC) + PhD studentship jointly with Glasgow)