Picturing language: Does it help or hinder?

[Originally published in January, 2006]

ResearchBlogging.orgClicking on the image below will take you to a short Quicktime movie. Make sure you have your sound turned up, because I’ve recorded a few sentences that play along with the movie. Your job is to determine, as quickly as possible, if each sentence is grammatically correct — while you focus your vision on the animated display.


This demonstration replicates part of an experiment conducted by a group of researchers led by Michael P. Kaschak. The researchers showed similar animations to a group of volunteers and asked them to make similar judgments about spoken language. The question: does our reaction time differ when the animation corresponds to the movement described in language?

In the demonstration you just tried, the first two sentences were distractors. In sentence 1, the motion was “towards,” but the animation was moving down. Sentence 2 was ungrammatical. The two sentences we’re interested in are 3 and 4. Sentence 3, “The leaves fell from the tree,” describes a downward motion, just like the motion in the animation. Sentence 4, “The balloon ascended into the clouds,” describes upward motion, opposite the animation.

Kaschak et al. have good reason to suspect that in the case of these last two sentences, the animation may indeed affect how quickly you can process the sentence. We recently reported on data that suggests that impairment of motor control of the hands may also impair our ability to visualize the same motion. Further, memory for visual objects also appears to make use of the visual system. Kaschak’s team points to other research showing that understanding sentences also involves a “sensorimotor simulation” of the action the sentence describes.

But prior to Kaschak’s team’s experiment, no one had tried to measure how quickly people process language when the motion they are viewing corresponds to an animated display. If viewing motion affects language processing, then there are two possibilities for how the two activities interact. It’s possible that watching motion that corresponds to the motion in a sentence will cause viewers to process language faster (i.e. participants will respond faster to “the leaves fell from the tree” when the animation is moving down). Another possibility is that viewing the animation burdens the same region of the brain that is needed to process the language, so when the animation is moving down, then sentences describing downward motion will be processed slower.

Kaschak et al. showed participants four different animations depicting basic motions: moving lines showing up or down motion, and a spinning spiral that could show movement towards or away from the viewer. During each animation, 10 sentences were read: 2 corresponding to the direction of the movement in the animation, 2 in the opposite direction, and 6 distractors. The test questions were always grammatically correct so that each participant was performing an equivalent task. The distractors included some nongrammatical sentences (to keep the task realistic) and some grammatical sentences describing movement that did not correspond to the animation (like Sentence 1 above).

Respondents took an average of 369 milliseconds to respond to sentences that matched the direction of the animation, but only 330 milliseconds to respond to sentences that described movement in the opposite direction. The difference was statistically significant: people take longer to process sentences that match the movement of an animation than they do to process sentences that don’t match it. Kaschak’s team reasons that we must be using the same region of the brain to process the motion itself as we do to process the language describing that motion.

Note that these results are only for animations showing a very generic sort of motion. There’s little doubt that if we saw an actual leaf falling, or balloon ascending, we’d be able to process that language very quickly. Yet the simple concept of “downward motion” does appear to distract from our ability to process a simple sentence describing a particular sort of downward motion.

We weren’t able to measure how quickly you processed the sentences in our demonstration, but did you notice anything different about trying to assess the “down” sentence compared to the “up” sentence? Let us know in the comments.

Kaschak, M.P., Madden, C.J., Therriault, D.J., Yaxley, R. J, Aveyard, M., Blanchard, A.A., & Zwaan, R.A. (2005). Perception of motion affects language processing. Cognition, 94, B79-B89.

Leave a reply