Multimodalities

In Siegel’s (2006) article, semiotics is defined as “a broad field of studies that looks at meanings and messages in all their forms and all their contexts.” (p.68) That is, it gives the same weights to all sign systems, be it languages, codes, symbols, pictures, gestures, and so forth. This discipline somehow pinpoints the importance of taking into account multimodality when it comes to literacy development. More often than not, literacy learning at school is characterized by monomodality; that is, to be literate is to display the language required for successful participation in schooling. However, language is all but one sign system/modality capable of enabling learners to construct meanings. Other modalities such as drawings, dramatic play, and 3-D constructions also abound to allow students to get across intended meanings and thus those proficient in these modalities should also be considered literate. As I recall, my 4-year-old niece once showed me a drawing that according to her portrayed her family members, though to me there were nothing more than 4 identical sets of symbols on the paper, each consisting of a circle on top, a triangle in the middle, and 2 ovals in the bottom. Yet, to come to think of it, this drawing indeed demonstrated that my niece had established the concept in regard to family members as well as an artistic skill as a way to convey this concept (a modality), though the skill still left a lot to be desired. However, in light of the multimodality theory noted here, she should be thought of as being literate, now that she was equipped with a sign system to help her “paint” her mind. I think this drawing example also serves as a piece of evidence lending support to Siegel’s (2006) proposition that children come into the classroom filled with prior knowledge in terms of multimodalities which, when tapped, would contribute to position the children as successful meaning makers.

I also find the idea transmediation really interesting. As Suhor (1984) put it, transmediation refers to the “translation of content from one sign system into another.” (p. 250) And this movement across sign systems emerges as a generative process giving rise to new meanings, due to the fact that there exists no shared code to help represent both sign systems and thus the connection between them needs to be invented. As I rolled my eyes through this idea of transmediation, I realized that it actually underlied some of the activities that I administered in my EFL classrooms and rendered them both fruitful and enjoyable. To take an example, I used to engage students in the story telling activity in which they looked at a comic strip with the bubbles emptied, collectively creating the lines they saw fit for each bubble, and then narrated a story taking place in the comic strip while at the same time presenting to the class their comic strip with all the bubbles filled. That is too say, this activity required students to move across 2 sign systems (modalities), that is, from pictures to oral discourse. And almost always the oral language students managed to produce in response to the comic strips was a lot richer than what they generated for the regular story telling task without any visual aids. On top of that, most students expressed fondness for this comic-turned-story activity because they found it a lot of fun to create dramatic dialogues for the comic characters (instead of themselves). As such, I believe transmediation indeed possesses great generative power that, when practiced appropriately, would lead to new meanings and come with a lot of joy. Picture 1 and 2 below serve as fine pieces of material for the activity in question.















Picture 1* The comic strip with empty bubbles















Picture 2* The original comic strip

*This comic strip comes from http://www.dilbert.com/comics/dilbert/archive/dilbert-20070318.html

As Halliday (1978) contended, “language as a social semiotic means interpreting language within a sociocultural context in which the culture itself is interpreted in semiotic terms.” (p.2) This statement reminds me of how sounds animals make seem to differ from culture to culture (onomatopoeia). Of course, it is common sense that all dogs on the face of the earth bark in the same way. Yet, people with different cultural memberships had come to mimic the barking in different ways. Here’s a list of animal sounds represented in English and Chinese. As can be seen, roosters cocka-a-doodle-doo or gu-gu-gu depending on where they are. As far as I am concerned, this, among other facts, aptly illustrates how meanings of language/concepts/values are situated in the sociocultural context.

Animals -> English -> Chinese
dogs -> woof woof -> wan wan
cats -> meow -> meow
roosters -> cocka-a-doodle-doo -> gu-gu-gu
pigs -> oink oink -> go-go
ducks -> quack -> gua gua
cows -> moo -> mo

A guy made some crazy and funny animal sounds in the video clip that follows. It totally cracked me up! Check it out!



* This video clip comes from YouTube.