Multimedia and multimodality
The terms multimodal and multimedia refer to channels with multiple avenues of meaning communication via each of the senses (multimodal) realised through a range of implements, real or virtual (multimedia). The visual can include modes such as layout, images, colour and fonts, each of which contribute elements to a composite framework of meaning in any given context: am I reading a print broadsheet, tabloid or lifestyle magazine, online news reportage or blog post? Audio may cover aspects of speech, music, or clicking, beeping and other sound effects: did my mobile device just notify me a work email came in, a personal text message or a social media update? Haptic extends to touchscreen and tilt manipulation, buttons and icons, use of stylus, texture, light, and 3D or responsive sound: am I playing a fighting or building game, getting a ball in a hole or reimaging a design?
These modes combined in technological media give us 4D cinema, video-conferencing with document share, readers where you can see and hear a realistic representation of a page turn and annotate and search, and real-time, social networking practices including interactive newspages. These aspects can be combined to create whole body, or holistic, experiences that may mirror social, economic, political or personal experience and as such can be read critically. Fowler (2012) conceptualises digital literacies for critical reading in her research on email as a social practice, drawing on the New Literacy Studies (Barton and Hamilton, 2000; Hamilton, 2000; Street, 2003), and literacy in action, drawing on Actor Network Theory (Brandt and Clinton, 2002). When we read a multimedia text, is the author orchestrating resources which create meaning for us, with the effect of influencing thought and action? Am I being encouraged to read a lifestyle magazine, receive an out-of-hours email or play a character in a game as an authoritative or sensual text and change behaviour? Do I agree? If I contact a buyer or seller outside an online marketplace's communication channels, do I invalidate the dispute resolution service? In new marketplaces, the detail of established social and literacy practices changes.
This has curriculum implications for our students being able to understand, evaluate and communicate meaning in multimodal texts and interactions. To what extent do our teaching, learning and assessment practices reflect the multiplicity of meaning manipulation utilised in the world of life and work outside our doors? Pahl (2012:31), for example, explores digital 'home literacies as situated, and argues[s] that the digital is part of a wider landscape of communicative practices' in her analysis of uses of Streetview in families. Kress and van Leeuwen (2001), extend the idea of design in this field to manipulation of critique and ideological meaning within diverse texts and social habitats, highlighting the wider range of semiotic modes to select from when we communicate, concrete and abstract. This touches on themes in the field of discourse analysis with regard to underpinning social structures and hegemony. In increasingly heterogeneous, mobile societies, which social and textual patterns constitute the norms of these underpinning structures? There are ethical implications also to be considered as far as Web 2.0 is concerned, in relation to how we interact and transact with others.
We have long used realia such as leaflets or forms, and kinaesthetic activities to develop literate practice for real social purposes in the classroom; and assessment of limited aspects of multimodal meaning can be found in Functional Skills and ESOL examinations (such as layout or formatting, graphs and charts, intonation). The range of meaning-making and audience in multimedia social interaction appears even greater in size and flexibility. For example, should the results of statistical analysis for effective social impact be presented by graph or by animation? Luke (2000:71) warns of the implications for literacy:
Literacy requirements have changed and will continue to change as new technologies come on the marketplace and quickly blend into our everyday private and work lives. And unless educators take a
21
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62