POLICY AND PRACTICE Choosing evidence
Before choosing, ask three questions
Steve Fleischman explains the diffi culties in making use of research, and offers some guidance
YOU MAY BE FAMILIAR WITH RECENT NEWS that the Federal Trade Commission (FTC) – the US consumer protection agency – fi ned Reebok $25 million in September 2011 for making unsupported claims regarding its EasyTone line of athletic shoes. According to the Daily Telegraph, “Fitness and toning shoes have been claiming to work leg- sculpting miracles for several years now.” However, the FTC found these claims to be unsubstantiated. Reebok may no longer assert that the shoes tone leg muscles or produce fi rmer, shapelier buttocks. As the director of the FTC’s Bureau of Consumer Protection stated, “Consumers expect to get a workout, not to get worked over”. Unfortunately, as educators we are all too familiar with claims that turn out to be too good to be true. Take the concept of “learning styles”. Perhaps you have used the concept to guide your classroom teaching? After all, the concept and practice have been widely shared for nearly two decades. Yet, you may be surprised to learn the conclusion reached in December 2008 by experts who reviewed the research literature on the subject and found “at present, there is no adequate evidence base to justify incorporating learning-styles assessments into general educational practice”. The team wrote: [We] found virtually no evidence ...
validating the educational applications of learning styles. Although the literature on learning styles is enormous, very few studies have even used an experimental methodology capable of testing the validity of learning styles applied to education. Moreover, of those that did use an appropriate method, several found results that fl atly contradict the popular ... hypothesis. The review cautioned against the practice until better evidence is provided. However, this has not stopped publishers and developers from making large amounts of money in promoting the use of “learning styles” in schools.
20
Thus, while teachers may be protected when it comes to which shoes they wear to tone their muscles, they have little protection against choosing to use what may be ineffective practices in their classrooms.
Why is choice so diffi cult? Experts in the fi eld note that making choices is a complex, challenging, and often irrational process. In The Art of Choosing, Sheena Iyengar, a business professor at New York’s Columbia University, points to psychological, sociological, and cultural factors that shape views, attitudes, and actions related to choice. She cites examples demonstrating that in blind tests of products such as water or wine, novices often fail to distinguish the most
when there is too much information, one’s brain is overwhelmed and “a person can no longer make sense of the situation. Correlation is confused with causation, and people make theories out of coincidences.”
What type of information do “education consumers” want? If making choices is a diffi cult process, which often requires outside guidance, then what type of help would education decision makers like to receive? My colleagues at Education Northwest in Oregon, US, explored this question in a 2009 study on evidence use in education. They reported a gulf between research design and “real world” practice, which often results in fi ndings with limited applicability. Furthermore, they found that educators struggle to acquire, interpret, or apply research because of their own lack of knowledge, skills, and time.
Having access to too much information ‘can actually interfere with understanding’
expensive or “best” from the rest. As Iyengar points out, “Most of us need to rely on external information in order to choose well.” In another example, Iyengar reports on an experiment she conducted that demonstrates that “too much” choice can lead to confusion and a decrease in choice making. With the co- operation of a large supermarket she set up a booth that at different times offered tastings of either 24 or 6 of the jams produced by Wilkin & Sons. Customers who tasted the jams received a coupon allowing them to purchase their preference at a discount. Iyengar reports that “30% of the people who had seen the small assortment decided to buy jam, but only 3% bought a jar after seeing the large assortment.” Thus, having more choice can inhibit choice! In How We Decide, science journalist Jonah
Lehrer provides further examples of how our choices are often irrational or poorly understood. Lehrer points to a particularly modern challenge – having access to too much information – which he argues “can actually interfere with understanding”. He writes that
Better: Evidence-based Education winter 2012
Despite these challenges, the study points to three principles which, if followed, might support improved use of evidence in education: 1. Research should be contextualised. Participants expressed strong preferences for research evidence linked to local contexts. 2. Research should be easy to read, absorb, and apply. Participants asked that research evidence be presented in brief reports, written in non-technical language. 3. Research should be “translated” and “transmitted” by intermediaries. Participants defi ned intermediaries as trusted organisations and individuals that help to locate, sort, and prioritise available evidence.
Evidence isn’t everything: Three critical questions
The past decade has seen an increase of evidence in education. Today, “consumer- reporting” projects such as the Best Evidence Encyclopaedia (UK and US), EPPI-Centre (UK), and What Works Clearinghouse (US) provide evidence reviews of leading programmes
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28