“The number of undergraduate students impossible to answer definitely, but nevertheless
at Cal Poly and the University of Georgia they are essential for formulating relevant compari-
choosing statistics as a major or minor has sons, designing meaningful studies, and guarding
steadily increased since the advent of AP against what I called incentive bias (Meng, 2009),
Statistics, with many students attesting that is, humans’ tendency, however subconsciously,
that their choice was based largely on of selectively collecting and presenting evidence that
their positive experience in AP Statistics. support one’s causes. Minimally, it reminds us that
Similarly, the number of students enrolling given that the general demand for statistics has been
in statistics courses at Kenyon College increasing rather dramatically, especially in recent
has increased substantially over this time years, any type of increase in enrollments over the
as well, with many students citing AP years itself cannot be taken as scientific evidence of
Statistics as the reason for their interest.” the effectiveness of a particular program intended to
attract students to statistics.
Putting aside the issue that different institutions
To see this clearly, let us use a generic binary “in”
are involved, the students quoted in RPFHS belong
and “out” variable; here “in” can mean to take a sta-
to the complement of A, A
C
. We statisticians under-
tistical course or to major in statistics, or some other
stand well that conditioning on A and condition-
outcome. Regardless of its actual meaning, Table 1
ing on A
C
are different matters, just as association
is applicable to any program designed to attract
(over time) and causation are not the same thing.
membership.
Therefore, the intended or unintended uses of such
mixed arguments in RPFHS puzzled me.
There is of course no puzzle whatsoever in the
fact that an education program can have both sig-
In Out
nificant positive and negative impact, depending on Would P
ALWAYS
P
OFF
P
WOULD
how it is actually implemented. When an AP statis-
tics course or any other introductory course is done
well, such as those described in RPFHS, it does
Would Not P
ON
P
NeVeR
P
NOT
a great service to our profession. When it is done
poorly, as experienced by some Harvard undergrad- P
IN
P
OUT
uates, it has a strong effect but in the opposite direc-
tion. Therefore, the evidence from these different
observations actually reinforce the same point: the
Table 1—The row and column variables represent before
first courses in statistics, regardless whether at the
and after “the treatment”
high school level or the college level, are absolutely
critical and we need substantially more passionate
and skilled educators in order to maximize their
Here P
ALWAYS
and P
NeVeR
are respectively the
positive impact. In other words, the worry discussed
percentages of students (at time t) who will enter
below is not about the AP program per se, which
and not enter statistics regardless of whether our
of course has increased awareness of statistics at the
program is in place or not; P
OFF
is the percentage of
high school level by putting statistics on an equal
students who would enter statistics but got turned
footing with many other subjects. The worry is sim-
off by the program, and P
ON
is the percentage of
ply due to the severe shortage of qualified statistical
students who would not enter statistics but got
teachers who can teach introductory courses in such
turned on by the program. From this setup, con-
a way to arouse students’ interests in statistics, or at
ceptually it is clear that observing P
IN
= P
ALWAYS
least not to turn them away from statistics.
+ P
ON
, or even directly observing P
ON
, large or
even increase over time, says little about the overall
The Off-Diagonal Paradox: Do We
effectiveness of the program because P
Turn On More than We Turn Off?
OFF
can also be
large and even increase over time, which can offset
With the evidence of the non-emptiness of both A the gain by P
ON
whenever P
OFF
> P
ON
.
and A
C
, a scientific assessment of the overall effec- This, of course, is trivial arithmetic. But just as
tiveness of any program such as AP statistics then Simpson’s paradox can be explained by trivial arith-
must ask, minimally, has the program attracted more metic, yet has led to numerous erroneous conclu-
students to statistics than if it were not in place? This is sions throughout the history of quantitative investi-
squarely a causal inference question, one that is argu- gations, it is easy for us to focus on the “In” column
ably as hard as—and therefore needs to be addressed because it is the population most easy to identify
as carefully as—“does smoking cause lung cancer?” and sample from (as in RPFHS), and arrive at asser-
Counterfactual causal questions as such are often tions of the benefit of the program while it actually
8 AmstAt News December 2009
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76 |
Page 77 |
Page 78 |
Page 79 |
Page 80 |
Page 81 |
Page 82 |
Page 83 |
Page 84 |
Page 85 |
Page 86 |
Page 87 |
Page 88 |
Page 89 |
Page 90 |
Page 91 |
Page 92 |
Page 93 |
Page 94 |
Page 95 |
Page 96 |
Page 97 |
Page 98 |
Page 99 |
Page 100