CHAPTER
,..
:I
I
i
8
once went to a good deal of trouble to lind out
if cigarette smokers make lower college grades than nonsmokers. It turned out that they did. This pleased a good
many people and they have been making much of it ever
since. The road to good grades, it would appear, lies in
giving up smoking; and, to carry the conclusion one
reasonable step further. smoking makes dull minds.
This particular study was, I believe, properly done:
sample big enough and honestly and carefully chosen,
correlation having a high Significance, and so on.
The fallacy is an ancient one which, however, has a
powerful tendency to crop up in statistical material, where
it is disguised by a welter of impressive figures. It is the
one that says that if B follows A, then A has caused B.
SOMEBODY
81
88
BOW TO LIE WITII STATISTICS
An unwarranted assumption is being.made that since
smoking and low grades go together, smoking causes low
grades. Couldn't it just as well be the other way around?
Perhaps low marks drive students not to drink but to to-bacco. When it comes right down to it. this conclusion is
about as likely as the other and just as well supported by
the evidence. But it is not nearly so satisfactory to propa·
gandists.
It seems a good deal more probable, however, that
neither of these things has produced the other, but both
are a product of some third factor. Can it be that the
sociable sort of fellow who takes his books less than seri·
ously is also likely to smoke more? Or is there a clue :in
the fact that somebody once established a correlation J.
tween extroversion and low grades-a closer relationship
POST HOC RIDES AGAIN
apparently than the one between grades and intelligence?
Maybe extroverts smoke more than introverts. The point
is that when there are many reasonable explanations you
are hardly entitled to pick one that suits your taste and
insist OD it. But many people do.
To avoid falling for the post hoc fallacy and thus wind
up believing many things that are not so, you need to put
any statement of relationship through a sharp inspection.
The correlation, that convincingly precise figure that seems
to prove that something is because of something, can acLually be any of several types.
One is the correlation produced by chance. You may
be able to get together a set of figures to prove some un-Jilcely thing in this way, but if you try again, your nen
set may not prove it at all. .As with the manufacturer of
the tooth paste that appeared to reduce decay, you simply
throwaway the results you don't want and publish widely
those you do. Given a small sample, you are likely to find
some substantial correlation between any pair of characteristics or events that you can think of.
A common kind of co-variation is one in which the relationship is real hut it is not possible to be sure which of
the variables is the cause and which the effect. In some
of these instances cause and effect may change places
from time to time Or indeed both may be cause and effect
at the same time. A correlation between income and
ownership of stocks might be of that kind. The more
money you make, the more stock you buy. and the more
stock you buy, the more income you get; it is not accurate
HOW TO LIE WITH STATISTICS
to say simply that one has produced the other.
Perhaps the trickiest oE them all is the very common
instance in which neither of the variables has any effect
at all on the other, yet there is a real correlation. A good
deal of dirty work has been done with this one. The poor
grades among cigarette smokers is in this category. as are
all too many medical statistics that are quoted without
the qualification that although the relationship has been
shown to be real, the cause-and-effect nature of it is only
a matter of speculation. .As an instance of the nonsense
or spurious correlation that is a real statistical fact, sOmeone has gleefully pointed to this: There is a close relationship between the salaries of Presbyterian ministers in
Massachusetts and the price of rum in Havana.
Which is the cause and which the effect? In other
words. are the ministers benefiting from the rum trade or
supporting it? All right. That's so farfetched that it is
ridiculous at a glance. But watch out for other applica~
tions of post hoc logic that differ from this one only in being more subtle. In the case of the ministers and the rum
it is easy to see that both figures are growing because of
the influence of a third factor: the historic and world-wide
rise in the price level of practically everything.
And take the figures that show the suicide rate to be
at its maximum in June. Do suicides produce June brides
-or do June weddings precipitate suicides of the jilted? A
somewhat more convincing (though equally unproved)
explanation is that the fellow who licks his depreSSion aU
through the winter with the thought that things will look
POST HOC RIDES AGAIN
rosier in the spring gives up when June comes and he still
feels terrible.
Another lhiug to watch out for is a cunclusiun in which
a correlation has been inferred to continue beyond the
data with which it has been demonstrated. It is easy to
sho)V that the more it rains in an area, the taller the corn
grows or even the greater the crop. Rain, it seems, is a
blessing. But a season of very heavy rainfall may damage
or even ruin the crop_ The positive correlation holds up to
a point and then quickly becomes a negative one. Above
so·many inches, the more it rains the less com you get.
We're going to pay a little attention to the evidence on
the money value of education in a minute. But for now
let's assume it has been proved that high-school graduates
make more money than those who drop out, that each
year of undergraduate work in college adds some more income. Watch out for the general conclusion that the more
you go to school the more money you'll make. Note that
this has not been shown to be true for the years beyond
an undergraduate degree, and it may very well not apply
to them either. People with Ph.D.s quite often become
HOW TO LIE WlTB STATISTICS
college teachers and so do not become members of the
highest income groups.
A correlation of course shows a tendency which is not
often the ideal relationship described as one-to-one. Tall
boys weigh more than short boys on the average, so this
is a positive correlation. But you can easily find a sixfooter who weighs less than some Bve-footers, so the correlation is less than 1. A negative correlation is simply a
statement that as one variable increases the other tend~
to decrease. In physics this becomes an inverse ratio:
The further you get from a light bulb the less light there
is on your book; as distance increases light intensity de-
creases. These physical relationships often have the kindness to produce perfect correlations, but figures from
business or sociology or medicine seldom work out so
neatly. Even if education generally increases incomes it
POST HOC IIIDES AGAIN
93
may easily tum out to be the financial ruination of Joe over
there. Keep in mind that a correlation may be real and
based on real cause and eHect-and still be almost worthless in determining action in any single case.
Reams of pages of figures have been collected to show
the value in dollars of a college education, and stacks of
pamphlets have been published to bring these 6guresand conclusions more Or less based on them-to the attention of potential students. I am not quarreling with the
intention. I am in favor of education myself, particularly
if it includes a course in elementary statistics. Now these
figures have pretty conclusively demonstrated that people
who have gone to college make more money than people
who have not. The exceptions are numerous, of course,
but the tendency is strong and clear.
The only thing wrong is that along with the figures and
facts goes a totally unwarranted conclusion. This is the
post hoc fallacy at its best. It says that these 6gures show
that if you (your son, your daughter) attend college you
will probably earn more money than if you decide to
spend the next four years in some other manner. This unwarranted conclusion has for its basis the equally unwarranted assumption that since college-trained folks make
more money, they make it because they went to college.
Achlally we don't know but that these are the people who
would have made more money even if they had not gone
to college. There are a couple of things that indicate
rather strongly that this is so. Colleges get a disproportionate number of two groups of kids: the bright and the
BOW TO LIE WITH STATISTICS
rich. The bright might show good earning power without
college knowledge. And as for the rich ones ... well,
money breeds money in several obvious ways. Few sons
of rich men are found in low-income brackets whether
they go to college or not.
The following passage is taken from an article in question-and-answer form that appeared in This Week magazine, a Sunday supplement of enonnous circulation.
Maybe you will find it amusing, as I do, that the same
writer once produced a piece called "Popular Notions:
True or Falser"
Q: What effect does going to coDege have on your chances
of remaining unmarried?
A: If you're a woman, it skyrockets your chances of becoming an old maid. But if you're a man. it has the opposite effect
-it minimizes your chances of staying a bac11elor.
Cornell University made a study of 1,500 typical middleaged college graduates. Of the men, 93 per cent were married (compared to 83 per cent for the general population).
But of the middle-aged women graduates only 65 per cent
were married. Spinsters were relatively three times as numer-
ous among college graduates as among women of the general
population.
When Susie Brown, age seventeen, reads this she learns
that if she goes to college she will be less likely to get a
man than if she doesn't. That is what the article says, and
there are statistics from a reputable source to go with it.
They go with it, but they don't back it up; and note also
that while the statistics are Cornell's the conclusions are
POST HOC RIDES AGAlN
95
not, although a hasty reader may come away with the idea
that they are.
Here again a real correlation has heen used to bolster up
an unproved cause-and-effect relationship. Perhaps it all
works the other way around and those women would have
remained unmarried even if they had not gone to college.
Possibly even more would have failed to marry. If these
possibilities are no better than the one the writer insists
upon, they are perhaps just as valid conclusions: that is,
guesses.
Indeed there is one piece of evidence suggesting that
a propensity for old-maidhood may lead to going to college. Dr. Kinsey seems to have found some correlation
between sexuality and education, with traits perhaps being
fixed at pre-college age. That makes it all the more questionahlf' to say that going to college gets in the way of
marrying.
Note to Susie Brown: It ain't necessarily so.
A medical article once pointed with great alarm to an
increase in cancer among milk drinkers. Cancer, it seems,
was becoming increasingly frequent in New England.
Minnesota, 'Wisconsin, and Switzerland, where a lot of
milk is produced and consumed, while remaining rare in
Cerlon, where milk is scarce. For further evidence it was
pointed out that cancer was less frequent in some Southern
states where less milk was consumed. Also, it was pointed
out, milk-drinking English women get some kinds of cancer eighteen times as frequently as Japanese women who
seldom drink milk.
HOW TO LIE WITH STATISTICS
A little digging might uncover quite a number of ways
to account for these figures, but one factor is enough by
itself to show them up. Cancer is predominantly a disease
that strikes in middle life or after. Switzerland and the
states mentioned first are alike in having populations with
relatively long spans of life. English women at the time
the study was made were living an average of twelve
years longer than Japanese women.
Professor Helen M. Walker has worked out an amusing
illustration of the foUy in asswning there must be cause
and effect whenever two things vlllJ' together. In iuvesU-
.J
.',
J)
., ,.J(
\
POST HOC RWES AGAIN
gating the relationship between age and some physical
characteristics of women, begin by measuring the angle of
the feet in walking. You will find that the angle tends to
be greater among older women. You might first consider
whether this indicates that women grow older because
they toe out, and you can see immediately that this is
ridiculous. So it appears that age increases the angle be.
tween the feet, and most women must come to toe out
more as they grow older.
Any such conclusion is probably false and certainly unwarranted. You could only reach it legitimately by studying the same women-or possibly equivalent groups~ver
a period of time. That would eliminate the factor responSible here. Which is that the older women grew up at
a time when a young lady was taught to toe out in walking, while the members of the younger group were learning posture in a day when that was discouraged.
When you find somebody-usually an interested party
-making a fuss about a correlation, look first of all to see
if it is not one of this type, produced by the stream of
events, the trend of the times. In our time it is P.:Jsy to
show a positive correlation between any pair of things like
these: number of students in college, number of inmates
in mental institutions, consumption of cigarettes, incidence
of heart disease, use of X-ray machines, production of
false teeth. salaries of California school teachers, profits
of Nevada gambling halls. To call some one of these the
cause of some other is manifestly silly. But it is done
every day.
HOW TO LIE 'WITH STATISTICS
Permitting statistical treatment and the hypnotic presence of numbcrs and decimal points to befog causal relationships is little bettcr than superstition. And it is often
more seriously misleading. It is rather like the conviction
amon~ the people of the New Hebrides that body lice produce good health. Observation over the centuries had
taught them that people in good health usually had lice
and sick pe(lple very often did not. The observation itself
was accurate and sound, as observations made informallv
over the years surprisingly often are. Not so much can be
said for the conclusion to which these primitive people
came from their evidence: Lice make a man healthy.
Everybody should have them.
~
POST HOC RIDES AGAIN
99
As we ha"e already noted, scantier evidence than this-
treated in the statistical mill until commOll sense could no
longer penetrate to it-has made many a medical fortune
and many a medical article in magazines, including professional ones. More sophisticated observers finally got
things straightened out in the New Hebrides. As it turned
out, almost everybody in those circles had lice most of the
time. It was, you might say, the nonnal condition of man.
When, however, anyone took a fever (quite possibly carried to him by those same lice) and his body became too
hot for comfortable habitation, the lice left. There you
have cause and effect altogether confusingly distorted.
reversed, and intemlingled.
Purchase answer to see full
attachment