Experimentation
is just trial and error. But good experiments offer the only trustworthy method
of resolving causal models.
|
What are
experiments? They are planned manipulations, or deliberate changes in the
natural associations of objects, undertaken to observe the outcome. |
|
Why use
experiments? To get rid of uncontrolled factors; they help destroy unwanted
associations in the data. With randomization, they give the best possible
controls. |
|
When to use
experiments? Experiments are appropriate when testing causal models but may
not always be feasible or ethical. |
|
What are
consequences of omitting experiments? Unwanted, confounding factors are
likely to be present in the data. The controls will not be entirely free of
problems in the absence of experiments. Experiments don't guarantee
easily-interpreted data, however. |
Experiments are
usually done to test causal models -- a manipulation to observe whether
something changes as a consequence. Consider the testing of a cancer drug. The
model tested is: the drug does (or does not) decrease the cancer rate of
patients -- a causal model. In testing this model, an experiment is a study in
which some patients are given the medicine to see whether it reduces their
cancer rates. Evaluation of this model from such an experiment obviously
requires some type of control -- baseline -- for the expected rate in the
absence of the drug, but the manipulation of giving the drug to test whether
the drug affects cancer rates is an experiment. In contrast, some manipulations
are done merely to obtain measurements -- inflating a condom to its breaking
point is a manipulation, but it is being done to test condom failure rates
rather than testing whether condom inflation causes a particular outcome.
So condom testing is not considered an experiment to test a causal model. In
general, if the intent is only to gather information about the natural
situation (unmanipulated), it is not an experiment.
Back to our
anti-cancer drug example. The alternative to an experiment is to study cancer
in people who took (or didn't take) the drug for their own reasons, such as
would be accomplished by merely putting the drug on the market and seeing
whether those peoples' cancer improved. Studies made in the absence of
experimental manipulations are often referred to as epidemiological or
correlational studies. In such an epidemiological study, there are many reasons
why cancer rates might differ between the two groups of people, none of them
having to do with the medicine. For example, those people most conscientious
about avoiding cancer (watching their diet and the like) might be the ones most
prone to take the medicine, hence we would observe a lower cancer rate in the
"drugged" group even if the medicine was ineffective.
A more obvious
example of the value of experiments involves studying the effects of alcohol on
social behavior. Letting people choose whether to drink alcohol at a party and
observing behavior as a function of alcohol consumption leads to the obvious
problem that, even in the absence of alcohol, those who choose to drink are
likely to behave differently than those who abstain. In both cases, we would
want studies in which people were deliberately assigned to a treatment or
control group, rather than giving them their choice. Other examples of
experiments are described in the following table. From these examples, it is
clear that many kinds of experiments are unethical or impractical when dealing
with humans.
|
An Experiment |
Epidemiological Observations |
|
HIV-infected
people are assigned to two groups, one group being given a drug and the other
a placebo |
Patients who
chose to and can afford to take the drug are monitored in comparison to those
who don't take the drug. |
|
People are
assigned to different "sex" groups to test condom breakage |
People are
surveyed for their experiences with condom breakage |
|
People are
assigned different occupations for 5 years to monitor cancer rates |
Cancer rates
are monitored in people who have pursued different careers |
|
You feed your
children sugar at predetermined times to observe changes in their behavior |
You observe
changes in your children's behavior according to whether they chose to eat
sugar or not |
When
experiments are performed, individuals are often assigned to different groups
randomly, and it may seem that randomization implies experimentation. It is
possible, however, to select subsets of data randomly in epidemiological
studies (in which no manipulation of nature is performed), so these two
features are distinct. But, as we will note below, randomization is virtually
an essential component of experimental studies if the study is to be maximally
useful.
Fortuitous
experiments.
Although deliberate, planned experiments offer the best source of data, there
is a category of manipulation that falls between them and pure epidemiological
observations. These intermediates are human-caused changes in the natural
order, but without intent to test a particular model. The subjects of
fortuitous experiments are subsequently observed for the effects of this
manipulation.
There is a
famous set of fortuitous experiments in the people exposed to large doses of
radiation: the Japanese survivors of atomic bombs, U.S. soldiers involved in
field tests of atomic weapons, certain occupational exposures, and medical exposures.
In all cases, there was no intent to study the long-term effects of radiation
on cancer, so these manipulations do not constitute proper experiments to study
these models. Yet these exposures are far above any exposures that would occur
through natural variation in background radiation levels, so we can think of
them as a kind of experiment (deliberate modifications of the natural order)
but without intent to study cancer. In the case of U.S. soldiers, the
distinction becomes even more subtle, because these soldiers were sent in to
detonation sites partly to observe the short-term effects of radiation (and the
soldiers wore film badges to monitor their exposures), but the study was not
planned to look at cancer.
Observations from fortuitous experiments tend to be uncommon, probably
because we don’t often accidentally create a situation that is so easily
regarded as a manipulation to test a model.
Experiments,
randomization, and controls. The main benefit of a well-designed experiment is
to produce a good control. By randomizing assignments between treatment and
control groups, virtually all confounding factors are eliminated, and any final
difference between control and treatment group should be due to the treatment.
Randomization is the only way to ensure this elimination of confounding
factors, but various pseudo-random methods may work just as well (e.g.,
assigning people by the second letter of their last name). Randomization must
be performed over the relevant factors and involve a large enough sample size
for the data to be interpretable, however.
As we have
indicated, a correlation may or may not be due to causation. Experimental
manipulation provides one way of discovering causation, but when a full-blown
experimental test is not feasible, there is another way that helps resolve
causation. This method merely involves delving deeper into the basis of the
correlations to trace causal chains. That is, correlations are usually detected
at a superficial level, and merely looking at the underlying mechanisms
producing the correlation may indicate whether the correlation is causal. In
the nuclear power plant example, one approach might have been to assess whether
levels of radiation were indeed higher around the plants than away from the
plants. The difficulty here, of course, is that a nuclear power plant might be
a cause of cancer for other reasons, and the measurement of radiation levels
would not detect other causes of cancer.
The approach of
partitioning a correlation into component mechanisms has worked in some cases,
one of them being the correlation between diet and heart disease. People with
high fat diets die of heart disease more often than do people with low fat
diets. Yet these data do not demonstrate that high fat diets actually cause
heart disease. It could be, for example, that people with high fat diets also
have diets high in protein, and that protein, not fat, causes heart disease.
Additional data
show that high fat diets really do cause heart disease. Experiments and other
evidence show that each step in the causal chain
high fat
diets--->high blood cholesterol---> hardening of the arteries--->heart
disease
is likely correct. Scientists have done experiments showing that when dietary
fat increases, so does blood cholesterol. These experiments did not show
directly that high-fat diets cause heart disease, but they did contribute to
showing that mechanisms are present that can cause the correlation. Each step
in the above chain has similar evidence supporting it.
The method of
investigating the cause of correlations can be applied in many other
situations. Some years ago consumer interest groups observed a correlation
between Pintos and death rates in car accidents: people riding in Pintos were
burned more frequently than were people involved in accidents with other types
of cars. In this case it was shown that the problem was the Pinto itself,
rather than the possibility that Pinto drivers were somehow the cause. The
finding that the gas tank in Pinto's often exploded after a rear-end collision
was the critical evidence establishing the cause of the high death-by-fire
rates suffered by Pinto drivers.
Copyright 1996-2000 Craig M. Pease &
James J. Bull