Chapter 21: Experiments make the best controls

Experimentation is just trial and error. But good experiments offer the only trustworthy method of resolving causal models.

Experiments

What are experiments? They are planned manipulations, or deliberate changes in the natural associations of objects, undertaken to observe the outcome.

Why use experiments? To get rid of uncontrolled factors; they help destroy unwanted associations in the data. With randomization, they give the best possible controls.

When to use experiments? Experiments are appropriate when testing causal models but may not always be feasible or ethical.

What are consequences of omitting experiments? Unwanted, confounding factors are likely to be present in the data. The controls will not be entirely free of problems in the absence of experiments. Experiments don't guarantee easily-interpreted data, however.

 

Experiments are usually done to test causal models -- a manipulation to observe whether something changes as a consequence. Consider the testing of a cancer drug. The model tested is: the drug does (or does not) decrease the cancer rate of patients -- a causal model. In testing this model, an experiment is a study in which some patients are given the medicine to see whether it reduces their cancer rates. Evaluation of this model from such an experiment obviously requires some type of control -- baseline -- for the expected rate in the absence of the drug, but the manipulation of giving the drug to test whether the drug affects cancer rates is an experiment. In contrast, some manipulations are done merely to obtain measurements -- inflating a condom to its breaking point is a manipulation, but it is being done to test condom failure rates rather than testing whether condom inflation causes a particular outcome. So condom testing is not considered an experiment to test a causal model. In general, if the intent is only to gather information about the natural situation (unmanipulated), it is not an experiment.

Back to our anti-cancer drug example. The alternative to an experiment is to study cancer in people who took (or didn't take) the drug for their own reasons, such as would be accomplished by merely putting the drug on the market and seeing whether those peoples' cancer improved. Studies made in the absence of experimental manipulations are often referred to as epidemiological or correlational studies. In such an epidemiological study, there are many reasons why cancer rates might differ between the two groups of people, none of them having to do with the medicine. For example, those people most conscientious about avoiding cancer (watching their diet and the like) might be the ones most prone to take the medicine, hence we would observe a lower cancer rate in the "drugged" group even if the medicine was ineffective.

A more obvious example of the value of experiments involves studying the effects of alcohol on social behavior. Letting people choose whether to drink alcohol at a party and observing behavior as a function of alcohol consumption leads to the obvious problem that, even in the absence of alcohol, those who choose to drink are likely to behave differently than those who abstain. In both cases, we would want studies in which people were deliberately assigned to a treatment or control group, rather than giving them their choice. Other examples of experiments are described in the following table. From these examples, it is clear that many kinds of experiments are unethical or impractical when dealing with humans.

An Experiment

Epidemiological Observations

HIV-infected people are assigned to two groups, one group being given a drug and the other a placebo

Patients who chose to and can afford to take the drug are monitored in comparison to those who don't take the drug.

People are assigned to different "sex" groups to test condom breakage

People are surveyed for their experiences with condom breakage

People are assigned different occupations for 5 years to monitor cancer rates

Cancer rates are monitored in people who have pursued different careers

You feed your children sugar at predetermined times to observe changes in their behavior

You observe changes in your children's behavior according to whether they chose to eat sugar or not

When experiments are performed, individuals are often assigned to different groups randomly, and it may seem that randomization implies experimentation. It is possible, however, to select subsets of data randomly in epidemiological studies (in which no manipulation of nature is performed), so these two features are distinct. But, as we will note below, randomization is virtually an essential component of experimental studies if the study is to be maximally useful.

Fortuitous experiments. Although deliberate, planned experiments offer the best source of data, there is a category of manipulation that falls between them and pure epidemiological observations. These intermediates are human-caused changes in the natural order, but without intent to test a particular model. The subjects of fortuitous experiments are subsequently observed for the effects of this manipulation.

There is a famous set of fortuitous experiments in the people exposed to large doses of radiation: the Japanese survivors of atomic bombs, U.S. soldiers involved in field tests of atomic weapons, certain occupational exposures, and medical exposures. In all cases, there was no intent to study the long-term effects of radiation on cancer, so these manipulations do not constitute proper experiments to study these models. Yet these exposures are far above any exposures that would occur through natural variation in background radiation levels, so we can think of them as a kind of experiment (deliberate modifications of the natural order) but without intent to study cancer. In the case of U.S. soldiers, the distinction becomes even more subtle, because these soldiers were sent in to detonation sites partly to observe the short-term effects of radiation (and the soldiers wore film badges to monitor their exposures), but the study was not planned to look at cancer.  Observations from fortuitous experiments tend to be uncommon, probably because we donít often accidentally create a situation that is so easily regarded as a manipulation to test a model.

Experiments, randomization, and controls. The main benefit of a well-designed experiment is to produce a good control. By randomizing assignments between treatment and control groups, virtually all confounding factors are eliminated, and any final difference between control and treatment group should be due to the treatment. Randomization is the only way to ensure this elimination of confounding factors, but various pseudo-random methods may work just as well (e.g., assigning people by the second letter of their last name). Randomization must be performed over the relevant factors and involve a large enough sample size for the data to be interpretable, however.

Dissecting correlations into manageable pieces

As we have indicated, a correlation may or may not be due to causation. Experimental manipulation provides one way of discovering causation, but when a full-blown experimental test is not feasible, there is another way that helps resolve causation. This method merely involves delving deeper into the basis of the correlations to trace causal chains. That is, correlations are usually detected at a superficial level, and merely looking at the underlying mechanisms producing the correlation may indicate whether the correlation is causal. In the nuclear power plant example, one approach might have been to assess whether levels of radiation were indeed higher around the plants than away from the plants. The difficulty here, of course, is that a nuclear power plant might be a cause of cancer for other reasons, and the measurement of radiation levels would not detect other causes of cancer.

The approach of partitioning a correlation into component mechanisms has worked in some cases, one of them being the correlation between diet and heart disease. People with high fat diets die of heart disease more often than do people with low fat diets. Yet these data do not demonstrate that high fat diets actually cause heart disease. It could be, for example, that people with high fat diets also have diets high in protein, and that protein, not fat, causes heart disease.

Additional data show that high fat diets really do cause heart disease. Experiments and other evidence show that each step in the causal chain

high fat diets--->high blood cholesterol---> hardening of the arteries--->heart disease


is likely correct. Scientists have done experiments showing that when dietary fat increases, so does blood cholesterol. These experiments did not show directly that high-fat diets cause heart disease, but they did contribute to showing that mechanisms are present that can cause the correlation. Each step in the above chain has similar evidence supporting it.

The method of investigating the cause of correlations can be applied in many other situations. Some years ago consumer interest groups observed a correlation between Pintos and death rates in car accidents: people riding in Pintos were burned more frequently than were people involved in accidents with other types of cars. In this case it was shown that the problem was the Pinto itself, rather than the possibility that Pinto drivers were somehow the cause. The finding that the gas tank in Pinto's often exploded after a rear-end collision was the critical evidence establishing the cause of the high death-by-fire rates suffered by Pinto drivers.

 

Table of contents

Problems

Copyright 1996-2000 Craig M. Pease & James J. Bull