Copyright © 1993, 1999 Kevin T. Kilty, All Rights Reserved
Some evidence is better than other for establishing cause or confirming a hypothesis. The table below summarizes the value of various classes of evidence for confirming or refuting hypotheses.
| Value Level | For Refutation | For Confirmation | |
|---|---|---|---|
| Zero value | Fabrication,symbolism,gestures | Anecdotes,fabrication,symbolism,gestures | |
| Near zero | Anecdotal | Second hand testimony, data gathered for a different purpose | |
| Moderate Value | Powerful anecdote in which all surrounding circumstances are well known | Statistical evidence or direct observation under careful control | |
| Great Value | Test of a direct consequence of a clearly defined hypothesis under carefully controlled conditions | Direct observation under control with a known and understood mechanism. |
Fabricated data or anecdote. It seems so ridiculous to use fabricated evidence, that we think "how could anyone consider such a thing?" No matter how unbelievable it seems, fabrications happen. In graduate school there were students who made a calculated decision to fabricate their data. They assume that committees rarely read a thesis or dissertation carefully, that thesis advisors often don't have time to double check results, that no one will likely ever attempt to reproduce the experiment.
More obvious is the fabricated anecdote, which appears in testimony, popular articles, public speeches, and documentary. The likelihood of this fabrication being exposed is higher than fabricated research data. However, there is still small likelihood of exposure, and there are strategies for minimizing exposure. One is to use vague anecdotes that are difficult to verify. Thus, Time Magazine could report on the Seveso incident with an anecdote that a farmer watched his cat keel over. When he picked it up its tail fell off. In a later examination the exhumed cat had decayed to only a skull. This is a perfectly vague anecdote. We are told only that this was a farmer. We don't know who, we don't know if he is a real farmer, or someone who looks like a farmer. There is no hope of finding this individual and confirming the story. The story itself is alarming, because of the morbid details (no pun intended). Yet so rarely does living fabric disintegrate in this way that the story is suspicious. Of course, people who are most likely to fabricate anecdotes will, if caught, justify the fabrication with "the details are unimportant it is the symbolism that matters."
Anecdotal information is sometimes important, but moreso for refuting sweeping generalizations than confirming hypotheses. For example, an 18th century astronomer trying to establish the reality of meteorites would find French farmers' observations of rocks falling from the sky important anecdotes to refute the alleged impossibility of meteorites. However, anecdotes are nearly valueless for generalizations. We cannot depend on the anecdote representing anything typical. Anecdotes represent no sample because they offer themselves. They are unusual, biased, and selected. There is no control over their circumstances. Often we rely on the memories of witnesses, their impressions, judgements of time, space, color, and so forth, which are known to be unreliable.
Something even worse occurs with anecdotes. They achieve the status of myth and become distortions. For instance, a pilot with 16,000 hours of military flight experience takes off from a small airport. He and his passengers are never heard from again. It might be he mistook his directions and flew into a mountain face in the dark. A case of pilot error--a testament that even great pilots make mistakes. But if he was flying from Miami to Bermuda, he and his passengers are victims of the Bermuda triangle. He becomes too good to have ever made a mistake, and becomes evidence for a genuine mystery.
Second hand testimony is what we get when the actor who portrays a protagonist in an event testifies in place of the protagonist himself. I have never understood why Congress decides to take testimony from Jane Fonda on nuclear power generation; why they seek testimony from Jessica Lange and Sissy Spacek regarding the mid-1980's farm crisis, and take testimony from Jane Seymore concerning the delivery of health care. Second hand parties testify to events outside their expertise. Moreover, they suffer no consequences for being utterly wrong about any of their testimony which allows them to be utterly irresponsible as well.
Statistical evidence allows us to infer, with some possibility of incorrect inference, that a cause translates into an effect. It allows some means of controlling confounding influences. This is valuable information. Yet wrong models, unaccounted-for variables, chance variation, and the biases of investigators all serve to hide true cause and effect as well as falsely indicate cause and effect. Thus, statistical evidence is more persuasive than anecdotal, but not always greatly so, especially if the statistical association is weak.
Direct observation under control is how physics is often done. Here we can introduce some factor, under conditions of avoiding or minimizing confounding factors, and decide that a cause translates into an effect. Still missing however, is the mechanism by which the cause becomes the effect. We may still be dealing with some unknown, hidden factor.
In order to eliminate the possibility of hidden factors, or at least to identify them, we need a known mechanism by which cause leads to effect and design a specific experiment that only works if the mechanism is as we expect. Now we have valuable evidence.
An example of criteria for obtaining evidence of cause and effect are Koch's postulates. I offer a generalized version of these here.