Potential Problems With Gathering Data.
There are a number of potential problems associated with gathering data. Some of these are problems with the design of experiments; others are problems with the actual process of data gathering itself. Collecting data without appropriate controls. Every experiment must use appropriate controls. What controls are appropriate is determined by the scientific aims of the experiments. Every possible confounding variable must be controlled in order to be able to conclude that the independent variable of choice is the one producing the changes in the dependent variable. In practice, it is impossible to control every confounding variable because it is difficult to know what all these variables are, because some variables can't be controlled or because the experiments would be too complicated to perform if all the possible variables were controlled at once. In real experiments, those variables the experimenter thinks may be relevant are controlled or distributed, on average, equally between comparison groups by the use of randomization to assign "treatments" to experimental groups. Failure to control important variables is often the reason for disagreement between rival scientific labs. The most common cause for replication of other investigators' work is the belief that the original investigator's explanation for his results is incorrect because some uncontrolled variable is actually responsible. This kind of failure is usually not considered an error by most people, and it is never treated as fraud unless it can be shown that the investigator knew all along that his results were due to that variable left uncontrolled. This kind of interaction between rivals working on a common problem is considered to be part of the normal progress of science. On the other hand, every investigator should control as many possible confounding variables as he can. Repeated failure to do so can be seen by others as sloppiness, which cannot improve one's scientific reputation, or it can be seen as an attempt to defraud, even though no accusation is made publicly. Omitting controls others have pointed out. The process of research involves not only doing the laboratory work but communicating to others the results obtained and the conclusions drawn. Frequently, others will make comments and ask questions about the research. Occasionally, these comments will be in the form of criticism for omitting some crucial controls. This is part of the normal give and take between scientists. However, if the investigator recognizes the need for some control and fails to use it in subsequent experiments, he may be guilty of misconduct. As an example, take experiments done to map the representations of various areas of the skin within the cerebral cortex of a monkey. We need not concern ourselves here with the reasons for doing the research; rather, we will concentrate on the conduct of the experiments. For these experiments, microelectrode penetrations were made of the cortex, and the areas of skin stimulation of which drove neurons within that region of cortex were noted. When a penetration was completed, the electrode was moved to a new site, following a predetermined plan. The plan required orderly penetrations, moving in a systematic (not random) way across the surface of the cortex. All observations were made by a single individual or a group of individuals who knew the research plan. A "blind" observer, one who did not know the research plan or the hypothesis of the experiments, was never used. Experiments of this nature are always open to experimenter bias. The experimenter who knows the plan of the experiments is more likely to see the results according to his expectations (rather than how they really are) than a person who is naive. In our hypothetical example, the experimenter in charge of these experiments is well-aware of the potential for experimenter bias in his experiments; it has been pointed out to him repeatedly. Yet, he has never included a "blind" control in his experiments. If an experimenter knows of a potential problem and does not attempt to deal with it, he may be guilty of misconduct. Using inappropriate sample sizes. This is a complex issue that is treated in detail in a paper by Mann et al. (1991). That paper deals with the ethical issue of sample size from an animal welfare perspective, but it includes a discussion of the statistical issues regarding sample size. In brief, a very large sample creates an ethical problem because it may waste animals. On the other hand, a sample that is too small may create a different ethical problem. If the effect of the experimental treatment is large, then a statistically significant difference may be detected in spite of the too small sample. However, failure to detect a statistically significant difference may occur either because there was no difference in the first place or because the sample size was too small to detect an existing difference in the face of normal variability of the parameters measured. This may be too common a problem. One study by Freiman et al. (1978) bears upon this issue. They examined a series of 71 clinical trials which reported negative results, i.e., they reported that the treatment studied was without effect. Careful examination of these studies revealed that 50 of them could not have detected a 50% improvement in the patient had it occurred. In other words, 70% of the studies used a sample size that was too small to reliably detect the effect that they were seeking. If the investigators were unaware of this problem, they were at least guilty of a gross error made in ignorance (as pointed out above, this is not excusable). If they were aware of the problem, they were guilty of misconduct! Usually, a scientist will formulate an hypothesis, design some experiments and then collect all of the results that come from those experiments as his experimental observations. These observations form the basis of the paper that is written about the experiments. There are some investigators who use a different approach. These investigators use graduate students or postdoctoral fellows to do the actual "work" of the experiments. The students or fellows are sent into the lab with instructions to look for what the investigator wants to find. They make observations in the lab and only call the investigator when they think they have found the result he is seeking. Only those observations that the investigator actually accepts are included in the data for the paper. The investigator has, in effect, selected what he will find. It is a good way to work if you want to get only positive results, but it is unethical! The investigator has an obligation to report all the results of an experiment, not just those that support his hypothesis. Failing to see events or seeing nonexistent ones. Our expectations strongly influence what we see or feel. This is related to the problem of experimenter bias mentioned earlier. Everyone is more-or-less prone to this kind of bias; errors created by it are not necessarily examples of fraud. On the other hand, investigators have an obligation to keep their assumptions in mind as they work, to try to combat this form of experimenter bias. This problem occurs when an investigator wants so badly to find a certain result that he imagines that he sees it even though it isn't there. Of course, the opposite outcome may also occur; he may want so badly not to see a result that he misses it when it occurs. There is no way to estimate how frequently this may occur among scientists. Similarly, the whole scientific community can miss an important observation because it does not fit into the community's current world view. For example, for years it has been a common belief that neurons in the somatic sensory cortex are driven from small areas of skin only on the contralateral body surface. Because everyone "knew" that this was true, no one looked for such areas on the ipsilateral body surface. It took a new perspective from an open mind to find the existence of such areas (Towe et al., 1964). The good scientist keeps an open mind and is ever vigilant for such "unusual" observations. The following admonition of Francis Bacon may serve the student well:
"In general let every student of nature take this as a rule--that whatever his mind seizes and dwells on with peculiar satisfaction is to be held with suspicion; and that so much the more care is to be taken, in dealing with such questions, to keep the understanding even and clear." (Novum Organum, 1620) Failing to preserve data for a suitable length of time. In a study by Wolins (1962), authors of 37 papers published in psychological journals were asked to provide raw data on which their papers were based. Twenty-one replied that their data had been lost or destroyed. Only 9 were willing to provide data without being given the purpose for which the data were to be used. Of those data sets actually received, 7 were analyzed; three contained "gross errors" in statistics. Conclusions to be drawn from this study are not clear. At best, most experimenters had a cavalier attitude toward data. Something as important as raw data should be carefully preserved. Mishkin (1988) has asserted that inability (or unwillingness?) to provide primary data "should give rise to a presumption that the data do not (or never did) exist." In other words, she believes that inability to provide data suggests that the possibility of misconduct exists. There may be legitimate reasons for inability to supply data--the lab burned down. However, failure to make backup copies of data on computers or in lab notebooks is silly. The investigator should have more concern for his research materials than that. It is important here to distinguish inability to furnish data from lack of willingness to furnish for legitimate reasons. Investigators embroiled in "hot pursuit" of some experimental goal may not want their data to fall into their "competitor's" hands, afraid they may be "scooped." They may not want to give data to "uninformed" persons, afraid that the results will be misinterpreted or made public before sufficient evidence has been gathered. However, failure to provide data to granting agencies, laboratory supervisors, University officials or the like can hardly be justified on these grounds. Guidelines currently being considered by the NIH would require an investigator to maintain raw data for at least three years following publication of a study. Actually, it's a good idea to keep data as long as possible; you never know when a future observation may cause you to go back and look at data from a previous study. Where to Go From Here:Introduction
|