You may recall the breathless reportage: a recent study showed that sponges were teeming with bacteria, and disinfection by microwaving, boiling or running the sponge through a dishwasher cycle might actually increase the number of pathogenic bacteria.
Except, the conclusions drawn from the study were misleading at best, and likely inaccurate when extrapolated to the world at large.
A quick trip to the methodology section of the study revealed that the research was based on a sample of 14 used sponges collected from households in the area of Villingen-Schwennignen, Germany.
Users were asked to specify “whether they regularly apply special measures to clean their sponge". The measures mentioned were "heating in a microwave" and "rinsing with hot, soapy water.”
Where should we start?
- Fourteen sponges cannot be considered a representative sample.
- Is a “user” the only person who cleans the sponge, or the person who happened to answer the survey?
- Did the sponges have special characteristics, such as a specific composition or density?
- What does the term “regularly” mean?
- What does the method “rinsing with hot soapy water” mean – is it rinsing the sponge under the faucet or running it through a full dishwasher or laundry cycle with detergent?
- Did respondents answer honestly or did they fudge their responses to avoid the fear of being judged on their cleaning efforts?
The problem was that the results were then used to draw general conclusions about sponges. For example, “However, our data showed that regularly sanitized sponges (as indicated by their users) did not contain less bacteria than uncleaned ones…. From a long term perspective, sponge sanitation methods appear not sufficient to effectively reduce the bacterial load in kitchen sponges and might even increase the shares of RG2-related bacteria.”
Later in the month the media, including the New York Times, walked back some of the coverage and pointed out potential shortcomings of the study.
What should we take away from this experience?
- Know the limitations of your study. It is virtually impossible to draw firm conclusions from data based on small sample sizes like this one, unless the universe you’re trying to extrapolate to is small and accurately represented by the sample.
- Understand potential sources of bias and try to reduce them when possible.
- Plan your study from two directions, first by outlining your research objectives and strategies, and then by developing an analysis plan and working backward. Then focus on the methodology.
- Be careful about the conclusions you publish…
- … and where they're publicized.
Or simply follow the advice of General Turgidson when he said, “I'd like to hold off judgment on a thing like that, sir, until all the facts are in”.