Poorly Designed Research and Its Outcomes
We know that conducting experiments using the scientific method is crucial for the advancement of scientific knowledge, therapies, and technology. But, when a research study is poorly designed, it can be detrimental and result in false conclusions. ​​
So, what can make a scientific study poorly designed?
1) The first of these is an inappropriate or missing theoretical framework.
This is the background information and theories that the researchers are basing their hypothesis on. Lacking this, or having one that is not relevant or related to the hypothesis, can lead to results that are not relevant to the aim of the study. To prevent this, researchers should have a hypothesis that is logically supported and based on core theories that are well-established (such as the theory of gravity).
2) The second design flaw to avoid is poor methodology, where the process and/or methods of gathering data are not well prepared or followed.
To improve this, researchers should carefully plan how they will collect data and how it will be analyzed. They must ensure that the methods used to gather information for analysis are relevant to the topic and the research questions that the study is addressing.
3) The last design flaw to avoid is a lack of specificity.
When designing an experiment (including writing the hypothesis, purpose statement, and research question) a lack of specificity or using vague qualifiers like “extremely,” “very,” or ”lightly” can be unclear and make it difficult to organize and define data. For example, quantifying things as “very ___” such as “very hot,” “very fast,” or “very tall,” is not specific enough to organize and define. Researchers can avoid this by using terms that are specific, clear, and concise such as using specific percentages or more binary (one or the other) definitions. For example, if researchers were studying different female athletes’ sprinting speeds, using the terms “fast,” “very fast,” or “extremely fast,” is too vague. Instead researchers should use specific speeds, such as “20-22 kilometres per hour,” “23-25 kilometres per hour,” and/or “26-28 kilometres per hour.”
​​​​​​Let's take a look at the following case study. Below is a hypothetical research report, let's dissect it together to determine whether the study was poorly designed.
Taking a look at the example above, are there any indicators of poor methodology?
Yes, in the study, the researchers only measured BMI as a measure of improvements in obesity after exercise and diet intervention. However, BMI is not the only parameter that indicator of progress for obesity and has been shown to be problematic due to not accounting for higher bone density in certain individuals. Researchers should not rely on only one measure as an indication of a successful intervention. Moreover, when looking at the graphs produced by the researchers, there was no explanation of how the data was analyzed and which statistical tests were used.
Hover over the box to reveal the answer...
Let’s take a look at the purpose statement in our example: “This quantitative study aims to determine if lifestyle interventions such as changing diets and exercising can improve obesity in women.”
Yes, the researchers must be more specific in their purpose statement to allow them to build a robust experiment. One thing that the researchers should specify for example is the specific types of exercise and/or diet regiment they will investigate. Will they be testing each alone and then as a combination? Which diet or exercise will they investigate? How long will participants be on the intervention during the study? More details about the ages of the women they will include in the study is also important. So, a more effective purpose statement would be: This quantitative study aims to determine if a 3-month intervention or daily low intensity exercise is effective alone or in combination with a ketogenic diet at reducing weight in women ages 20-25 with obesity.
Hover over the box to reveal the answer...
Okay, now the researchers have a more well-defined research question and they embark on their journey to answer it. Taking a look at the results they produced, do you think there are any inconsistencies with these data?
Yes, first there are many missing data points among the participants, for example, participants missing up to four weeks worth of data (see the lines with less dots!). Since there is no explanation of this inconsistency (eg. participants dropped out of study after a short time), this suggests a potential manipulation of data.
Hover over the box to reveal the answer...
When looking at any scientific findings remember to look out for the three signs of poorly designed research studies to avoid spread of misinformation and inaccurate scientific reporting. Remember that it is important for any research study to have an appropriate theoretical framework, well-reported and relevant methodology, and specific reporting of results.
About Us
Science for Everyone is a Canadian Nonprofit Organization that provides educational resources to help raise the level of scientific literacy in the general population.