Evaluating Limitations to a Research Study
Research is everywhere - from the medical studies we rely on for health decisions, to the scientific breakthroughs shaping our future. But, how can we know if the results of a study are truly reliable, or if they are influenced by design flaws or biases? In this module, we will dive into the complexities of research design to help you understand what data from a research study actually tells us (and what it does not). We'll also explore how to evaluate a study's trustworthiness and applicability, and how to spot potential study design flaws.
1. Scope and Applicability of Data
Every research study is designed to answer specific questions. The scope of a study refers to the boundaries of what the study was designed to investigate. The data from the study can only provide direct answers to the questions within its boundaries. It is important to not make broad assumptions based on these results. It is also worth considering whether the findings are useful to your local population or area of focus.
For example, if a study examines the effects of a medication in healthy adults, its results should not be applied to children, older adults, or individuals with other health conditions unless those groups were included in the study.
2. Validity and Generalizability of Results
Internal validity refers to whether the study was designed, conducted, and analyzed in a way that allows us to draw trustworthy conclusions about cause and effect.
There are a few types of internal validity to consider when designing or evaluating a research study:
This happens when participants alter their behaviour simply because they know they are being observed.
​
Example: In a workplace productivity study, employees might work harder if they know they’re being watched.
3. Experimental Variables
Control variables are factors that researchers intentionally keep constant across all conditions to prevent them from affecting the results.
Example: Imagine you’re conducting a study on the effect of a new exercise routine on weight loss. Which of the following could be a control variable? A. The type of exercise routine used. B. The participants’ age and gender. C. The amount of sleep each participant gets. Answer: B. Participants’ age and gender should be controlled because they could influence weight loss independently of the exercise routine.
Confounding variables are factors that influence both the independent and dependent variables, making it difficult to determine a cause-and-effect relationship.
Example: If you’re studying the relationship between coffee consumption and productivity, but you do not control for sleep quality, your results could be influenced by how well participants slept. How would you control for a confounding variable such as sleep quality in your study? A. Conduct the study at the same time each day to ensure consistency in sleep. B. Only recruit people who sleep 7-8 hours per night. C. Both A and B. Answer: C. Both A and B are valid strategies to control for sleep quality in your study. By standardizing the study time, researchers minimize fluctuations in alertness that could be due to sleep timing rather than coffee consumption, and by selecting participants with a consistent and optimal sleep range (7-8 hours), the study ensures that sleep quality is relatively uniform across participants.
So, what can we do to exclude or control for confounding variables?
The aim is to evenly distribute the known and the unknown confounders between study groups/samples/subjects. The groups might still differ in potential confounders by chance but randomization minimises these differences.
4. Design Flaws to Avoid in Research
When conducting research, it is crucial to be aware of common design flaws that can compromise the validity and reliability of your study. Avoiding these flaws ensures that findings are robust, credible, and applicable. We will dive into more detail of the most common design flaws:
Lack of or Inappropriate Theoretical Framework
A theoretical framework serves as the conceptual foundation of a study. It provides structure and guidance, ensuring that the research is grounded in relevant theories and models. A strong framework should:
-
Clearly align with the research problem and objectives.
-
Include a well-defined set of hypotheses, postulates, or assumptions that can be systematically tested.
-
Offer logical explanations and justifications for the research approach.
Poor Methodology
A well-developed methodology is essential for collecting and analyzing data effectively. To avoid methodological flaws:
-
Develop a clear, transparent plan detailing how data will be collected, measured, and analyzed.
-
Ensure that the data collection method is suitable for the research questions and study objectives.
-
Consider potential biases and errors, and take steps to minimize them.
Lack of Specificity
Vague descriptions and ambiguous terms can lead to misinterpretation of research findings. To enhance clarity:
-
Avoid using vague qualifiers such as extremely, very, entirely, completely.
-
Provide precise descriptions of the research design, procedures, and analysis.
-
Clearly define key terms and variables to ensure consistency in interpretation.
5. Strategies to Improve Research Design
To minimize design flaws and improve the accuracy of research findings, consider doing the following:
-
Increase Sample Size: A larger, more representative sample enhances the study's generalizability and reduces the likelihood of false negatives. If possible, conduct longitudinal studies (see Unit 3 Lesson 1.5) to observe trends over time.
-
Use Appropriate Analysis Methods: Employ robust statistical and analytical techniques that align with the research design. Use consistent methods and repeat trials to improve precision and accuracy.
By addressing these potential design flaws in research, researchers can strengthen their study design and enhance the credibility of their findings.
About Us
Science for Everyone is a Canadian Nonprofit Organization that provides educational resources to help raise the level of scientific literacy in the general population.