Data Collection: Bias and Quantitative vs. Qualitative Data Collection
After deciding what your sample population will be, you want to ensure that whenever possible, your study results are not skewed by anything that may affect the interpretation of data.​​​​
For blinding, refer to Unit 1 Lesson 3.5
For bias, refer to Unit 1 Lesson 3.1
For experiments (study design), refer to Unit 1 Lesson 1
Let’s first look at unblinded or open label studies to give you an idea of things that can skew your results.
Unblinded/Open Label Study
-
Definition: The participants and the researchers (individuals administering the treatments) are aware of what study group they are assigned to
-
Application: In studies such as exercise therapy interventions, it is not possible to blind the person providing the treatment, or the participant receiving the treatment
-
Advantage: Significantly less organizational/logistical preparation than blinded studies
-
Disadvantage: If participants know which group they are assigned to, they may change their behaviour which may change the outcomes [13]
​When participants of your study have information that may influence their decisions and behaviour, it can ultimately bias your study results. Thankfully, researchers have come up with various strategies that eliminate this type of bias. One such strategy is implementing a placebo in the study.
​Placebo/Sham
-
Definition: Control groups are provided an inactive substance that looks the same as the real intervention (placebo) or a surgery or procedure that goes through the same process as the real treatment (sham)
-
Application:
-
Helps researchers know if the drug/treatment/procedure of interest is having its intended effects
-
If the placebo/sham yields positive results, this may be the “placebo effect” [14]
-
Can be used in all types of blinded studies (single, double, or triple) - we will introduce these below!
-
Single-blind Study
-
Definition: Participants do not know what study group they are assigned to but the researchers are aware
-
Advantage: Less organizational/logistical preparation than double or triple-blinded studies, but still some effort in place to limit any bias that would change the behaviour of the participants
-
Disadvantage: Researchers may treat those in a control group different than those in a treatment group [13,14,15]
Double-blind Study
-
​Definition: Participants and researchers (those administering the treatments) are unaware of which study group they are assigned to
-
This is considered the “gold standard” when it comes to blinding
-
Prevents participants and researchers from being biased in their responses or how they interact with participants
-
Minimizes many types of biases (refer to Unit 1 Lesson 3.1 for more on bias!)
-
Advantage: Less organizational/logistical preparation than triple-blinded studies, and does a good job of limiting any bias that would change the behaviour of the participants or the perspective of the researcher making their observations
-
Disadvantage: Requires more organizational/logistical preparation than unblinded or single-blind studies​ [13,14,15]
Triple-blind Study
-
Definition: Participants, researchers, and those doing the data analysis are all unaware what study group participants are assigned to
-
Advantage: Does a thorough job of limiting any bias that would change the behaviour of the participants or the perspective of the researcher making their observations, as well as the analysis of the data
-
Disadvantage: Requires significantly more organizational/logistical preparation than unblinded, single-blind, or double-blind studies, and if data analysts expect a certain outcome, and analyse the data in a different way it may lead to different conclusions [13,14,15]
Study design methods
​
With the above in mind, let’s look at different methods you can use to design your study. For observational studies, the most common way is to distribute surveys or questionnaires [16]. ​
​​
When you create one-time surveys, for instance about participant experience at an event, these are called cross-sectional studies. If you were to collect survey results over several time points, for instance following a group of people who smoke regularly and evaluating their lung health every few years, you would be performing a longitudinal study [17].
Another way studies can be conducted is in the form of experiments, where you aim to establish a relationship between variables [16]. Experiments can be performed in the real world (e.g., clinical trials), or within a computational or theoretical setting. If you are interested in learning more about experimental design, check out our module on the scientific method!
Data collection methods
Finally, let’s go over a few data collection methods for both qualitative and quantitative data. No one method is perfect. Based on what you want to get out of your data, you should choose the method that is most suitable for the situation [18,19,20].
Qualitative or Quantitative?
Definition
Advantages
Disadvantages
Simple Observations
Qualitative​​​
​​
​
​
​
​
​
Making observations about a phenomenon or event from your own perspective.
Very easy to collect data, few resources needed.
Very limited in depth—perspective is limited to only that of the researcher. Often we need more than this when studying things like human behaviour or the experiences of others.
Interviews
Qualitative
​
​
​
​
​
​
​
A structured conversation where one party asks questions and the other responds. There can be a variety of styles depending on the researcher’s needs from structured repeated interviews, to off-site more casual interviews.
Deeper understanding: Interviews allow you to get a nuanced understanding of participants' attitudes, beliefs and emotions. The nature of collection (it being a conversation) allows for participants to use their own words when answering.
​
Diverse perspective: Participants can be selected to bring different backgrounds and viewpoints, allowing for a more diverse perspective while still using an identifiable demographic.
Time consuming: Most time-consuming out of all the qualitative data collection techniques, can take many hours to schedule and conduct all planned interviews.
​
Bias: Interviewer must be careful in avoiding creating bias when asking and prompting questions.
​
Variability in data: Often difficult to replicate, as individuals can provide varying answers depending on circumstances, data is also prone to wide ranges of responses depending on the interviewee’s response, from incredibly lengthy responses to curt responses.
Open-ended survey or questionnaire
Both
A series of questions that gather information on specific topics, behaviors, attitudes, experiences, or opinions from individuals.
Flexibility on participants: Rather than gated to a scheduled time, the participant is able to complete the survey on their own time. Also allows for the possibility of anonymity.
​
Consideration: When writing people have the ability to write in multiple instances, and check over their answers before submitting, allowing for greater time to think about answers.
​
Least resource intensive qualitative data collection method: Requires the least amount of resources dedicated to fulfilling the data collection. Once the survey has been created and sent to participants, the researchers can just wait for responses to trickle in.
More rigid answers: No one speaks like they write, and in speaking there is the ability for greater truth to come out (when asking about perspectives). In writing, people are often less personable, and may not provide the answers to the depth wanted.
​
Cannot additionally probe: In the other examples, if there was a thread that seemed relevant/important to further examine, participants could be probed for additional information. In a survey, you get the responses you get and often don’t have the ability for a follow-up.
​
Potential for misinterpretation: With the subject being further removed from the response, there is a greater likelihood of misinterpretation by the researcher in question.
Measurements
Quantitative
Assignment of numerical values or scores to the attributes or characteristics of phenomena being studied.
Standardization in collection methods, easy to compare values from different participants, times, etc. Very easy to see patterns.
​
Simple to collect, other people can be easily trained to collect data which can speed up the process.
​
Usually aided by a measurement tool (e.g., ruler, scale, etc.).
No detailed data—information is limited to numbers that cannot tell a deeper story of why things are happening.
About Us
Science for Everyone is a Canadian Nonprofit Organization that provides educational resources to help raise the level of scientific literacy in the general population.