Flaws in Research & False Claims

Historically there have been numerous flawed research studies which, due to poor study design, bias, as well as data manipulation, caused harm to the scientific community and the general public due to the spread of misinformation. A famous example of this is the Wakefield study from 1998 where data was modified to lead to an incorrect conclusion linking vaccines to autism. This study had serious repercussions and resulted in widespread concern among parents, leading to lower vaccination rates and wider spread of disease. As such, it is important to be aware of common flaws in research to be able to identify poor, inaccurate, or biased studies.

Only 12 children were involved in the study. This is a very small sample size to draw such conclusions from.
The parents of most of the children involved in the study (8/12) were of the belief that MMR vaccines diminished their children’s skills such as speech. This suggests two things: first, a potential role of confirmation bias in the results reported in the study, and second, that there was no proper control group in the study.
Wakefield’s research was funded by lawyers suing the manufacturers of the MMR vaccine, making this a great conflict of interest.
The reports made by Wakefield in his paper about the children’s health or diagnoses could not be traced back to their medical records, suggesting misreporting and manipulation of data.

A study conducted by Dr. Andrew Wakefield was published in 1998 where the researchers concluded a positive association between the measles, mumps, and rubella (MMR) vaccine and autism. This resulted in numerous parents refusing to vaccinate their children and spread of these infectious diseases. Let’s take a look at some of the reasons this paper is an example of flawed research (21, 22).

Importantly, since the Wakefield study there have been many studies to explore the link between autism and vaccines and so far, none have proven an association between the two. These studies were published in higher quality journals and used a much bigger sample size with appropriate controls compared to the Wakefield study (22).

Let's explore the different types of research flaws:

1. Statistical Analysis Manipulation

Statistical analysis is the step after collecting data in an experiment. It’s where various language-based and mathematical techniques are used to showcase data in different ways, and to determine if the relationship between data is significant (i.e., if there is more to it). Click through the tabs to learn about the most common ways of statistical analysis manipulation.

Incorrect Use of Percentages

Percentages may be used to hide the actual values collected during the experiment, or make them seem more reputable. For example, in a drug study on athletes, the drug treatment group has ten out of 500 athletes experience improved performance. In comparison, the control or non-treatment group has only five out of 500 athletes experience improved performance. The researchers may choose to emphasize how “the drug increases athlete performance by 100% compared to placebo.” Even though in percentage terms this statement would be correct, it still obscures the fact that only five more athletes (out of 500) experienced an improvement after taking the drug.

Misuse of Means or Averages

Using means or averages without really thinking about their relevance can oversimplify data. This can be misleading if the data has outliers, for example, as averages will be skewed in the direction of the outlier. Imagine you ’are recording changes in swimming speed in female swimmers after they drink a new electrolyte. For the majority of participants, a 1%-10% increase in swimming speed is recorded, but one participant sees an improvement of 40%. The average improvement in swimming speed when taken of all participants, will be higher than if the outlier was discounted. This may suggest a moderate effect of treatment on performance, even if the majority of participants did not experience much of an improvement. The outlier should be mentioned in the analysis section to ensure truthful reporting of all experimental results, instead of being included in the analysis and distorting the averages.

Omission of Statistical Qualifiers

Statistical qualifiers are terms such as “approximately,” “likely/unlikely,” or “on average.” Statistical qualifiers help explain data while acknowledging variability. They may be omitted from or not explained properly in the study. For example, thinking about another potential study conducted on athletes, the researchers may conclude that athletes “improved their running performance by 15 seconds,” when improvement actually ranged from five seconds to 20 seconds. This statement, without a qualifier such as “approximately” being used, implies that every single athlete individually improved by 15 seconds (which is not the case, it’s just an average).

P-hacking

P-hacking is a way to manipulate statistical analysis. It is used to show results as being statistically significant by rigorously tweaking or “slicing up” a dataset to make desired conclusions (23). Research results are usually considered statistically significant, or show a real pattern and are unlikely to be random, when a calculation called a p-value is less than or equal to 0.05. In other words, the p-value tells us the likelihood that our results are actually due to changes in the variables we’re interested in, or if they’re just based on random chance. This value is considered the standard cutoff for significance across science. Playing around with the data between any two variables can lead to a p-value being <0.05 purely based on chance or by cherry-picking the data. This flawed technique is best exemplified by a study published by Wansink, Just, and Payne (2012), called “Can Branding Improve School Lunches? (24)” This study looked at improving nutrition in children's school lunches through making healthier choices, such as apples, seem more attractive by putting stickers of characters from Sesame Street on the fruit. The study was retracted after it was revealed that authors used p-hacking to find statistically significant results (25).

2. Misreporting

Misreporting study elements can be another way to create study conclusions that match what the researcher wants to present, rather than what is actually being observed in the experiment. This flaw includes either omitting or changing elements of the study such as the sample size, hypothesis, scope, or other elements. Thinking back to the retracted study by Wansink, Just, and Payne (2012), the reported age of the population was also found to be incorrect (25). The study initially declared the population to be 8-11 year olds, but later admitted the observed population during the experiment were actually preschoolers. This study misreported the participants' ages, potentially influencing the level of funding the researchers could have secured for targeting an older and more diverse population.

3. Selection Bias

Selection bias occurs when researchers cherry-pick the articles they are referencing. They use and compare them in a way that prevents forming an objective view on the topic of interest. This may be due to researchers having a certain bias or desire towards achieving a specific conclusion. A 2022 study by Gaertner et al. on the effectiveness of homeopathy on the treatment of ADHD was retracted because articles included in this meta-analysis used bad data to push the conclusion of homeopathy as being beneficial (26). The study included articles that had altered or falsified results in favour of homeopathy, selecting them in a biased way to prove their preconceived hypothesis.

4. Data Manipulation, Fabrication, or Falsification

Data manipulation, fabrication, or falsification may also be observed in flawed research studies. These alterations to data may be made by the researcher to fit a certain conclusion or narrative. The desired results may not be reflected in the experimental data they collect, which the researcher may then feel compelled to change or fake, to support their initial hypothesis. This may be because they are receiving funding from a source that is pushing them towards receiving specific results, or that they want more attention or fame for having results supporting their predictions. In a study conducted by Modlmeier et al. (2014), which looked at how socialization shapes personality differences in the spider species, Stegodyphus dumicola, it was revealed that data had been falsified (27). It was unclear whether the spiders in question were actually collected and used for this study (28). Specific sequences in the data were also found to have been duplicated without statistical or biological explanations.

Quiz Yourself!

Question one: A scientist is studying the efficacy of a new potential cancer drug. The scientist conducts an experiment with cancer cells in a dish, and hopes that when they add the drug, it will kill the cancer cells. However, when they do the actual experiment, they find that the results are inconclusive; the drug does kill some of the cancer cells but not enough to have a “significant” result. So, the scientist changes some of their data to show that the cancer drug is very effective at killing cancer cells.

This is an example of which type of research flaw:

Misreporting
Selection bias
Statistical analysis manipulation
Data manipulation, fabrication, or falsification

Hover to reveal correct answer

4. Data manipulation, fabrication, or falsification

Question two:

A researcher was analyzing scientific articles on lifestyle interventions to investigate which interventions are the most effective for older populations. The researcher hypothesized that daily exercise and caloric restriction are the most effective for weight loss in elderly populations. As the researcher was collecting various scientific articles to analyse, they find many differences in opinion on which interventions are the best. Instead of analyzing all the articles they found, the researcher decides to only analyze the studies that concluded that exercise and caloric restriction were the most effective.

This is an example of which type of research flaw:

Misreporting
Selection bias
Statistical analysis manipulation
Data manipulation, fabrication, or falsification

Hover to reveal correct answer

2. Selection bias

In conclusion, we have learned through this learning module that rigorously designed research studies are crucial for producing reliable results and conclusions. Therefore, it is important to evaluate the limitations of research studies, and understand the flaws and biases that may be present in order to spot them more easily. With this knowledge we can learn to look at research studies more critically to protect from misinformation and disinformation as shown with the example of the Wakefield study.

Previous Lesson

Poorly Designed Research

References

About

About Us

Science for Everyone is a Canadian Nonprofit Organization that provides educational resources to help raise the level of scientific literacy in the general population.