SPED102 Lecture Notes - Lecture 7: Central Tendency, Subgroup Analysis, Comorbidity
2 Jun 2018
School
Department
Course
Professor
Bad Research: A Primer for the Ethically Challenged
• Multiple comparisons fallacy and the sharpshooter fallacy: Research
applications
• Other strategies to bias your research
• Some tricks to misrepresent your presentation of data
Biasing Results Without Fraud:
• For those who are amoral and unethical, the question arises:
o How can we do research in such a way to maximize a desired and
predetermined outcome without overt fraud (i.e. fabricating data)?
Revision → the Magic p value:
• The p value reported in research gives an approximation the probability that
you have a false positive result (compared to a true negative) – p < .05 means
that there is less than 5% chance of a false positive
• BUT this is based on the assumption that you only do ONE comparison (test)
• If you do multiple comparisons, the chance of getting false positive result is
above the allowable 5%
The Multiple Comparisons Fallacy:
• Many sneaky (or incompetent) ways to do lots of comparisons (and hide them)
The Texas Sharpshooter Fallacy:
• Involves drawing the target after the shots have been fired
• Choosing your outcomes after the experiment
Multiple Comparisons and Sharpshooting Applied to Research:
• Multiple outcomes
o Have lots of outcomes (dependent variables)
o Some of them are likely to be significant by chance
o Report/focus on those that turn out to be significant
o E.g. Humphries et al. 1992 → compared Sensory Integration Therapy
to:
▪ Comparison intervention (perceptual-motor program)
▪ No treatment
o How to handle multiple outcomes:
▪ Specify/predict one or small number of primary outcomes
BEFORE you start research
▪ Other outcomes are secondary
▪ Interpret results for secondary outcomes cautiously – they may
be just chance results
▪ Replicate the research for secondary outcomes
• Switching outcomes after research
o Specify one primary outcome variable and change it after the research
(this is decidable unethical at best, fraudulent at worst)
o Effectively setting a target and then moving it after you had fired your
shot
o Trial registers
find more resources at oneclass.com
find more resources at oneclass.com
Document Summary
Bad research: a primer for the ethically challenged: multiple comparisons fallacy and the sharpshooter fallacy: research applications, other strategies to bias your research, some tricks to misrepresent your presentation of data. If you do multiple comparisons, the chance of getting false positive result is above the allowable 5% The multiple comparisons fallacy: many sneaky (or incompetent) ways to do lots of comparisons (and hide them) Involves drawing the target after the shots have been fired: choosing your outcomes after the experiment. Multiple comparisons and sharpshooting applied to research: multiple outcomes, have lots of outcomes (dependent variables, some of them are likely to be significant by chance, report/focus on those that turn out to be significant, e. g. 1992 compared sensory integration therapy to: comparison intervention (perceptual-motor program, no treatment, how to handle multiple outcomes, specify/predict one or small number of primary outcomes. Before you start research: other outcomes are secondary.
