III. Evaluating Studies: Validity – Chapter 1 (2 of 2)
Many have argued that the glue uniting scholars and researchers is a shared attitude of skepticism.
Skepticism or critically evaluating knowledge claims requires an analysis of. In this section of the
lecture template, we will review TWO types of validity that you will be required to know for the
short answer exam.
It is important to note that a single research study will never achieve absolute validity or truth;
validity is a matter of degree.
1. Internal Validity – Are my causal inferences (conclusions) valid?
If you are a researcher that is testing a causal research question, you’d like to be able to say (with a
high degree of certainty) that one thing is causing another thing to happen. However, in order to
state this (in the results of your study) you must ensure that the following 3 criteria have been met.
a) Association [social science] (or Covariance [science]) – relationship
This criterion demands that you are able to demonstrate that there is an association between the
variables in your research question (IV and DV).
- Arelationship exists between 2 variable
(Change in one variable results in change in another)
- There has to be an effect on the DV
- The IV and DV must have a relationship
How can researchers establish that an association exists between the IV and DV in their study?
- In a survey, a significant correlation provides evidence that there is an association
between 2 variables. (can see the nature of the survey)
- As long as it’s not 0, a relationship exists
- In an experiment/quasi-experiment (may manipulate not measure IV), you look for mean
differences in the DVs between the treatment and control groups.
o If you change the IV, it should result in a change to the DV.
o No change to the IV = no change to the DV.
b) Directionality (or Temporal Ordering)
This criterion requires you to demonstrate that the association is unidirectional.
- Associations are assumed to be bi-directional relationship (IV DV)
Each are affected by the other
- You must prove that the association is unidirectional (IV DV)
o Prove the cause happened before the effect was observed (Temporal ordering) - E.g. spanking cause aggression – prove that spanking before aggression started
How can researchers establish that the association between the IV and DV is unidirectional?
1) Use a longitudinal design (measuring IV/DV multiple times, over time)
2) Manipulate the IV (not merely measuring it) and then measure the effect it has on the DV
e.g. manipulate how much spanking parents are using; assign parents to treatment to
increase spanking rate, treatment to diminish spanking rate
• Surveys (when you measure IV + DV at the same time) fail to meet this criteria; unlike
experiments/quasi experiments; cannot measure directionality
c) Elimination of Alternative Variables
This criterion requires you to demonstrate that the effect (change in the DV) is due to the
presumed cause (change in the IV) and not an alternative cause (i.e., confounding variable).
What is a confounding variable?
- Avariable that is associated and correlated with the DV but is not under investigation
- It’s not accounted for in the research question (not the variable of interest or IV)
o e.g. aggression because the parents spank? Other reasons?
How can researchers eliminate (or minimize) alternative variables?
- Randomly assign participants to the treatment/control groups
o This is the gold standard to eliminate confounding variables
o Washes away differences between participants in your treatment/control
o Cancels out the other conflicting variables
- Making you confident that the change in DV is because of the change to the IV, results
due to manipulating variable
- If you use a quasi – use matching technique
1. Why does correlation not equal causation?
- Association is necessary for causation for not sufficient (need to establish other 2 criteria)
- Correlation establishes association (a relationship), it does not cause something
- Correlation does not establish directionality
2. Why is a quasi-experiment not the best research method for causal research questions?
- 3 criterion is a problem – not well established even if matching used
- There is no random assignment; therefore, we are concerned about the effects of other
variables If you missed this lecture, see http://www.socialresearchmethods.net/kb/causeeff.php
If you are still not clear about the difference between random assignment and random selection, see
2. External Validity – are your generalizations valid?(depend on how you obtain sample)
This aspect of validity relates to the degree that the research findings from a single study can be
extended or generalized. For example, when it is reasonable to conclude that the research findings
obtained from a particular sample will hold for other samples that were not examined in the
study? In this section, we will differentiate between a (a) universal generalization, (b)
generalizing back to the target population, and (c) generalizing to similar contexts.
(3 types of generalizations but only 2 are valid)
If you missed this lecture, see http://www.socialresearchmethods.net/kb/external.php
a) Universal Generalization: usually not valid
- Generalize results observed beyond the population (ie. To everyone)
o E.g. 3 year old in one area is the same as one in another part of the world
There are at least TWO different sets of rules or standards that researchers use to help them
formulate appropriate generalizations:
b) Sampling Model – generalize findings back to target population
Researchers use this model (or set or rules/standards for generalizing) IF they have clearly
defined their population before they conduct their research study. For instance, researchers would
have to identify their target population both quantitatively and qualitatively. Once they have
identified their target population, they draw either a fair or representative sample from this
clearly defined population, and then begin data collection with the sample/participants.
- Quantitatively defined = size of the population
- Qualitatively defined = membership criteria
Two types of sampling from the target population:
i) Fair Sample – every single member in the population had a fair or equal chance of being selected
Definition: Randomly selected sample from the target population
Generalizing ‘Rules’: automatically generalize findings to target populati