Psychology 201W: Introduction to Research Methods
Good and Bad Experiments
- Class Midterm Average = 73%
- Scores are posted on RCB 7242
- First Day of Data Collection, Bring 2-signed forms (proposal sheet and ethics
- Make sure to act professionally when collecting data
- Next assignment = 7% short written assignment (creating an experiment) brief
and concise. Due in 2 weeks (more information in hand out)
- Final exam is cumulative, but questions are weighted to the second part of the
Review on Last Week’s Lecture:
- Non-Experimental Research:
The Case Study Method
o Little Hans
Sample Surveys – Descriptive and Relational Surveys
o Literary Digest
o Political Polls
o Kinsey Report
Different Aspects of Sampling:
o Non-Probability Sampling
o Probability Sampling
Reliability: A measurement is reliable if it can be replicated with
Validity: Refers to the correctness or accuracy of the research
Target Shooters Analogy
- Measuring reliability
1. Test Retest
2. Alternate Forms
3. Internal Consistency
- Other aspects of measurement accuracy – how we actually go about
measuring the construct of interest?
- Interviewer bias, loaded questions, ambiguous or poorly
------------------------------------------------------------------------------------------------------------ PSYCH 201W
Good and Bad Experiments
In this Discipline, Language Plays a Huge Role.
- We have to be clear and concise with our language and consistent with our
- So critics outside, or even within the discipline the application and the use of
language is one of the major criticisms because there are some people feel that this
discipline lacks at precision and lacks the consistency, which is required for the
When referring to science, let’s not refer to it as an institution because it is a
method of inquiry.
- It is a method of inquiring about various phenomena.
- If we think of science in these other ways it can create claims and arguments that
are really tenuous.
- Cozby: “In the experimental method, all extraneous variable are controlled”
- Rationale behind the experimental design: If nothing differs between the
experimental conditions except the manipulated IV and any observed changes in the
DV MUST have been caused by the change in the IV.
- The internal validity of an experiment is compromised if confounds are present.
- Confounding variables are ones that take on systematically different values across
the levels of the IV and if the IV is not the only variable allowed to vary then changes
on the DV cannot be attributed to changes on the IV.
- Causes a lot of problems with making causal statements.
- The biggest challenge is controlling or eliminating confounds.
Good Experimental Designs:
- Good experimental designs are concerned with eliminating possible confounds.
- Eliminating possible confounds allows our confidence to increase when
contributing changes in the DV to the manipulation in the IV.
- So when you do your experiment and do your write up and you see a difference
between groups in the expected direction – it’s okay to say that given that all the
confounds seem to be controlled or eliminated, we can attribute that changes on the
DV to the manipulation of the IV.
* Be aware of your causal claims, be open to the possibility of alternative
explanations and do your best to try and identify all alternative explanations that
- Involves two variables: 1 IV and 1 DV
- Must have at least two levels of the IV PSYCH 201W
- Think of those 2 levels of the IV as: treatment group and control group
- All extraneous variables are controlled, either by keeping them constant or by
some kind of randomization.
- Simple experiment can take a couple of different forms:
Post-Test Only (a.k.a. Simple Random Groups)
Post-test Only (a.k.a. “Simple Random Groups”)
- Participants are randomly assigned to one of the two levels of the IV
1) Random assignment ensure equivalent groups – if the sample size is
sufficiently large, the larger the sample size the more confident that
those two groups are going to be equal in randomly assigned
2) Groups are treated with systematically different levels of the IV, and
the DV is measure and the average values of the DV are compared
- When thinking about it, conceptually compared to the posttest only design the
pretest-posttest design does not leave it up to chance that those groups are going to
be equivalent through random assignment.
- The pretest allows the researcher to have all kinds of advantages:
Match those participants on some particular variable and make sure
that those groups are equivalent on important variable that related to
- One thing you need to know about the pretest-posttest design is that the
comparison between groups isn’t between one set of scores to another set of scores
or an average of two groups, it’s rather between the groups pretest scores and their
Take the posttest scores and subtract the pretest scores, and it’s the
difference (the changes) between one testing time over another
testing time that we are interested in. (Different way of measuring
some kind of an effect.
- Groups are compared on their changes rather than their raw score.
- Advantages of a Pretest:
Guards against non-equivalence of groups, particularly when sample
size is small.
Can help identify subjects that vary with a specific variable.
Can give us valuable information that is directly related to the IV.
- Disadvantages of a Pretest:
Can sensitize participants and influence (bias) their behaviour in the
“Demand Characteristics” and participants are tested twice as long
within subject confound can have an impact of the IV, time and effort. PSYCH 201W
Changes within the subject could be affecting their scores (e.g. fatigue,
Time, cost and effort. Increases the cost of research.
Participants Experimental Measure
Assigning Participants to Experimental Conditions:
1) Simple Random Assignment
2) Matched Random Groups
3) Repeated Measures Designs
* The real fundamental difference is between Simple Random Assignment and
Matched Random Groups with the Repeated Measures Design.
- 1) and 2) relatively similar, when it comes to assigning participants to conditions.
Simple Random Assignment:
- The assumption that underlines Simple Random Assignment is the principle of
- The law of averages provides groups with long run of equivalence in all subject
- The larger the sample size, the more confidence we can be in that assumption.
Matched Random Assignment Groups:
- Don’t want to leave it to chance for the groups to be equivalent on some important
- Make sure groups are equal in 1 or 2 main important variables
- Measure participants in some important variable and then rank order their scores
on whatever that variable we’re interested in is and then randomly assign them to
one of the two groups.
- The variable must be strongly dependent or associated with the dependent
variable. Otherwise, there is no point in matching the participants on some
- Only useful when the DV is strongly related to the IV
- Advantages of Matching:
Equivalent groups or very close to being equivalent on some
important variable (important for small sample sizes)
Improves the ability for certain statistical tests of inference to detect
group differences, if sample sizes are small then the statistical test we
can use is inferential statistics because it will be more effective.
- Disadvantages of Matching: Time, Cost and Effort. PSYCH 201W
Repeated Measures Design:
- Fundamental Difference: Any kind of advantage that is gained from matching is
taken to the absolute extreme when using Repeated Measures Design.
- Same subjects are used in all conditions, therefore we know for absolute certain
that there are no differences in the groups because one group is assigned to all the
conditions in the experiment.
- Groups are known to be the same because they are all the same participants.
- Use of neutral Terms when describing the experimental and control groups,
usually refer to the various conditions of the experiment, or referring to the
different levels of the application of the IV.
- Problem to various levels of the IV: Order of which the participants encounter
the conditions in the experiment.
- Solution: Counter-Balancing – a method of arranging the order of how the
participants are exposed to different levels of the IV.
Class Example: Counter-balancing
- Treatment and Control groups
- 3 levels of the IV
- BAC and etc.
Question we need to consider when choosing the Type of Design:
- How much control does the different designs allow the experimenter over various
Choice of Design:
- Control over various confounds:
Between Subject Confounds
Within Subject Confounds
Between Subject Confounds:
- We use randomization to control Between Subject differences in the simple
Random Groups Design.
- Randomization leaves it up to chance whether the groups will be equal.
- Matched Random Groups Design has much more control due to the fact that at
least one variable has been equated – we measure one variable that we think is
related to the DV. PSYCH 201W
- The Most Control in Between Subject Confounds: Repeated Measures Design
because it guarantees that all between subject variables have been equated because
all the participants are exposed to all levels of the IV.
Within Subject Confounds:
- Within Subject Confound