242-08 Research Designs and Research-Based
Claims (Ch. 3: p. 55-66)
● Define the terms variable and constant, and provide examples of each in the context of a
research study. Be sure to include the concept of values in your definition.
Variable - something that varies in the study.
❏ Takes on at least two values, but can have more
❏ Dichotomous Variables = only two values
Constant - something that does not vary in the study.
❏ Only one value in the context of the study.
● Differentiate between measured variables and manipulated variables, and discuss why some
variables can be either manipulated or measured, but others can only be measured. Provide
examples in the context of a research study.
Measured Variables - are observed and recorded.
❏ Ex: Absorbance of light within a leaf
Manipulated Variables - have multiple levels (ex: values) that are controlled by the researcher
❏ Ex: Anxious people in a study. Levels could be low/medium/high anxiety.
Variables that are only measured are done so because they cannot be manipulated like
naturally occurring variables OR maybe it’s unethical to manipulate it so one just results to
● Explain the process of operationalizing a conceptual variable (sometimes called a
psychological construct) to produce an operational definition. Be able to generate multiple
measures of the same conceptual variable and discuss their pros and cons.
Conceptual Variable - abstract idea that may not be directly observed. [Ex: aggressiveness]
Operational Definition - path taken to either measure or manipulate that conceptual variable.
There are multiple operational definitions for one conceptual variable.
Process of Operationalizing a Variable = You’ll be given a variable. Take that variable and
decide whether you’ll be either measuring it or manipulating it. After that, decide different ways
you can measure/manipulate (because there are always multiple ways). Finally, look at your
variable and see if there are any “levels” within the variable. [Note: levels can mean anything
from low/med/high → male or female → presence and absence of variable → time
● List and describe what Dr. Gobel identified as the three REAL goals of science and relate each
goal to a type of research-based claim and the type of research design that supports that type of claim. Distinguish among descriptive research, correlational research, and experimental
research. For each type of research design, indicate the number of measured and manipulated
variables involved and the types of conclusions that can be drawn. Give examples of each.
GOAL 1: DESCRIBING THE WORLD
❏ Done through frequency claims which describe a particular rate or degree of a single
❏ Ex: proportions or means ~ Four out of five doctors recommend taking Vitamin D.
❏ The research that supports a frequency claim is descriptive research which is done
through merely observing and measuring individual variables. [A.K.A. there is only an
involvement of ONE measured variable.]
❏ Conclusions that can be drawn: allows us to know more about the world so we can react
GOAL 2: MAKING PREDICTIONS
❏ Done through association claims which argue that one value of a variable is associated
or correlated with another variable’s particular value.
❏ Ex: associations or correlations ~ Children who eat more green vegetables have more
❏ The research that supports an association claim is correlational research which is
measuring two or more variables and testing for relationships among them. This involves
at least TWO measured variables.
❏ Conclusions that can be drawn: helps us predict the value of an unknown variable which
then allows us to take appropriate actions in anticipation based on our estimate.
GOAL 3: CONTROL THE WORLD
❏ Done through causal claims which argue that one variable is responsible for changing
the other variable.
❏ Ex: Spaced studying improves exam performance.
❏ The research that supports a causal claim is experimental research which is done by
conducting a controlled experiment to look for causal connections among variables.
Requires manipulating independent variable(s), measuring dependent variable(s), and
controlling extraneous variables. This involves at least ONE manipulated variable and
ONE measured variable.
❏ Conclusions that can be drawn: a cause-and-effect relationship is known which we then
use to take appropriate actions to obtain a desired outcome. ^^To help distinguish association vs. causal^^
● Describe the pattern of data observed when variables have a positive (direct) correlation,
negative (inverse) correlation, or no correlation. Be able to sketch each type of relationship
and identify which type of relationship is shown in a scatterplot.
Positive (Direct) Correlation - as one increases, so does the other.
Negative (Inverse) Correlation - as one increases, the other decreases. No Correlation - no pattern between the two variables.
● Describe the three types of variables in an experiment (independent, dependent, and
extraneous variables). Be sure to define the terms level and condition/treatment when
describing independent variables and to briefly discuss control of extraneous variables.
Independent Variable (IV) - manipulated variable.
❏ At least two levels (or values) of the IV
❏ Each different level is a unique condition or treatment of the IV
Dependent Variable (DV) - measured variable.
Extraneous Variable (EV) - all other variables.
❏ Good experiments exert control over EVs to eliminate them as alternative causal factors.
**Given a research-based claim, identify the conceptual variables and type of claim, indicate whether
supporting research should have manipulated or measured each variable, and propose operational
definitions for each variable.
242-09 Four Validities of Research-Based Claims
(Ch. 3: p. 66-81) 1. Define and describe the concepts of construct validity, statistical validity, external validity,
and internal validity. Be sure to identify important aspects of statistical validity and various
factors that are threats to external validity and internal validity.
Construct Validity - Quality of the measures and manipulations. Adequacy of operational
definition(s). Is the operational definition used a quality (valid) measure or manipulation of the
variable or was there a better way?
Statistical Validity - Statistical conclusions are appropriate and reasonable. Do the numbers
support the claim? Is the use of statistics, margin of error, and statistical significance
appropriate? Is the conclusion statistically significant to the world (high probability this is true
outside the lab)?
❏ Type I Error - false positive
❏ Type II Error - false negative
External Validity - To whom, what, or where can we generalize? The degree to which the
findings generalize beyond the sample and research context. Is this conclusion true outside the
sample, outside the lab?
❏ Threatened by anything that may make the data unrepresentative!
❏ Threats to generalizing beyond the sample:
❏ Use of restricted populations or enrollment.
❏ Biased sampling or subject self-selection/volunteer bias
❏ **The above don’t allow for the results to be in association with populations
outside the lab.
❏ Threats to generalizing beyond the research context:
❏ Subject reactivity
❏ Use of highly controlled lab setting
❏ Using unique measures or manipulations in the study
❏ **The above don’t reflect everyday environments populations usually are a part
Internal Validity - No other alternative causal explanations for the outcome. The
degree to which the experimental design ensures that only the IV manipulation is
responsible for the changes in the DV.
❏ Extraneous/Confounding Variables ~ there needs to be an elimination of
alternative causal factors so the study can truly support IV causes changes in
2. Describe how to apply each of the Big Four Validities to each type of research-based claim
(frequency, association, and causal), identifying the questions to ask and which validities are
most important for that type of claim.
FREQUENCY CLAIMS: ❏ Construct Validity ~ how well is the variable measured?
❏ Statistical Validity ~ how large is the sample and margin of error? How confident are you
that the true value is given.
❏ External Validity ~ Essential. Can we generalize this stat to our intended population of
❏ Construct Validity ~ how well was EACH variable measured?
❏ Statistical Validity ~ how strong is the association? Is the statistic significant?
❏ External Validity ~ Does this generalize to other people or contexts?
❏ Construct Validity ~ how well was both IV manipulated? How well DV measured?
❏ Statistical Validity ~ How big is the difference? Stat significant?
❏ External Validity ~ Does this effect/conclusion apply to other people/contexts?
❏ Internal Validity ~ Essential because it establishes causation. How well have EVs
3. Explain the general trade-off between internal and external validity and the situations in which
each is more important.
NOTE: construct validity and statistical validity important for all claims!!!
Steps are usually taken to increase internal validity but that then decreases external validity,
and vice versa. Internal validity essential for causation experiments. External essential for
frequency claims off a sample.
4. List and describe the three criteria for establishing causation; explain how an experimental design
meets those criteria to support causal claims. Be able to evaluate whether a given research
design and result meets the criteria for establishing causation.
Experiments need to meet these three criteria:
a. Covariance - variables are associated. Demonstrated through group differences.
b. Temporal Precedence - manipulation A before B. IV manipulation comes first and DV is
measured after manipulation.
c. Internal Validity - controlling all EVs through consistency and random assignment.
If two or more groups are alike in all respects except for the variation created by the
manipulation of the IV, then any differences noted between the two groups must be due to the
manipulation of the IV.
242-10 Principles of Measurement (Ch. 5: p. 121-
128) ● Define and give examples of self-report, behavioral observation, and physiological measures.
For each, describe advantages and limitations.
Self-Report - when subjects report on their own behavior or state of mind. Often used to
measure psychological states, attitudes, opinions, motivations, etc. [Usually measured through
a rating scale. Likert scale is the most common. Likert scale = 1 → 5 rating agreement.]
❏ Advantages: easy to use
❏ Disadvantages: you cannot be sure if one is telling the truth and you can’t be sure if one
is responding accurately or consistently.
Behavioral Observation - a record of actual (overt) observable behavior of the subjects or
physical traces of that behavior.
❏ Advantages: easy to conduct
❏ Disadvantages: may not allow you to determine the underlying cause for the behavior
(like a disturbed mental state or neural mechanism for example).
Physiological Measures - a measurement of brain activity or peripheral responses. Typically
requires special equipment on a noninvasive or minimally invasive level.
❏ Advantages: allows one to make precise measurements of underlying biological
mechanisms that may not be directly observable within a behavior
❏ Disadvantages: expensive testing sometimes that requires trained individuals
● List several common ways of measuring observable behavior (frequency, latency, accuracy).
Frequency → counting # of behavioral events
Latency → recording time it takes for behavior to begin or occur (ex: reaction time)
Accuracy → noting the proportion of correct responses or errors made
● Give a brief description of various data collection techniques, including those types of measures
above and other mixed techniques such as ethnography, case study, and archival research.
Self-Report - individuals respond to items verbally, in written form, or on electronic device.
Naturalistic Observation - observing behavior of individuals in normal environments (field).
Structured Observation - observing behaviors within controlled tasks (lab).
Physiological - measurement of structure or biological function of the brain or body.
Ethnography - researcher becomes immersed in the behavioral or social system being studied.
Case Study - observe and report a single case or a few cases.
Archival Research - using available records to collect data. ● Describe the defining characteristics of the four scales of measurement: nominal, ordinal,
interval, and ratio. For each, indicate the information content and mathematical/statistical
operations that can and cannot be performed with data on that scale.
**Information one can grasp from a scale of measurement increases alongside statistical power
from nominal → ordinal → interval → ratio**
❏ No order
❏ Can count frequencies but no other mathematical operations can be done.
❏ Ex: Male vs. Female
❏ Categories that can be ranked by quantity
❏ Unknown spacing between the ranked values ~ don’t know the magnitude
❏ Can count frequencies and rank-order comparisons
❏ Ex: Gold, Silver, Bronze Medals
❏ Quantitative scale with values expressed as numerical units
❏ Known spacing (magnitude) between the values
❏ No true zero that refers to an absence of something
❏ Do all operations but ratio judgments because of the lack of a true zero point
❏ Ex: Temperature in Celsius OR Book rating from 1 to 5
❏ Quantitative scale with numerical units
❏ Presence of an absolute zero ~ zero indicates an absence of quantity measured
❏ Can apply all mathematical operations and on top of those ratio comparisons as well
❏ Ex: Weight OR Heart Rate
● Identify the scale of measurement of a given measure and be able to generate a measure that
uses a given scale of measurement.
242-11 Self-Report Measures (Ch. 6: p. 157-168)
● Define the self-report data collection technique and the term questionnaire (also called a
survey or poll). Discuss how self-report measures can be used in different research designs and
settings; give some examples of how data from self-report items can be used/displayed.
Self-Report - measure variables by directly asking participants to respond to questions. Questionnaire (Poll/Survey) - an organized collection of self-report items administered to
Self-Report measures are used in all types of research designs and in both lab and field
settings. Ex: Commonly used electronic device among college students, global use of internet,
● Describe the different types of self-report items (open-ended, forced-choice, and rating scale
including semantic differential and Likert scale formats), including the data generated from
each and their pros and cons, and generate examples of each.
❏ “Essay Questions” ~ Respondents answer a question in their own words
❏ Pros: provides rich qualitative information.
❏ Cons: respondent may not understand your question; difficult to code/categorize diverse
❏ “Multiple choice” questions
❏ Respondents choose from a list of alternatives
❏ Data obtained are counts/proportions from each question and answer choice which we
can then calculate differences and means off of
❏ Pros: gives you control over participant's responses
❏ Cons: answers provided may not align with how a respondent feels
❏ Partially Forced-Choice ~ when an “other” alternative option is added allowing
respondent to specify an alternative answer.
RATING (SEMANTIC DIFFERENTIAL) SCALE:
❏ Rate on a scale anchored by antonyms
❏ Respondents choose a # on a scale or place a mark on a line that best reflects their
❏ Data obtained allows us to calculate differences and means
❏ Pros: respondents given wide range of options on how they feel
❏ Cons: way scale is measured/labelled will affect the responses
❏ Numerical ratings of agreement with statements
❏ Respondents indicate their degree of agreement with a statement on a 5 point scale
❏ Assesses attitudes and behaviors
● Discuss different ways of constructing a rating scale, including the number of points on the scale
and how to label the scale, and describe the specific type of rating scale that is a Likert scale or
Likert-type scale. Whelp, you can label just the endpoints, the endpoints and a neutral middle, all the numbered
options, or even get rid of labels and have a continuous rating scale. Points can either be from 1
to 5 or from 1 to 7.
● Provide suggestions for assembling a questionnaire, particularly in terms of question order.
Clearly define the topic and goal. Determine clear conceptual and operational definitions for
variables measured. Write items that capture the targeted behaviors or attitudes. Question wise
though, organize items coherently. Establish a logical navigational path with your questions and
keep question order like interesting first and sensitive following.
● Be able to calculate a composite score, being sure to properly account for reverse-scored
Add all responses up and that is your composite score. Be aware of reversely scored answers
thought! Reverse score = (k+1) - original score where k = magnitude of scale.
● Identify and describe some common wording problems in self-report items, including leading,
double-barreled, and negatively worded questions and avoiding inverted rating scales.
Provide suggestions for writing good self-report items and evaluate the quality of a given item,
suggesting improvements to the wording, if needed.
Leading - using words to bias one’s opinion on a topic while asking the question
Ex: Do you agree that it is a terrible idea to legalize marijuana?
Double-Barreled - asking about two topics within one question
Ex: Do you enjoy long walks and taking exams?
Negatively Worded - using double negatives to confuse respondents
Ex: People who do not drive with a suspended license should never be punished. *then
rate on a 1-5 Likert Scale**
Inverted Rating Scales - Having the low end of the scale being a “high value”
Ex: How spicy is this salsa? (Spicy) 1 2 3 4 5 (Bland)
● Describe common problems with how respondents respond to self-report items, including the use
of response sets such as acquiescence and fence sitting, socially desirable responding,
unconscious influences, and memory distortion. Suggest solutions for these problems.
Response sets (shortcuts):
❏ Acquiescence - yea-saying or nay-saying
❏ Fence-Sitting - respond neutral to all
Dishonest Responses: ❏ Socially Desirable Responding - faking good (trying to look good) or faking bad (trying to
❏ Unconscious Influences
❏ Memory Distortion - one does not explicitly remember things in full detail so any question
regarding memory is going to be biased.
Solutions to all these problems can be solved using reverse-scored items to “catch” these