Research Methods Final Study Guide
Survey Research: B&B Chapter 8 and lecture
• What types of data do researchers gather with surveys and what challenges are involved
o Levels and kinds of factual knowledge held by respondents
o Cognitive beliefs or perceptions about some phenomenon
o Affective feelings or emotional responses
o Reports on behaviors
o Trait or state orientations or dispositions
o Networks of communication
o Demographic features
People don’t always do what they say they do
Attitudes, beliefs, habits, interests change often
Changes in wording creates different responses
Respondents can misinterpret questions
Order of questions can affect answers
Question format changes respondents answers
Respondents will answer questions when don’t know a lot about a topic
• What are the guidelines for asking good questions (i.e., things to do and things to avoid
when developing survey questions)?
o Create good openended and good closedended questions
o Create questions with clarity
o Avoid double barreled questions
o Make simple questions
o Avoid negative wording
o Avoid biased wording
• What are some strategies to consider when working on survey question
o Use more than one version of the questionnaire
Counterbalancing : question appears early to half of the respondents and later
to the other half
Funnel Format: broad, open questions appear first followed by narrow,
Reverse Funnel Format: Narrow to broad questions
Coherence Order: group together questions that are similarly focused
Contingency Ordering: when relevance of some questions is contingent on
how respondents answered earlier questions
o Have a spread out and uncluttered layout
o Use clear instructions and introductory comments
• What is a composite measure? 2
o The use of several questions to measure a given variable
• What is the difference between a unidimensional and multidimensional composite
o Unidimensional : variable has one facet. all questions focus on single aspect
o Multidimensional : variable has several components. Set of questions focus around
each underlying dimension.
• What is a scale and what is an index? How do they differ? Know and be able to identify
the examples of indexes (e.g., Likert, semantic differential) and scales (Guttman’s
o Scale : constructed through assignment of scores to patterns of responses
Thurstone equalappearing Interval Scales : develop items that are empirically
demonstrated to be equally distant from one another.
Guttman Scalograms : items included in the questionnaire have a definite
order of intensity to them
Comparative judgments : directly compare two phenomena (1: which
phenomena are perceived to be in close proximity to each other; 2: Do the
psychological distances among phenomena converge into meaningful
clusters; 3: what are the dimensions of judgment that underlie respondents
o Index : constructed through simple accumulation of scores assigned to individual
Likerttype : words or numbers represent possible responses (strongly agree
to strongly disagree, 15)
Semantic Differential : choose between two opposite positions (three
dimensions would be evaluation, potency, or activity, THEN choose two
bipolar words eg. Goodbad. Put an X on the space nearer to the word that
represents their feelings)
• Know the style/format issues to consider when developing survey questions (e.g.,
formats for rating scales, labeling choices, common question types).
Likert Scale 3
Experimental Research: B&B Chapter 9 and lecture
• What two basic steps do experiments involve?
o Taking action and observing the consequences of that action
• What is the goal of experimental research? How does it differ from the goal of survey
o to develop generalized understandings about the world
• What is “control”? And how does it relate to experiments?
• What is the difference between a lab experiment and field experiment?
o a lab experiment is in a controlled setting, a field experiment is is studying something
in its natural setting
• How do internal and external validity relate to experiments (particularly lab vs. field
o Experimenter effect : researcher or confederate aware of the nature of the study
unconsciously treats participants differently
o Observer Bias : researchers knowledge of the purpose of the study biases their
observations of the dependent variable
o Researcher Attribute Effect : A characteristic or feature of the researcher effects the
o Hawthorne Effect: Participants responses are influenced because they know they’re
o Testing Effect: Measuring and remeasuring can influence people’s behaviors
o Maturation: Any systematic change in an experiment’s participants during the course
of the experiment that is related to how they score on the dependent variable
o Experiment Mortality: participants will drop out before its completed
o Selection Biases: Groups being compared are not the same. Happens in self selection,
participants decide which experimental treatment they receive
o Intersubject Biases: participants in different experimental groups influence each other
o Compensatory Rivalry: participants deprived of experimental stimulus try to
compensate for missing stimulus by working harder
o History: results of the study are results of current events that take place while study is
o Instrumentation: instrument used to measure DV changes from 1 measurement to 2
o Treatment Confound: Effect of the IV is confounded by another variable
o Statistical Regression: People that score extremes during pretest produce scores on
posttest that regress towards the middle
o Compensation: control group deprived of something that has value, so others offer
compensation to them, but then the group isn’t valid
• What do all experimental designs examine?
o Cause and effect relations between independent and dependent variable
• Know the key terms involved in experimental research designs.
o What is an experimental group and what is a control group? 4
Experimental group receives the stimulus, the control group does not and it
o What is a pretest and what is a posttest?
Pretests are tests before a stimulus is given. Posttests are given after the
stimulus have been given
o What is random assignment? Why do we like random assignment?
Random assignment is when participants are randomly assigned to the
control and experimental groups. This is good because is decreases
o What is matching? When would we use it?
Have two groups that are comparable in terms of the variables being studied
o What is a doubleblind experiment? Why would we do it?
Neither the participants nor anyone that come in contact with the participants
has knowledge of the group to which a given participant has been assigned.
Used to reduce bias.
o What is a manipulation check? Why do it?
To make sure the operationalization of the independent variable was what the
researcher intended. Ask the participants their perceptions of the conditions
to which they were exposed.
• Be familiar with each of the designs (preexperimental, quasiexperimental, and true
experimental) noted in the book, in lecture, and on your “cheat sheet.” Also consider
how these designs relate to (are subject to or help to alleviate) internal validity threats.
This info is noted within each design section in the book.
o Preexperimental : little control by the researcher, conditions not randomly assigned,
IV can be manipulated or naturally occurring
One shot case study:
One group pretest posttest design: randomly assigned to experimental/control
groups. One pretest, given stimuli, 1 posttest. Can compare.
Static groups comparison: two groups, one gets the manipulation. Not
randomly assigned. Group 2 doesn’t get manipulation but test them on DV.
Cant assess change in group 1 because there’s no pretest
o QuasiExperimental : Some control. Not randomly assigned. IV often observed in
natural content (field experiment). Often used for evaluation research.
Time series design : study of processes occurring over time. 1 group tested 3
times, give manipulation and then test 3 more times. Assess stability.
Nonequivalent control group design : create a comparison group that is as
similar as it can be to the experimental group, only it is not exposed to
independent variable. Pretest and posttest. 1 group receives IV and control
group doesn’t. Not randomly assigned. As similar as possible.
Multiple time series Design : collect time series data on two or more matched
comparison groups. 3 pretests with 3 corresponding control groups then add
the manipulation and then another 3 posttests with 3 corresponding control
groups. No random assignment. Can assess change.
o True Experimental Designs : most controlled (laboratory). IV is manipulated. Random
assignment. Limits possibility for interaction between participants.
Pretest posttest control group design : randomly assigned to experimental and
control groups. 1 pretest, given stimuli, 1 posttest. Can compare. 5
Posttest only control group design : 2 randomly assigned. 1 group given
stimuli one not. Only assess posttest. No pretest. Can only assess difference
between two groups.
Solomon four group design : 1 control, 1 experimental. 2 posttest 2 pretest
groups. Can assess change and differences.
(Preexperimental) (Quasiexperimental) (True experimental)
• Be able to read an explanation of a design and note which type of design it is.
• What is evaluation research?
o (Program evaluation): the purpose is to evaluate the effect of social interventions
such as new training programs, the introduction of innovations such as new ways of
farming, and so forth. Frequently used method is the quasiexperiment.
• What do factorial experiments allow us to assess?
o Studying multiple independent variables at once to see how they function separately
Quantitative Analysis: Greenstein Chapter 11 and lecture
• How do we prepare for analysis? What is data cleaning?
o Assign variable names to columns, participant/interview/case number for rows, data
entered for each case into cell
o Data Cleaning: computers may be able to check for errors in wrong data, or you can
check your data over yourself by looking at the distribution
• What do descriptive statistics help us do?
o A way to concisely summarize a large number of cases
• What is a frequency distribution? What do frequency distributions help us assess?
o A summary of the frequencies with which each reported value appeared in the
sample. Helps us see the valid percent
• Know how a distribution might depart from a normal curve (e.g., skew and kurtosis).
Be able to identify the different types of skew and kurtosis and what they mean.
o Skew : positive Negative
5 4 3 2 5 4 3 2 1
o Kurtosis : height of the middle peak 6
• What do central tendencies measure?
o Summary averages
• Know each central tendency statistic and how to perform it (if given data).
o Mode: most often occurring
o Mean: average
o Median: middle attribute
• What do measures of variability tell us?
o (Measures of dispersion) Spread of score values from lowest to highest
• What are range, standard deviation, and variance?
o Range : Distance separating the highest and lowest value (for example ages ranged
o Standard Deviation: an index of the amount of variability in a set of data. A higher
standard deviation means that the data are more dispersed, a lower standard deviation
means that they are more bunched together. Standard score is how many standard
deviations a given score is above or below the mean of a distribution
o Variance: value of the standard deviation squared
• What is Pearson’s r?
o Assesses the degree to which two variables, both interval or ratio, are linearly related
o What is a correlation coefficient?
(r) ranges from 1.00 (a perfect negative relationship) to +1.00 (perfect
o What constitutes a small, medium, and large effect size?
.10= small effect size, .30= medium effect size, .50= large effect size
o What is the coefficient of determination?
(r ) square of the correlation coefficient (ranges from 0 to 1)
• What are crossclassification tables and what do they help us examine?
o (Contingency table) Values of the dependent variable reported as frequencies, are
contingent on values of the independent variables
• Why use inferential statistics?
o Examine the patterning of at least two variables at a time, make claims of difference
or relationship beyond the immediate sample included in the study…make broader
claims of how the variables under study function. Inferential statistics assist you in
drawing conclusions about a population from the study of a sample drawn from it
• What is statistical significance? Level of significance? Substantive significance?
o Statistical significance : likelihood the relationship between variables is because of
sampling error only (we want it to be low)
o Level of significance: we must meet the level of significance (.01, .05, .001) to reject
the null hypothesis (to accept the research hypothesis)
o Substantive significance: asks whether or not the difference matters
• Know the different inferential statistics discussed in lecture and in the book (e.g., chi
square, ttest, ANOVA). Be familiar with all types within each of those statistics, when
you would use them, and what they would tell you.
o ChiSquare : Nonparametric test of significance to identify differences in frequency
Onesample chi square : when interested in number of participants, objects, or
responses, or other units that fall into two or more categories
Chi square for contingency tables
o Ttest: used to determine whether two groups, representing nominal level attributes of
an independent variable, differ on some dependent variable, measured at the interval
or ratio level of measurement.
Independent samples: whether the observed difference in means produces a t
value sufficient for us to reject the null hypothesis
Paired samples: when you compare two related groups on a dependant
variable measured at the interval or ration level
o ANOVA pg. 287
Oneway : tells us whether the several groups differ significantly on the
Factorial : use in examining more than one independent variable against a
dependent variable measured at the interval or ratio level. Use when there are
two or more independent variables, and the attributes of each are nominal
o What does statistical significance imply for each?
Independent samples Ttest : if Tvalue exceeds critical value, reject the null.
Sampling error is unlikely to be the cause of relationship between variable.
Paired Samples : T value larger than critical value, reject the null. Sampling
error is very low
One Way ANOVA : reject the null if ‘f’ ratio is no different than the critical
‘f’ value at or beyond the .05 level. There is a significance but it is unclear
• Omnibus : there is a significant difference in results. But where?
• Post hoc comparisons : used to find where the difference is located
Factorial ANOVA : There is a significant difference in the groups, but can be
o Be able to come up with examples.
Independent samples Ttest : 10 med students randomly assigned to two