HSS 3101 Lecture Notes - Lecture 12: Needs Assessment, Indirect Costs, Logic Model
49 views6 pages
Class 12: Evaluation Research and Mixed Methods
What is evaluation research (aka program evaluation)?
- Research undertaken to see whether a program or activity is meeting or has met objectives set for
- Purpose depends on the objectives of the program (Process = how questions)
- Evaluates the impact of social interventions: new teaching methods, innovations in parole etc.
- Evaluation research is a form of applied research—intended to have real-world e&ect.
- Emphasis on the process rather than the product
- Re'ects social scientists’ increasing desire to make a di&erence in the world
oHowever: in'uence of governmental requirements that program evaluations must
accompany the implementation of new programs
oThe availability of research funds to ful+ll those requirements
Purpose of evaluation research:
Can be looked at from di&erent vantage points:
1. The program evaluator
2. The administrator
3. The consumer or public
4. The organization
Whatever the purpose, those involved in the process must be sure to accomplish the task by using
appropriate methodologies, processes, and analyses.
The research project must be able to be replicated by other evaluation groups. (With
-Logic model: so that other programs can copy your model
- Participants interested in outcomes, organizers interested in cost bene+ts
Reasons for Evaluation Research
- Management and administrative reasons
- Assessment of the appropriateness of program changes
- Identi+cation of ways to improve the delivery of interventions
- Accountability to funding agencies
- To ful+ll planning and policy purposes
- To test innovative ideas
- To decide if programs should be curtailed or expanded, or to support one program in lieu of another
What can be evaluated?
•Sta& and personnel
•Evaluation models and
•Topics appropriate to evaluation research
- Evaluation research is appropriate whenever some social intervention occurs or is planned
oSocial intervention: action taken in social context to produce some intended result
- Process of determining whether a social intervention has produced the intended result
- The topics appropriate for evaluation research are limitless.
- The questions appropriate for evaluation research are of great practical signi+cance: jobs,
programs, and investments as well as values and beliefs.
•Steps in conducting evaluation
- They’re interactive, can work at the same time:
oWho is the client?
oWhat are the purposes of the
oWhat is the methodology to be
oWhat is included in the contract?
oHow is the evaluation to be
oWho will use the results of the
oMethodological approaches in evaluation research (types of research design)
- Experimental and Quasi-Experimental Designs
- Personnel Assessment
- Expert Judgment
- Case Study
oSeveral methods may be used in any one evaluation project, as they each contribute to the total
oHow to choose the method
- When determining the methodology to be used, the evaluator must be familiar
with the purposes of the evaluation.
- Characteristics that must be considered when determining the feasibility of the
oAvailability of data
oThe legal, political, ethical, and administrative constraints when conducting an
oFormulating the problem
- Problem: What is the purpose of the intervention to be
- This question often produces vague results.
- A common problem is measuring the “unmeasurable.”
- Evaluation research is a matter of +nding out whether something
is there or not there, whether something happened or did not happen.
- To conduct evaluation research, we must be able to operationalize, observe, and measure.
oOutcome or response variable
oIf a social program is intended to accomplish something, we must be able to measure that
- Achieve agreements on de+nitions in advance
- Achieve agreements on how the measurements will be made
- De+nitions of a problem and a suBcient solution may be de&ned by law or by agency
oMust be aware of such speci+cations and accommodate them
- There may be several outcome measures, for instance surveys of attitudes and behaviors, existing
statistics, use of other resources.
oNew vs Existing measures
- If the study addresses something that’s never been measured before, the choice is easy—new
- If the study addresses something that others have tried to measure, the researcher will need to
evaluate the relative worth of various existing measurement devices in terms of her or his speci+c
research situations and purpose.
oAdvantages of using existing measures
- Greater relevance and
validity than using existing
- Less time and energy
- Greater scienti+c signi+cance, possible comparisons to
- Known degrees of validity and reliability
- New measures require pre-testing
oSuccess or Failure?
- Potentially one of the most taxing aspects of evaluation research is determining whether the
program under review succeeded or failed. De+nitions of “success” and “failure” can be rather
oTypes of Evaluation Research
- Needs assessment
- Formative evaluation
Open ended boxes = people
don’t really write things, maybe
just write more speci+c questions
Can’t measure everything,
just have to zone in on
Secondary data analysis
-Review existing data
-Maximizing time on