Choosing and working with an external program evaluator
Abstract
According to 2003 AmeriCorps guidelines, ongoing performance measurement should be supplemented with more in-depth rigorous evaluation studies that measure the particular impact of national and community service programs. Sometimes this means that programs will need to hire an outside evaluator. This effective practice offers tips on choosing and working with an evaluator or evaluation team. Ivan Murillo, Evaluation Specialist at Project STAR, shared this information via the AmeriCorps e-mail discussion list (ACList) on March 11, 2003.Issue
Choosing and working with an outside evaluator can help to satisfy accountability and improve program value for participants and those served, but may be a new experience for some program directors.Action
According to Ivan Murillo at Project STAR, when considering working with an outside evaluator, take the following steps when finding an evaluator, budgeting program resources, and managing the process:
Finding an Evaluator
- Consult the W.F. Kellogg Foundation Evaluation Handbook (PDF, 844 KB). This is a basic primer on planning an evaluation, and includes information on selecting an evaluator. Chapter 5, "Planning and Implementing Project-Level Evaluation" has sections on types of evaluators, evaluator's roles, how to find an evaluator, and working as an evaluation team.
- Make sure the evaluator is a good fit. The time initially spent framing a project or an evaluation serves well throughout the project year. Think through such issues as
- Type of evaluator role (internal, external, consultant)
- Type of relationship
- minimal - where an evaluator is already an employee of the project
- key - an evaluator is hired as a part-time or full-time employee
- contract based - an external consultant who works either as an individual or as part of another organization
- team - involving staff, relevant stakeholders and an evaluator
- Type of evaluation approach (top-down, grass-roots)
- Stakeholder and audience interests (philosophical, financial, timelines)
- Universities and community colleges can be good resources for finding an evaluator or evaluation. Faculty members often have graduate students that can help defray research costs and occasionally a class will even take on a program evaluation. Some graduate departments to consider contacting include education, education/research evaluation, social work, and sociology. One of the downfalls of working with graduate students, however, is that your program might not be able to maintain a long-term relationship with this particular evaluator. This might be important to programs that value consistency as part of overall program management and goals.
Budgeting Program Resources
- Consult the W.F. Kellogg Foundation Evaluation Handbook for information on budgeting for an external evaluation.
- A common evaluation budget is usually five to ten percent of a program's overall budget. But this can be adjusted, depending on the degree to which programs can support outside evaluator's work internally (e.g., perform data collection).
- Keep in mind that cost differences are usually due to the burden of the data collection process so performing these activities internally can save money.
- It is appropriate to ask a potential evaluator to talk about different evaluation designs, the questions each will answer, and the associated costs.
Managing the Process
It is very important to manage the evaluation process activity on an ongoing basis. This can be accomplished in part by:
- Remaining in contact with the evaluator throughout the entire process.
- In addition to working with the evaluator to set up an evaluation, continually soliciting feedback from the evaluator as the study progresses.
Context
Evaluations use scientifically based research methods to systematically investigate the effectiveness of programs by comparing the observed program outcomes with what would have happened in the absence of the program. Evaluations estimate the impacts of programs by comparing the difference between the outcomes for individuals receiving a service or participating in a program to the outcomes for similar individuals not receiving a service or not participating in a program.
When considering evaluation studies in assessing an applicant's proposal, the Corporation takes into account that the existence, scope, and nature of evaluations varies based on the organization, its resources, program activities, and model.
Citation
Curnan, Susan , et.al. W.F. Kellogg Foundation Evaluation Handbook. January, 1998.Outcome
The Corporation encourages all grantees to provide for evaluations as part of their programs in order to continuously improve the value of efforts for both participants and those served.Posted On
April 11, 2003For More Information
Resources
To order a copy of the W.F. Kellogg Foundation Evaluation Handbook
Contact:
Collateral Management Company
1255 Hill Brady Road
Battle Creek, Michigan 49015
phone: (616) 964-0700.
(Ask for item number 1203.)
Programs funded by the Corporation for National and Community Service can borrow this resource from The Resource Center library:
W. F. Kellogg Foundation Evaluation Handbook
Library item number: R0818