Providing outcome indicators for reporting on educational support and enhancement programs
Abstract
The Corporation for National and Community Service uses information derived from various sources to evaluate and report on its performance with the objective of improving the lives of recipients of assistance provided by members, volunteers, and sub-grantees. The Urban Institute Report on Performance Measurement (July 2002) for the Corporation for National and Community Service provides a critique of current measurements and recommends ways to enhance reporting. This effective practice, excerpted with permission from the Urban Institute's Report, shares standardized outcome indicators for educational support and enhancement programs such as tutoring school-age children, providing reading/literacy training for adults, and mentoring youth. Read the report, Outcome Indicators and Outcome Management,.Issue
Using standardized outcome indicators for program management and reporting to enhance credibility, frequency, and value.Action
- The Urban Institute Report on Performance Measurement (2002) for the Corporation for National and Community Service provides a critique of current program measurements and recommends ways to enhance reporting. Although the various divisions within AmeriCorps programs have different specifications for their service, they all follow the same general service goals. Consequently, these outcome indicators are common across programs (AmeriCorps, National Civilian Community Corps, AmeriCorps*VISTA) despite differences in activities and in the means by which they are accomplished.
- The outcome indicators have been selected to be of value to both project sponsors (for internal management purposes) and the Corporation for National and Community Service (for policy development and aggregate reporting purposes).
- The following represent recommended outcome indicators for the major issue area of "Educational Support and Enhancement" in which grants from the Corporation for National and Community Service are primarily used. This category includes programs for tutoring school-age children, providing reading/literacy training for adults, and mentoring youth.
1. Percent of tutored students, to whom AmeriCorps members or volunteers provided a significant amount of service, who showed improvement in: (a) reading performance scores using a standardized reading test or (b) overall school performance as measured by grade point average or other standard indicators in the post-tutoring period.[End Outcome]
Source(s) for future collection of data for this indicator include: Aggregation of differences between standardized tests administered prior to start of tutoring, and (to be determined) months following completion of tutoring that occurred during the reporting period.
Note: For the outcome indicators for educational support and enhancement, the Corporation -- preferably with considerable input from the field-- will need to define for each indicator the minimum amount of member-time necessary to meet the "significant-amount-of service" criterion.
2. Percent of tutored students' parents and teachers reporting improved: (a) attitudes, behavior, and reading skills for their child/student; (b) motivation, attendance, classroom behavior, homework, and academic skills; and (c) improvements in self-esteem and social skills. [Intermediate Outcome]
Source(s) for future collection of data for this indicator include: Aggregation of surveys of parents or teachers of tutored students administered (to be determined) months following completion tutoring that occurred during the reporting period. The tutored students would be those to whom AmeriCorps members or volunteers provided a significant amount of service.
3. Percent of tutored students, to whom AmeriCorps members or volunteers provided a significant amount of service, who report improvement in reading skills, grades, subject understanding, school-related activities, self-esteem, and/or constructive/positive school-related behaviors as a result of tutoring experiences, and who are willing to recommend their tutor to a friend. [Intermediate Outcome]
Source(s) for future collection of data for this indicator include: Aggregation of surveys of tutored students administered (to be determined) months following completion of tutoring that occurred during the reporting period.
4. Percent of adult reading/literacy training program participants to whom AmeriCorps members or volunteers provided a significant amount of service, who passed the GED or other high school graduation equivalent. [End Outcome]
Source(s) for future collection of data for this indicator include: Aggregation of surveys of training participant administered (to be determined) months following completion of training that occurred during the reporting period.
5. Percent of adult reading/literacy training program participants, to whom AmeriCorps members or volunteers provided a significant amount of service, who report improvement in reading skills, self-esteem, and or/quality of life as a result of the program.[Intermediate Outcome]
Source(s) for future collection of data for this indicator include: Aggregation of surveys of training participant administered (to be determined) months following completion of training.
6. Percent of mentored youth (or alternatively, their parents), to whom AmeriCorps members or volunteers provided a significant amount of service, who report: (a) reduced or no crime activity, (b) reduced or no alcohol use, (c) reduced or no engagement in anti-social behavior, (d) improved school attendance or performance, and/or (e) improved relations with peers and families.[End Outcome]
Source(s) for future collection of data for this indicator include: Aggregation of surveys of mentored youth, or their parents, administered after (to be determined) months of mentoring activities.
As AmeriCorps programs transition from outcome evaluation to performance measurement, the Corporation has selected terminology to describe this new method to determine program results. Other funding agencies may use different terms to describe these same performance measurement concepts.
Additionally, as the Corporation works to ensure measurement standardization, program directors might review outcome indicators used by other Federal agencies. Since the public service efforts of AmeriCorps members and volunteers are often similar to beneficiary assistance provided by agencies such as the U.S. Departments of Education, Health and Human Services, Housing and Urban Development, and Labor, their outcome indicators might help programs gauge their own effectiveness.
Context
A major objective of AmeriCorps is to improve the lives of the direct recipients or beneficiaries of the assistance provided by members, volunteers, and sub-grantees. The Corporation for National and Community Service uses information derived from various sources to evaluate and report on its performance with respect to this objective, including:- Project/program accomplishments data assembled and submitted by sponsoring agencies/non-profit organizations
- Surveys of sponsoring agencies'/non-profit organizations' ratings of project success
- Surveys of corporation members' ratings of project success
- Surveys of "community representatives" ratings of project success
- Reports of ad-hoc program evaluation studies
The Urban Institute's report recommends that ideally, for each of the appropriate outcome indicators there would exist one standard data collection instrument for use by all programs providing that particular service. However, complete standardization is not likely to be feasible. Therefore, the Corporation, working with stakeholder organizations to establish a limited core set of acceptable data collection items, might be a potential compromise. These would allow establishment of quality standards and permit comparisons and summation across programs or program categories. This would also enable individual projects to add other data items they believe would be useful to them.
The Urban Institute is a nonpartisan economic and social policy research organization, concentrating on issues important to the nation. Quantitative modeling is a core competency of the Urban Institute, as is revamping research methodologies to broaden their application. Frequently the Institute collaborates with outside researchers and other institutions or works in partnership with federal, state, and local agencies.
Citation
Hatry, Harry P., et al. Outcome Indicators and Outcome Management. Washington, DC: The Urban Institute, Final Report, July 15, 2002.
Submitted to the Corporation for National and Community Service.
Contract No. GS-23F-8198HUI No. 07112-004-00
The Urban Institute is a nonprofit, nonpartisan policy research and educational organization that examines the social, economic, and governance problems facing the nation. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.
Outcome
According to the Urban Institute, if outcome indicators were at least roughly standardized over many, if not all, projects, the annual findings could serve both local project management purposes, such as identifying if, and where, problems exist; and be converted into index scores for reporting to the Corporation for National and Community Service and aggregating across projects.
The more such indicators are standardized across projects, the more useful they will be for purposes of comparison and aggregation.
Posted On
November 6, 2002Resources
From The Resource Center library:
Outcome Indicators and Outcome Management
Item number: 1819
Source Documents
Related Practices
Related sites
The Corporation for National and Community Service