Implementing outcome measurement: ten tips
Abstract
Outcome measurement is achievable for small, grassroots programs as well as large multi-site organizations. These ten tips for agencies on implementing outcome measurement are excerpted, with permission, from the article, "Outcome Measurement: Showing Results in the Nonprofit Sector," by Margaret Plantz and Martha Taylor Greenway of the United Way of America in Alexandria, Virginia and also co-authored by Michael Hendricks, an independent consultant. This article first appeared in the Fall 1997 issue of New Directions For Evaluation: Using Performance Measurement to Improve Public and Nonprofit Programs. A full text link to the article is provided.
Issue
The Corporation is placing increased emphasis on the importance of performance measures as both a program management tool and a means by which to communicate program impact.
Action
According to the article by Plantz, Greenway and Hendricks, "Outcome Measurement: Showing Results in the Nonprofit Sector," outcome measurement's non-experimental design and basic data-analysis requirements make it manageable for even small, grassroots programs. These ten ways for agencies to implement outcome measurement are excerpted from this United Way resource. Effective practices include:
- Commitment at the top is essential. Otherwise, the task gets overcome by the other demands of program operation. Before they will commit to outcome measurement, however, agency directors and board presidents must see its value for their agencies.
- Programs must identify their own outcomes, outcome indicators, and data collection procedures that are relevant and useful to their efforts. Outcomes and indicators imposed by outsiders are unlikely to meet these criteria.
- Creating a written logic model of program inputs, activities, outputs and outcomes is a helpful way to think through the changes participants experience during and after the program and to check the logic of the if-then influences the program intends to set in motion.
- Agencies should tap many perspectives when identifying program outcomes. Program volunteers, current and past participants (and perhaps family members), persons such as teachers and employers, and other agencies can point out important outcomes that do not occur to staff.
- Outcome measurement does not always require new data collection efforts. Agencies already compile data that reflect on outcomes.
- Data collection and analysis may pose technical challenges that agencies do not have the in-house capacity to meet. The first time around, guidance on collection and analysis methods from a technical expert will save time, offer reassurance, and improve results.
- A trial run of the outcome measurement system is essential and will lead to changes in the system. The trial run must last long enough to encompass all key data collection points and must involve at least a representative group of program participants. Everyone should expect that the trial run will identify problems; that is its purpose.
- Developing a sound outcome measurement system takes time — to plan, try out, to adjust, and to implement. It easily could take an agency seven months or more of preparation before collecting any data, and it could easily take three to five years or more before the findings from a program's outcome measurement system actually reflect the program's effectiveness. Rushing the development process decreases the likelihood that the findings will be meaningful.
- One useful approach to creating an outcome measurement system (Hatry, van Houten, Plantz, and Greenway, 1996) identifies eight steps grouped in three developmental phases: initial preparation (getting ready to begin, choosing outcomes to measure, specifying indicators for the outcomes and preparing to collect data), a trial run (trying out data collection procedures and data analysis and reporting methods), and implementation (adjusting the outcome measurement system and using the findings).
- Once implemented, the outcome measurement system must be monitored and improved continuously. Programs change and learn. The system must keep up.
Context
Outcome measurement is becoming increasingly common among nonprofit organizations. The United Way system has been a major promoter of outcome measurement since the mid-1990s. Today, approximately 450 United Way organizations ask agencies and programs they fund to identify the outcomes they intend to have for their clients and measure the extent to which those outcomes are achieved.
These organizations — each independent, separately incorporated, and governed by local volunteers — serve communities ranging from major metroplexes to small, rural towns and are located in every state in the Nation. United Way of America, the national leadership organization for the United Way system, has created a series of resources to help United Ways and nonprofit organizations implement and benefit from outcome measurement. These include articles, reports, how-to information and an agency manual.
Outcome
According to a 1999 survey of program directors conducted by James Bell Associates (JBA) of Arlington, Virginia (391 United Way programs, each operated by a different agency) implementing program outcome measurement has helped to:
- Communicate program results to stakeholders
- Focus staff on effort on common goals/shared purposes
- Clarify the intended purpose of the program
- Identify effective practices within the program
- Successfully compete for resources/funding
- Enhance record-keeping systems
- Improve service delivery
- Share effective practices with other programs in other agencies
- Inform program participants about outcome measurement results
- Assess staff performance
- Identify staff training needs
- Allocate resources within the program and the agency
- Increase program participant's investment in achieving positive outcomes
- Recruit staff to work in the program
- Recruit volunteers to work in the program
