Choosing off-the-shelf performance measurement instruments
Rate
Abstract
When used correctly, ready-made evaluation instruments can save time, gather high-quality data that can be compared with similar programs using the same instruments, and help demonstrate program outcomes. However, if used incorrectly, these types of instruments may miss program outcomes, increase burden, and confuse clients about the purpose and scope of the program’s efforts. This effective practice contains helpful tips to guide in the selection of off-the-shelf instruments, and was shared by Project STAR in September 2007.
Issue
Evaluation and measurement is essential for service programs, both from a reporting standpoint and for program development and improvement.
Action
You can choose pre-existing instruments effectively by carefully considering the following three issues:
- Check the Fit
When using an off-the-shelf instrument, make sure there is a good fit between the instrument selected and the program's service activities, beneficiaries, information needs, reporting timeline, and intended use of data:
- Select instruments designed to measure intended outcomes that you expect your program's service activities will produce. Don't judge what the instrument measures simply by its title. Look carefully at the questions.
- Make sure the instrument you choose is intended to measure results for beneficiaries similar to that of your program. Consider factors such as beneficiary age, gender, language, and culture.
- Make sure the data source (the place, organization, or group of individuals that has the information you need to collect) identified by the instrument is accessible and appropriate for the program, especially if the data source is someone other than the beneficiary.
- Be sure that the instrument will provide the information needed. For example, if your performance measurement target states "80 percent of youth will increase their reading rates by five words per minute," the instrument should allow you to measure words per minute.
- Pay Attention to Correct Use
Many off-the-shelf instruments have administration guides or manuals and some require training to administer. Failing to follow these instructions correctly may compromise the reliability and validity of the data you collect. Review the instrument directions to determine:
- The qualifications needed and the training required for individuals administering the instrument
- When, where, and how the instrument should be administered
- Information about scoring or compiling data
- Will adding, changing, or deleting items alter the how the instrument works, or what the data mean.
You may be tempted to change the instrument, including shortening the instrument, rewording or deleting questions, collapsing scales, using a different data source, or changing how the instrument is administered. However, be aware that altering an off-the-shelf instrument can change how it works and may undermine reliability and validity.
- Consider Data Analysis
After choosing an instrument, make sure you will be able to aggregate and analyze the data. Some off-the-shelf instruments come with helpful documentation, such as score interpretation guides or instructions for data analysis. In other cases, the instrument may come with little or no information on how to analyze and interpret data, or you may be required to send the data to a company that will charge a fee for data analysis. Carefully consider data aggregation and analysis issues before adopting an off-the-shelf instrument.
Additional Considerations
- Pre-existing, off-the-shelf instruments are useful because:
- They have a short development time
- Information on use in other settings may be helpful
- You may be able to compare data from your program with data from other programs using the same instrument
- Difficulties with these instruments to consider:
- May not be appropriate for program's activities or beneficiaries
- May not address program's stated outcome targets
- Data may be of limited value if the instrument is of poor quality, inappropriate for your program, or administered incorrectly
- May require extensive training for administration and analysis
- May be required to pay a fee to use instrument or for data analysis
- Instruments developed by your program are useful because:
- Instrument and administration methods can be tailored to program's activities, beneficiaries, and intended results
- There is no fee for use of instrument or to analyze data
- Difficulties with these instruments to consider:
- Instruments take time to develop and pilot test
- Data may not be comparable with data from other programs
- Data may be of limited value if the instrument is poorly developed or administered inconsistently
Context
Off-the-shelf instruments refer to a wide variety of ready-made or pre-existing instruments developed by someone other than you (or your organization). Examples include standardized tests, assessment tools developed by researchers and evaluation professionals, and instruments available on the web, including those from Project STAR. These instruments may be lengthy and complex, or simple and short. They may come with complete instructions for administration and data analysis, or they may come with no instructions at all. They may or may not have undergone testing for reliability and validity.
Outcome
Most programs have a choice between using off-the-shelf instruments, creating instruments tailored to the program's needs, or using a combination of both. By carefully considering data collection needs and instrument choices in light of program resources, activities, beneficiaries, and desired results, programs are more likely to collect data that are relevant to stakeholders and useful for continuous program improvement.
Comments
Post new comment