Planning and conducting a pilot test

Abstract

Pilot testing involves conducting a preliminary test of data collection tools and procedures to identify and eliminate problems, allowing programs to make corrective changes or adjustments before actually collecting data from the target population. This effective practice describes advantages of pilot testing, lists simple procedures to pilot test instruments, and provides additional resources to help programs conduct their own pilot tests and was submitted by Project STAR in September 2008.

Issue

A pilot test usually involves simulating the actual data collection process on a small scale to get feedback on whether or not the instruments are likely to work as expected in a “real world” situation. A typical pilot test involves administering instruments to a small group of individuals that has similar characteristics to the target population, and in a manner that simulates how data will be collected when the instruments are administered to the target population.

Pilot testing gives programs an opportunity to make revisions to instruments and data collection procedures to ensure that appropriate questions are being asked, the right data will be collected, and the data collection methods will work. Programs that neglect pilot testing run the risk of collecting useless data.

Pilot testing provides an opportunity to detect and remedy a wide range of potential problems with an instrument. These problems may include:

Questions that respondents don’t understand
Ambiguous questions
Questions that combine two or more issues in a single question (double-barreled questions)
Questions that make respondents uncomfortable

Pilot testing can also help programs identify ways to improve how an instrument is administered. For example, if respondents show fatigue while completing the instrument, then the program should look for ways to shorten the instrument. If respondents are confused about how to return the completed instrument, then the program needs to clarify instructions and simplify this process.

Action

The following guidelines can be used to conduct a simple pilot test of an instrument.

Find at least 4 or 5 people from the same group of people whom you will actually measure (the target population).
Arrange for these people to complete the instrument under conditions that match as closely as possible the actual conditions under which the instruments will be administered when you collect performance measurement data for your program. Consider the time of day, the location, and the method. If it is a phone interview, then conduct the pilot test over the phone. If it is a mail survey, then make sure the pilot test is completed via the mail. Whenever possible, record the time it takes for respondents to complete the instrument so that you can inform the data collectors of the approximate time needed for respondents to complete the instrument.
After each respondent completes the instrument, take some time with the respondent to discuss his or her experience. The following are some questions you might want to ask.

How long did it take you to complete the instrument?
What do you think this instrument is about?
For what purposes do you think this information will be used?
What problems, if any, did you have completing the instrument?
Are the directions clear?
Are the instructions clear on what to do with the instrument after completing it?
Is there any words/language in the instrument that people might not understand?
Did you find any of the questions to be unnecessary or too sensitive?
Were any questions difficult to answer?
[For specific questions:] What do you think this question is asking?
Did the answer choices allow you to answer as you intended?
Is there anything you would change about the instrument?

Collect the completed instruments. Read through the responses. Did respondents interpret the questions the way you intended?
Analyze the data and present the results of the pilot test as you would when you actually administer the instrument. Will the results give you the information you need?
Share the results of your pilot test with other stakeholders who will be using the data. Does this instrument provide the data they need to answer their questions?
Modify your instrument based on the information you have gathered.

When reporting performance measurement results, be sure to describe any pilot testing you have done. This gives readers greater confidence in the results you report.

It may be tempting to skip the pilot testing step, but remember that you run the risk of collecting useless data. A pilot test almost always reveals ways to improve an instrument. Once data are collected, it will be too late in the data collection cycle to fix problems that were missed because an instrument was not pilot tested.

Context

Grantees of the Corporation for National and Community Service conduct performance measurement to assess the results of their programs. Pilot testing is an important way to improve the usefulness, reliability and validity of data collected for performance measurement. Reliability has to do with the reproducibility or stability of the data. Validity refers to the accuracy of the data.

Outcome

Examples of pilot tests conducted by Project STAR are briefly described below:

In 2006, Project STAR collaborated with the AmeriCorps Historically Black Colleges and Universities (HBCU) Mentoring Alliance in Florida to pilot test a set of instruments designed to measure outcomes for children who participate in mentoring. In 2007, Project STAR collaborated with a national working group made up of Senior Companion Program stakeholders to pilot test three surveys designed to measure outcomes for respite care, companionship/outreach, and independent living services.

Each of these pilot tests provided Project STAR with valuable information on revisions that were needed to strengthen the instruments. These pilot testing efforts also demonstrated that the instruments were capable of collecting useful, valid and reliable data.

A detailed report on the pilot test of instruments for the AmeriCorps HBCU Mentoring Alliance is available upon request from Project STAR. For more information on the pilot test of instruments for Senior Companion Programs, go to http://nationalserviceresources.org/files/legacy/filemanager/download/performanceMeasurement/SCP_Pilot_Update.pdf.