Using surveys with scaled response options to yield high-quality data
Abstract
Identifying appropriate methods for data collection — and subsequently developing them — is an important part of performance evaluation. This effective practice will help programs decide when to use survey questions with scaled response choices and understand how to develop clear questions for these types of evaluations. Three commonly used instruments will be addressed — agreement, evaluation, and frequency scales. When used appropriately, scaled questions can yield high-quality data in a format that is relatively easy to summarize and report. Project STAR shared this effective practice in June 2007.Issue
Offering only two choices to a survey question, such as yes/no, all/none, or agree/disagree, may frustrate respondents if the potential answers are not likely to be simple or straightforward. For example, it may be difficult for a respondent to answer accurately by stating only "yes" or "no" to the question, "Did the information presented in the workshop help you to implement your project?"Action
Surveys and other commonly used instruments ask individuals to respond to questions by selecting from a set of possible answers. Answer options that represent points along a continuum from "strongly agree" to "strongly disagree", "very good" to "very bad," "often" to "rarely," and so on, are referred to as scaled response options.For example, when determining if a workshop was helpful to participants, asking the question, "Did the information presented in the workshop help you to implement your project?" and soliciting a yes/no or agree/disagree response is less beneficial than asking "How helpful was the information presented in the workshop for implementing your project?" Phrasing the question in a manner that allows for a range of choices, such as "not at all helpful," "somewhat helpful," "helpful," and "very helpful," rather than just a yes/no option will yield more meaningful results that are still relatively easy to quantify.
Agreement, evaluation, and frequency are the three basic scales used to collect high-quality data when measuring program outcomes. Each type has different scaled response choices and is appropriate for particular measurement needs.
1. In an agreement scale, the respondent indicates agreement or disagreement with each item, and the magnitude of agreement or disagreement. A neutral point can be included in the scale. This type of response scale works well with items that are posed as statements, but it can also be used with items posed as questions. For example:
I am confident in my driving skills.
Ο Definitely NO Ο Probably NO Ο Probably YES Ο Definitely YES
Since I have been receiving respite services I feel less fatigued.
Ο Strongly Disagree Ο Disagree Ο Neither Agree nor Disagree Ο Agree Ο Strongly agree
2. In an evaluation scale, the respondent provides a rating along a good-bad continuum. Choices can range from very positive to very negative. There is no neutral point in the scale. Evaluation response choices can be used to measure attitudes or to assess performance. For example:
How do the home-delivered meals taste to you?
Ο Terrible Ο Inferior Ο Passable Ο Good Ο Excellent
Please rate the disaster preparedness training session.
Ο Poor Ο Fair Ο Good Ο Excellent
3. In a frequency scale, the respondent is asked how often or how many times something has happened. Frequency scales can be used to measure how often respondents engaged in certain behaviors or how often certain events occur. For example:
How often do you add salt to your food?
Ο Never or Almost Never Ο Seldom Ο Sometimes Ο Often Ο Always or Almost Always
Approximately how many times did you go to a movie theatre in the last 3 months?
Ο Not at all Ο 1-2 times Ο 3-4 times Ο More than 4 times
When developing scales, remember to pose questions that are simple and clear, and test your scale with respondents to make sure it works.
Pitfalls to avoid include:
- Asking questions that are "double-barreled" or evaluate two different things (e.g., "Please rate the quality of the training material and the training presentation"). Be sure your question is aligned with the right type of scaled response choices.
- Including parenthetical numbers (used for coding responses) in frequency scales; respondents may mistake these numbers for the response options.
- Ambiguous or complex wording.
- Response choices that overlap or leave gaps between responses
Context
Surveys, questionnaires, and other data collection tools that make use of scaled response choices are widely used by national service programs to measure outcomes. With a little practice, program staff can learn to develop simple but effective scales to measure the impact of their own service activities.Outcome
Scaled data can provide rich, high-quality information for assessing results and program management effectiveness. In particular, information generated from surveys using scaled response choices can facilitate data analysis and help programs more accurately determine their ability to meet established performance measurement targets. Aggregating and comparing the responses across the scaled response choices of each question can also assist stakeholders in making more informed decisions for program improvement.Posted On
July 17, 2007For More Information