Using qualitative data to add value to performance measurement
Abstract
Qualitative data is descriptive information from clients served that is documented through open-ended questions, interviews, or structured focus groups. Details provided in these open-ended responses create a fuller picture of a program, and can add vitality to program reports. For example, in a neighborhood watch program, summary statements such as “The majority of participants felt that this effort really improved their sense of safety in their community and they really got to know their neighbors,” can be included in a report alongside quantitative data showing a reduction in crime rates. When presented together, qualitative data and quantitative data can make compelling statements of program results. This effective practice, submitted by Project STAR in August 2006, describes strategies for gathering and utilizing this type of information.Issue
Qualitative data can add value to performance measurement by providing more in-depth information than can be obtained solely from quantitative data (numbers). Qualitative data provides more detailed descriptive information about the results being measured, as this information provides "the meaning behind the numbers.Action
Methods to obtain qualitative data include interviews, journals, structured focus groups and open-ended questions on surveys.An effective technique to summarize qualitative data is as follows:
Organize and simplify open-ended responses using basic content analysis. As with any data analysis technique, the purpose is to identify meaningful patterns in the data. Responses to open-ended interview/survey questions can be analyzed using the following three steps:
Step 1: Identify Response Categories.
Begin by looking through the responses of each participant to a single open-ended survey question and group responses by categories. There are two methods for identifying response categories.
Predetermined categories are identified beforehand, either based on the types of responses you expect to find, or based on the types of responses that are of particular interest to you.
Emergent categories are identified by looking through the data to see what kind of response categories emerge naturally from the data.
An example of predetermined categories for a program that provides transportation services for homebound elderly and disabled persons, you might ask an open question such as: "How did the transportation service improve your quality of life?" For this question, you might anticipate some typical responses, such as improved access to health care, and increased opportunities to socialize outside the home. You might report that "Ten out of 20 clients served stated that transportation services improved access to health care. Seven out of 20 clients stated that transportation services increased opportunities to socialize outside the home."
An example of emergent categories for an open-ended survey question such as: "Please describe your experience with your participation in the program?" Here you may need to read the responses several times before you are able to identify categories such as "friendly staff" or "high quality care."
Step 2: Classify the Data.
The next step is to place each response into one of the categories you have identified. At this stage it is often helpful to have another staff member sort the responses to see if they classify them in the same categories as you. You don't need to be in perfect agreement about how to sort responses. However, if there are major differences in your analyses, then it may be necessary to go back to the first step and see if new or different response categories are needed.
Next, identify significant subcategories within each category. For the previous example of improved quality of life, the following results might be reported: "Five clients stated that the transportation service allowed them to get to doctor appointments. Three clients stated that the transportation service allowed them to get their prescriptions filled."
Note that a few responses may be unique and cannot be placed in a specific subcategory; they may need to be categorized as "miscellaneous."
Step 3: Tabulate the Data/Reduce the Volume of Text (But Don't Lose the Information).
There are several ways to reduce the amount of text within each category.
Create a Composite Response: Create "composite" responses (or use a quote) that reflect the content of all the responses in each category.
Count the Responses: Count the number of responses placed in each category, and report the number of responses for each category.
Write a Short Paragraph: For more complex or extensive data, a short paragraph can be written to summarize the contents of each category. These can be reported with, or without, a count of the individual responses.
When reporting data, present the categories developed, summary of comments, and the number of responses by category for each question. Include the exact question that was asked in the survey or structured interview. Adding quotations from respondents that illustrate positive outcomes can increase the richness of the results.
Context
Program staff should be aware of, and plan for, the additional time involved in collecting, aggregating and analyzing qualitative data. If program staff do not feel they have the time or expertise to analyze open-ended questions or interviews, local academic institutions may be able to assist. Many social science professors are eager to have students learn skills by getting involved in local projects.Outcome
Qualitative data can be used to:
- Add descriptive richness to program outcomes
- Provide insights into why outcomes happened
- Validate results measured quantitatively
- Reveal unintended outcomes of your program
- Provide additional information on needed improvements
Evidence
Rich descriptive information provided by reporting qualitative data allows the reader to feel the impact of the program on a more personal basis, helping to contextual targets and outcomes in the program's performance measures.Posted On
September 13, 2006For More Information