Examples of Evaluation Use

The question has been asked before - Are evaluation findings and recommendations ever used by program planners and implementers? During my life as an evaluator, I have had a number of times when findings and recommendations were used by staff and leaders of the programs with whom I worked. Here are three examples, and some reflections on plausible reasons they were used.

Example 1 of Evaluation Use

Most recent example of evaluation use involves an evaluation of a youth development project in Massachusetts involving a Boys & Girls Clubs program to provide an intervention for youth who are at risk for delinquency and school behavior problems. This program was developed in a rural setting where there were previously no resources for youth in two target communities of about 30,000 residents. Police, juvenile court, school principals, and parents in a community needs assessment provided ample support for the need for a strengths-based youth development program. Steering committee members included police, court, local united way funder, bank executive, public housing coordinator, and school principal. Evaluation collected and reported on youth functional behavior (using standardized behavioral assessment) at home, school, community, social-emotional, self-injury, cognitive functioning. Also collected and reported on youth surveys on participation and satisfaction variables.

• Report of results in formal report and in power presentation, and in memoranda to steering committee highlighting findings;
• Results noted the positive changes in youths’ average behavior functioning from beginning of participation in home, community, but not in school;
• Results of youth surveys showed that youth were not spending much time on homework and activities that were related to literacy or math;
• Advisory group and program director modified activities, built in reward incentives for school work; re-trained counselors to include literacy activities within groups and presentations in fun ways.
• Recent results noted shift in participation time on educational activities in the program.

Reflection on Reasons for Use

• positive relationships throughout formative stages of project forged between the program evaluator and program leadership, steering committee members and staff;
• use of plain English to report results, using graphics, narrative, and conversation
• telling the results in easy to understand pictures and words

Example 2 of Evaluation Use

Going back to the 1990s the Head Start Bureau in Administration for Children and Families invested in a demonstration project to help transition Head Start children into the first three grades of public school: Head Start Public School Transition Demonstration Study. In all 31 sites were funded from 30 states. Each site had local evaluator and outside firm coordinated common evaluation measures across all sites. There were common measures and local measures (which varied depending on site interests and capacity). My site was most interested in incorporating Family Service Coordinators into the public school system. Data was collected on the activities of the FSC and school climate factors, including student literacy, numeracy, and social functioning.

• Report of results of the FSC activities, parent satisfaction, teacher satisfaction and principal satisfaction, and changes in school climate were very positive.
• Student academic progress was negligible compared to schools in the community w/o all of the services provided under the Transition Study.
• FSC so positively viewed by school system and community that to this day school budget includes six FSC working across the seven schools in the community, now 15 years later.

Reflection on Reason for Use

• Once principals were sold their major support was solidified; evaluation results was almost irrelevant but often quoted by principals. FSC solved their major time-killer – work with under-involved or over-involved parents;
• Use of narrative stories of principals, parents and FSC provided to school committee and superintendent with graphs and bullets of findings to support the narrative

Example 3 of Evaluation Use

Evaluators do all kinds of things in the course of evaluating programs. Another example comes from evaluation of work with a community HIV/AIDS prevention program that is located in an urban city in the middle of Massachusetts – Worcester has about 180,000 people. The program serves about 1,500 individuals per year. They initiated a new project based on a yearlong needs assessment to provide prevention and referral to treatment for women of color who were at risk for HIV/AIDS and interpersonal violence. They formed a collaborative of health clinic representatives, shelter staff, domestic violence groups, and Red Cross trainers to build the program. During the formative stages, they needed to develop a standardized way to assess women who were at risk of HIV/AIDS and interpersonal violence. There were no existing standardized screening tools available at the time. The evaluators collected screening tools from each of the partners and developed a screening tool with the partners’ full participation and agreement, pilot-tested the screening tool.

• After reporting on the results of the use of the screening system, project and partners incorporated it into their intake and referral system.

Reflection on Reason for Use

• Evaluator built close relationships with counselors, Director of agency, and program director;
• Wrote clear guidelines of how to use screening form;
• Reviewed and discussed problems and exceptions throughout project implementation in regularly scheduled meetings;
• Could quantify number of women with HIV/AIDS and IPV, and those women with 2, 3 or more risk factors for HIV/AIDS and for IPV separately and combined;