David S. Robinson, Ed.D (Master of Education: Harvard Graduate School of Education; Doctor of Education: Harvard Graduate School of Education)

 Contact:

Mobile: 617-733-5979

Email: davidr46@gmail.com

Address: 11 Round About Way, Greenville, SC 29609

Slide Presentation: https://www.slideshare.net/davidr46

Ask A Question Here: https://forms.gle/JuwQiW1gAmGJ3huD7

Program Evaluation in an Era of Equity

Program Evaluation in the Era of Equity: What this means for your work

Image Source: FreeImages

‍

The equity agenda has the potential to shape how we think about and evaluate programs for years to come. In response, program evaluators are asking new questions, developing novel techniques, and adopting new measures of success that align with equity principles. Program evaluation is a set of processes used to assess the effectiveness of programs or interventions in achieving their intended outcomes. Program evaluation can help us understand what is working and what isn’t working so that we can make informed decisions about whether to continue or end the program or take other actions. Programs often struggle with these questions: Are we reaching the people who need our services? Are they using them? Are they seeing positive impacts as a result? And if not, why not?

What is equity?

Equity is the idea that all people deserve to be treated fairly, with dignity and respect, regardless of their differences (e.g., income, race, ethnicity, gender, sexual orientation, language, religion, immigration status, disability status, etc.). In the context of program evaluation, equity is the extent to which a program is meeting the needs of the people it intends to serve or the degree to which a program is having positive impacts on its intended beneficiaries. Equity is a core principle developed for the field of evaluation by the W.E.B. Du Bois Institute for Social Research at Harvard University and the Evaluation & the Public Program (EPP) at the University of Washington. Equity is a broader framework for evaluating programs that goes beyond questions of impact and effectiveness to also include issues of fairness, inclusivity, and participation.

Why is equity important in program evaluation?

Programs that are truly successful expand beyond their initial reach and impact. For this to happen, we need to know who is being left out, why, and what we can do about it. Program evaluation can help answer these questions. But to do so, we must ensure that the evaluation is designed with equity principles in mind. Equity-oriented program evaluations include methods that enable us to identify and understand who is being left out and why, and also provide information on how to better include those people. A common problem in program evaluation is that people who are not served by a program are less likely to participate in the evaluation. This means that their perspective and experiences go missing from the evaluation, resulting in an incomplete and potentially biased picture of the program. There is growing recognition that many of the traditional methods used in program evaluation create significant barriers for people who are left out by the process. Equity-oriented program evaluation methods are designed to reduce or remove these barriers.

Key questions for evaluating equity-oriented programs

- In whose name are we doing this work? Whose experiences and perspectives are shaping our ideas, discussions, and decisions? Whose needs and experiences are we trying to address? - For whom is this work missing? Who is not being reached by the program? Who is not participating in the evaluation? Why are they missing? - What are the barriers that prevent people from participating? Are there ways to remove the barriers to participation?

Novel methods for evaluating equity-oriented programs

- Focus groups with people who are not participating in the program evaluation: One method for addressing the first question above is to invite people who are not participating in the program evaluation to participate in a focus group discussion. This provides an opportunity to learn more about their experiences and perspectives. People who have not been participating in the evaluation are often those who are being left out. - Mixed methods approaches: Mixed methods approaches combine quantitative and qualitative methods, such as surveys and interviews, to provide a more complete and holistic understanding of program impacts and outcomes. - Engagement-oriented evaluation approaches: Engagement-oriented evaluation approaches involve meaningful participation from program participants in designing, conducting, and analyzing the evaluation. Through these approaches, people who are served by the program play a greater role in the evaluation, creating a more comprehensive and complete picture of the program’s impacts and outcomes. - Creative and participatory methods: Creative approaches to program evaluation expand the ways in which the evaluation is done, including the use of art, poetry, and other creative methods. Participatory methods involve participants in designing and conducting the evaluation. Both creative and participatory methods are intended to broaden the reach of the evaluation by making it more accessible and open to a broader range of people and perspectives.

Conclusion

Programs are most effective when they reach a broad range of people and address their needs and experiences. Equity-oriented program evaluation can help us identify who is being left out and why, and suggest ways to more fully include these people. As the field of program evaluation evolves, we are seeing increasing interest in equity-oriented evaluation approaches. This is welcome and necessary because the equity agenda has the potential to shape how we think about and evaluate programs for years to come. In response, program evaluators are asking new questions, developing novel techniques, and adopting new measures of success that align with equity principles.

AEA - American Evaluation Association : Blogs : What is Evaluation?

AEA - American Evaluation Association : Blogs : What is Evaluation?

What is Evaluation?

By: AEA Administrator on Jan 10, 2014

This statement was developed by a previous  AEA Task Force commissioned by the AEA Board to “define and communicate the value of evaluation to the media, the public, and other audiences as well as be used comfortably by evaluators throughout the field without regard for specialty or area of expertise.” The statement is meant to encourage dialogue -- so based on comments and responses it will be revised periodically. Thus, your reactions and comments are encouraged (see comment section below).The Task Force was comprised of both long-time evaluation professionals and AEA members newer to the profession. All have experience and expertise in communicating to others about evaluation. The task force included:

Michael Quinn Patton, Chair, Edith Asibey, Jara Dean-Coffey, Robin Kelley, Roger Miranda, Susan Parker, and Gwen Fariss Newman

What is Evaluation? 

Evaluation is a systematic process to determine merit, worth, value or significance. So what does that mean in practice?  Let’s use one kind of evaluation, program evaluation, to illustrate. Programs and projects of all kinds aspire to make the world a better place. Program evaluation answers questions like: To what extent does the program achieve its goals? How can it be improved? Should it continue? Are the results worth what the program costs? Program evaluators gather and analyze data about what programs are doing and accomplishing to answer these kinds of questions.

A program evaluation has to be designed to be appropriate for the specific program being evaluated. Health programs aim to make people healthier and prevent disease. School programs strive to increase student learning. Employment training programs try to help the unemployed get jobs. Homelessness initiatives work to get people off the streets and into safe housing. Chemical dependency programs help people using alcohol and drugs. Community development programs plan initiatives to increase prosperity among those in poverty. Juvenile diversion programs try to keep kids out of jail and put them on a path to becoming productive adults. For each kind of program, an evaluation would gather and analyze data about that program’s effectiveness. But program evaluation is only one kind of evaluation. 

To read more visit:  http://www.eval.org/p/bl/et/blogid=2&blogaid=4#a_comm_6 

Jeffrey Levi: Making the Healthy Choice the Easy Choice: Eliminating Health Disparities

I just read this article by Jeffrey Levi, Executive Director of Trust for America's Health, and came away convinced even more that the choices that we make as a nation this next election will shape our future economy.  The wrong choice will restrict control of our public health priorities, reduce opportunities to invest in shoring up our most discouraged communities, and deplete resources for improving the infrastructure for bridging pathways to economic health.  The right choice will leverage our most effective capacities to strengthen communities and reduce disparities in health, welfare and education.  I recommend this article, especially for those who have always made the hard but right choice, are discouraged now, and need replenishment.

Jeffrey Levi: Making the Healthy Choice the Easy Choice: Eliminating Health Disparities
"In fact, black children are four times as likely to die from asthma as non-Hispanic white children, and Hispanics are 1.6 times more likely than non-Hispanic whites to die of diabetes. Health disparities are intricately linked with social inequalities based on a variety of factors, including race and ethnicity, socioeconomic status, sexual orientation and gender identity, gender, age, disability, geography, and religion. In order to improve the health of vulnerable communities, we must create and leverage opportunities to address the social determinants of health and promote health equity."

What Good Are Public Health Agencies?

I recently read the following quote from an excerpt in The Mississippi Press from Charles Daughdrill, executive director of the Mississippi Public Health Association - http://blog.gulflive.com/mississippi-press-opinion/2012/04/public_health_funding_why_you_should_care.html.  “As you eat your meals in an inspected restaurant or visit a family member in a licensed hospital or nursing home, public health is the reason.  Each day that you and your family live without fear of polio or other illnesses capable of disabling healthy children and workers, public health is the reason. When a disaster such as Hurricane Katrina or the Gulf oil spill occurs and emergency providers quickly respond to you, public health is the reason.  All of these actions and many more are in place to protect you, your family and your community. Every day, public health touches your life and is an under-appreciated component of our health care system.” 

And  this from Vermont Public Health Department (http://healthvermont.gov/news/2011/040411_day_nphw.aspx)

A Day in Your Life –

6 a.m. - Up out of bed, you head for the shower and then brush your teeth. You have confidence that your water is safe and clean because you’ve had it tested by the Health Department Laboratory. In the kitchen, you set out milk, fruit, cereal and whole grain bread. You know that a nutritious breakfast is the best start for kids and adults alike. On the radio, you hear that flu season is coming to a close. Your family is up to date on their vaccinations, and no one's been sick this year.

7 a.m. - You head off to the car. Fastening seatbelts is a habit – and your policy is “hands off” mobile devices, for safety’s sake. It’s snowing and you drive at a safe speed. You know from public health messages that most crashes and injuries are avoidable.

8 a.m. - At work you enjoy a smoke-free environment, just like almost everywhere else. Vermont was the first state to ban smoking in public places, thanks to the efforts of public health. You smile because your best friend has succeeded in quitting, with help from Vermont’s Quit Network. Your kids have never smoked, thanks to your example and public health messages all around them.

9 a.m. - Your sister calls. She and her new baby are just back from the pediatrician. The six-week checkup was great! The Health Department promotes early prenatal care, and every baby gets a newborn screening to test for conditions that, if untreated, can cause development and learning delays and severe health problems. If a child has a special health need, the Health Department coordinates specialty care. She’s also thrilled with her visits to her Health Department district office, free Text-4-Baby health tips, breastfeeding support, healthy WIC foods and nutrition services, plus a debit-like card to buy fresh fruit and vegetables at local markets.

12 noon - It’s still snowing, but you change into treaded boots and head out the door to join your lunchtime Fit & Healthy walking partners. Public health studies have shown that people who exercise regularly live longer and healthier, with less chronic disease. You have learned to fit regular physical activity into your busy day.

12:30 p.m. - For lunch you stop for take-out soup and salad. You assume that the food is safe to eat because the Health Department oversees inspection and licensing of restaurants, publishes scores, and investigates foodborne illness outbreaks.

1 p.m. - Back at work, you hear the sound of sirens as an ambulance passes your building. Thanks to the Health Department, it is fully equipped and inspected, with EMTs trained and certified and part of a system to make sure critically injured people have the best possible chance of survival.

4 p.m. - Another day ends. You start your car and the radio comes on. You hear a public service announcement advising parents to make sure their young children have all their vaccines. Public health promotes early childhood immunizations to protect against serious, sometimes life-threatening diseases.

5 p.m. - Your children are home, after a day of school. You limit sitting time in front of TV in favor of outdoor activity. You are all on track to keep the extra pounds off and the energy on to live, work, learn and play for a long time to come!

6 p.m. - Your family comes together for dinner, and you tell them to wash their hands before eating. “To keep germs from spreading, we know” they say. “And cover your cough, wash your hands, stay home and away from others when you’re sick.”

7 p.m. - The evening news tells of a mysterious illness among visitors at a local event. Thankfully, the outbreak is under control. The Health Department says planning, practice and close coordination with other local and federal agencies has made for a faster and more effective response to any public health emergency.

8 p.m. - It’s time to talk with your pre-teen about drinking and drugs. You’ve prepared for this, with help from the Health Department’s and your community’s Parent Up! campaign. You say a silent thank-you and make a mental note to join the local health coalition.

10 p.m. - It’s time for bed and you drift off to sleep. Public health has touched your lives in countless ways today, working behind the scenes – nearly invisibly – to keep you all safe and sound.

Health Care Reform for U.S. Posterity

A Successful and Sustainable Health System — How to Get There from Here, by Harvey V. Fineberg, M.D., Ph.D.,N Engl J Med 2012; 366:1020-1027,  March 15, 2012

(Link: http://www.nejm.org/doi/full/10.1056/NEJMsa1114777?query=TOC&utm_source=Twitter&utm_medium=Tweet&utm_campaign=Hootsuite)

A colleague and friend of mine recently sent this thoughtful article by Dr. Fineberg at IOM.  In a nutshell, he recommends supporting the efforts of ACA but also:

• Redouble efforts to enhance quality and safety of medical care
• Focus on health needs of patients who require high cost care to reduce hospital admissions
• Honor patients' preferences regarding late-stage illness, allowing those who want it to spend more quality time with loved ones in more familiar and comfortable surroundings
• Rely on systems engineering and operations research to smooth the flow of patients through the health care system
• Participate in data gathering and performance comparisons regarding pertinent aspects of patient care so more widespread adoption of simple, advantageous advances in care
• Adopt new ethos of medical professionalism that values accountability above autonomy; supports team-based care, and inter-professional education, and accepts responsibility for a system to serve all patients, not only one's own patients

He concludes that we need to do many things at once to reduce inefficiency (compared to other industrialized European countries), supporting prevention, championing competition, comparing effectiveness, removing commercial influences on medical decisions, changing the way we pay doctors, and re-engineering the system - all these changes and more...

With March Madness all around us, I am reminded of the difference between a basketball team and other sports teams - five team members all doing different things, but together focused on one goal 80 or 100 times during the 60 minutes of play time: and doing it all at once.  Health care reform is really about the ingenuity of the American people working together for prosperity..... 

Difference between surveillance and program evaluation

What is the difference between research and program evaluation?
In public health program evaluators have only recently been introduced into the mix of public health personnel. The implementation of evaluations of public health interventions have increased over the past 5 years. There has been some confusion or confounding of the role of the public health surveillance systems for program evaluation. But this is not an effective use of the public health epidemiologist when they are called upon to provide program evaluation. Program evaluation has different purposes and stages from surveillance research activities. The following document provides a detailed distinction between research and evaluation. Although there continues to be language in this document that contributes to the confusion.

Foundational Elements for Program Evaluation Planning, Implementation, and Use of Findings, Steps to a Healthier US Cooperative Agreement Program from:

Prev Chronic Dis. 2006 January; 3(1): A19.

Published online 2005 December 15.

See: PubMed Central, Table: Prev Chronic Dis. 2006 January; 3(1): A19. Published online 2005 December 15.

The Dragonfly Effect and Evaluation

The Dragonfly Effect and Evaluation

"When working to grab attention in a social media campaign, we suggest four design principles: 1)personal: create with a personal hook in mind; 2) unexpected: people like consuming and then sharing new information—draw them in by piquing their curiosity; 3) visual: show, don’t tell—photos and videos speak millions of words; and 4) visceral: design the campaign so it triggers the senses through sight, sound, hearing, or taste." Downloaded from the web on 1/9/2011 at http://www.ssireview.org/articles/entry/the_dragonfly_effect/ This morning a former student of mine sent me an email which described his fear at plunging into the Atlantic ocean in January - and his willingness to do so because he wanted to help raise funds for the Special Olympics (a program for special needs children and adults). He was successful at grabbing my attention and generated a small donation to his cause. He successfully used social media to pique my interest and compelled me to care about his cause. This approach is catching on and I expect to see more and more social programs (with psychological and healthcare programs following as well) using the "dragonfly effect" to raise funds. Evaluators will have to understand the basic principles and dynamics of this approach to successful social marketing.

9 months ago

- About The Dragonfly Effect (November 18, 2010) | Stanford ... - ssireview.org

More people using free preventive benefits provided by Affordable Care

More people using free preventive benefits provided by Affordable Care

Family Physician Urges Residents, Students to Consider 'Social Determinants' of Health -- AAFP News Now -- American Academy of Family Physicians

Family Physician Urges Residents, Students to Consider 'Social Determinants' of Health -- AAFP News Now -- American Academy of Family Physicians

Create Profile on GovLoop - Social Network for Government - GovLoop - Social Network for Government

Create Profile on GovLoop - Social Network for Government - GovLoop - Social Network for Government

Examples of Evaluation Use

The question has been asked before - Are evaluation findings and recommendations ever used by program planners and implementers? During my life as an evaluator, I have had a number of times when findings and recommendations were used by staff and leaders of the programs with whom I worked. Here are three examples, and some reflections on plausible reasons they were used.

Example 1 of Evaluation Use

Most recent example of evaluation use involves an evaluation of a youth development project in Massachusetts involving a Boys & Girls Clubs program to provide an intervention for youth who are at risk for delinquency and school behavior problems. This program was developed in a rural setting where there were previously no resources for youth in two target communities of about 30,000 residents. Police, juvenile court, school principals, and parents in a community needs assessment provided ample support for the need for a strengths-based youth development program. Steering committee members included police, court, local united way funder, bank executive, public housing coordinator, and school principal. Evaluation collected and reported on youth functional behavior (using standardized behavioral assessment) at home, school, community, social-emotional, self-injury, cognitive functioning. Also collected and reported on youth surveys on participation and satisfaction variables.

• Report of results in formal report and in power presentation, and in memoranda to steering committee highlighting findings;
• Results noted the positive changes in youths’ average behavior functioning from beginning of participation in home, community, but not in school;
• Results of youth surveys showed that youth were not spending much time on homework and activities that were related to literacy or math;
• Advisory group and program director modified activities, built in reward incentives for school work; re-trained counselors to include literacy activities within groups and presentations in fun ways.
• Recent results noted shift in participation time on educational activities in the program.

Reflection on Reasons for Use

• positive relationships throughout formative stages of project forged between the program evaluator and program leadership, steering committee members and staff;
• use of plain English to report results, using graphics, narrative, and conversation
• telling the results in easy to understand pictures and words

Example 2 of Evaluation Use

Going back to the 1990s the Head Start Bureau in Administration for Children and Families invested in a demonstration project to help transition Head Start children into the first three grades of public school: Head Start Public School Transition Demonstration Study. In all 31 sites were funded from 30 states. Each site had local evaluator and outside firm coordinated common evaluation measures across all sites. There were common measures and local measures (which varied depending on site interests and capacity). My site was most interested in incorporating Family Service Coordinators into the public school system. Data was collected on the activities of the FSC and school climate factors, including student literacy, numeracy, and social functioning.

• Report of results of the FSC activities, parent satisfaction, teacher satisfaction and principal satisfaction, and changes in school climate were very positive.
• Student academic progress was negligible compared to schools in the community w/o all of the services provided under the Transition Study.
• FSC so positively viewed by school system and community that to this day school budget includes six FSC working across the seven schools in the community, now 15 years later.

Reflection on Reason for Use

• Once principals were sold their major support was solidified; evaluation results was almost irrelevant but often quoted by principals. FSC solved their major time-killer – work with under-involved or over-involved parents;
• Use of narrative stories of principals, parents and FSC provided to school committee and superintendent with graphs and bullets of findings to support the narrative

Example 3 of Evaluation Use

Evaluators do all kinds of things in the course of evaluating programs. Another example comes from evaluation of work with a community HIV/AIDS prevention program that is located in an urban city in the middle of Massachusetts – Worcester has about 180,000 people. The program serves about 1,500 individuals per year. They initiated a new project based on a yearlong needs assessment to provide prevention and referral to treatment for women of color who were at risk for HIV/AIDS and interpersonal violence. They formed a collaborative of health clinic representatives, shelter staff, domestic violence groups, and Red Cross trainers to build the program. During the formative stages, they needed to develop a standardized way to assess women who were at risk of HIV/AIDS and interpersonal violence. There were no existing standardized screening tools available at the time. The evaluators collected screening tools from each of the partners and developed a screening tool with the partners’ full participation and agreement, pilot-tested the screening tool.

• After reporting on the results of the use of the screening system, project and partners incorporated it into their intake and referral system.

Reflection on Reason for Use

• Evaluator built close relationships with counselors, Director of agency, and program director;
• Wrote clear guidelines of how to use screening form;
• Reviewed and discussed problems and exceptions throughout project implementation in regularly scheduled meetings;
• Could quantify number of women with HIV/AIDS and IPV, and those women with 2, 3 or more risk factors for HIV/AIDS and for IPV separately and combined;

Racial disparities in asthma exist even among children with equal access to health care

Racial disparities in asthma exist even among children with equal access to health care

Changing for Accountability and Performance

I attended a conference on Saturday morning in Rhode Island focusing on how medical practices are shaping and innovating on the concept of Patient Centered Medical Home (PCMH). Not new to the American Academy of Pediatrics, PCMH is now cresting on a wave of value-added models of health care practice for both small and large practices.

The Change Curve above was presented at a workshop by Suzanne Houck (of Houck and Associates, Inc.) and Barbara C. Johnson (TransforMED) titled "Adopting a Culture of Innovation."

The takeaway for me? Change is almost always possible in every setting, and it usually follows a predictable pathway, with ups and downs, slow periods and flash flooding moments. Persist, lead and be positive and hopeful along with way to guide colleagues.

The Dragonfly Effect and Evaluation

"When working to grab attention in a social media campaign, we suggest four design principles: 1)personal: create with a personal hook in mind; 2) unexpected: people like consuming and then sharing new information—draw them in by piquing their curiosity; 3) visual: show, don’t tell—photos and videos speak millions of words; and 4) visceral: design the campaign so it triggers the senses through sight, sound, hearing, or taste." Downloaded from the web on 1/9/2011 at http://www.ssireview.org/articles/entry/the_dragonfly_effect/

This morning a former student of mine sent me an email which described his fear at plunging into the Atlantic ocean in January - and his willingness to do so because he wanted to help raise funds for the Special Olympics (a program for special needs children and adults). He was successful at grabbing my attention and generated a small donation to his cause. He successfully used social media to pique my interest and compelled me to care about his cause.

This approach is catching on and I expect to see more and more social programs (with psychological and healthcare programs following as well) using the "dragonfly effect" to raise funds. Evaluators will have to understand the basic principles and dynamics of this approach to successful social marketing.

Prof. Judith H. Hibbard: The King's Fund Annual Conference

Check out this SlideShare Presentation:

Prof. Judith H. Hibbard: The King's Fund Annual Conference

View more webinars from The King’s Fund.

Helping Behavioral Health Programs Succeed

The following announcement was released on March 4th. For more information visit: http://www.hfcm.org/default.asp and click on the Grants tab.

Thursday, March 4, 2010

Boys & Girls Club gets grant

DUDLEY - The Health Foundation of Central Massachusetts has awarded a $175,455 grant to the Boys & Girls Club of Webster-Dudley. This grant will continue the foundation's funding through 2010 for Choices, a comprehensive program to direct at-risk young people toward positive alternatives.

"Choices serves teens by offering a wide variety of programs to teach them how to avoid negative influences and develop healthy habits in their lives. They attend life skills and Internet safety classes, participate in athletics, and receive homework help," said Tony Poti, executive director of the Boys & Girls Club of Webster-Dudley. "We are focused on helping these children develop decision-making and leadership skills, using the most effective and proven programs that have been developed by the Boys & Girls Club of America."

Dr. David S. Robinson, president of DSRobinson & Associates, has been working with the Choices program to evaluate its success and reported very positive results at the grant announcement. "I have studied the key indicators collected on youth who participate in the Choices program, I have come to agree with the Public/Private Ventures research conclusion about the high quality and benefits of Boys and Girls Clubs programs. The youth who participate in Choices programs at the Boys and Girls Clubs of Webster-Dudley show significant gains in social skills, improved attitudes toward school, and positive relationships with peers and adults. All of the Choices participants take advantage of homework help and three out of four teens report that they are doing better in school since involvement in the program," he said.

Scheduling meetings can be difficult and expensive

In this period of cutbacks on funding and fiscal restraint, holding meetings can be expensive. One way to reduce the cost is to hold online meetings. But how do you schedule meetings when everyone is busy and in different locations? Here are three online meeting schedulers that may be helpful.

Tungle.com is located at www.tungle.com. It is a calendar and scheduler that you can sync with Google calendar or other online calendars.

Doodle.com is a simple online meeting scheduler in which you invite others to log on and each is polled on a list of meeting times that you set. See www.doodle.com

MeetingWizard.com is capable of more complex tasks including sending invitations to participants proposing alternative times, summarizing responses, updates to meeting organizer and sending confirmations. See www.meetingwizard.com

Evaluation Follow-up as a model of Good Evaluation Practice

Michael Quinn Patton notes the importance of "walking the walk" of good evaluation practice. Here is his most recent post to Evaltalk.

from MICHAEL PATTON reply-to American Evaluation Association Discussion List to EVALTALK@bama.ua.edu date Wed, Nov 4, 2009 at 9:36 AM subject Re: Tools for evaluating the effectiveness of consulting practice? unsubscribe Unsubscribe from this sender

hide details 9:36 AM (12 minutes ago)

Jane Davidson wrote: "I know MQP is big on insisting any evaluation contract includes time specifically set aside for formal follow-up. Perhaps he might share some of the questions he asks at that time?"

Here's a surprise: I focus first on issues of use. How were findings used? What has happened to recommendations (if some were generated)? What issues have surfaced in putting findings to use? (This opens up and circles back to how the evaluation was conducted and the kinds of issues Jane astutely surfaced in her report on synthesizing what happened in a set of evaluations -- very important observations, Jane).

Having dealt with use of findings and recs, I turn to process issues: What was particularly useful about the evaluation process? Not useful? Strengths, weaknesses, gaps? What, if any, impacts did the evaluation process have (separate from use of findings, i.e., process uses)?

I don't ask directly about the evaluator's or consultant's value-added, but comments on this inevitably emerge from the focus on findings and process uses.

As regards the value of doing this after the fact, I think of it as walking the talk of evaluation. If we expect programs to evaluate their outcomes and impacts, we need to model (i.e., role model) good evaluation by evaluating our own practice and work. And, of course, it informs our scholarship and inquiries into effective and useful evaluation, which in my case means I get to use the findings in my writings, an added cost-benefit of doing this. :)

Michael Quinn Patton
Utilization-Focused EvaluationSaint Paul, MN

MQPatton@Prodigy.net

Preliminary Analyses: How much to share with clients?

While working on a new project, I did a preliminary analysis of pre- and post-reading test scores sent to me by examiners in 5 different sites. I ran frequencies and descriptives for each time period and noticed a large difference in only one site out of the 5. I decided to include this preliminary information in a progress report and asked the question - Did Xsite use a different test during this period? No, was the response. She checked on it and found that the one site with much lower scores had reported the data as standardized grade-level scores, not raw scores!

I typically don't report means and frequencies prior to the first draft of the report, but this example has me thinking. I have avoided a problem early on by deciding to sent the preliminary results. But I also wonder, what future problem have I created, if any? Time will tell as this evaluation case example unfolds.