Evaluation Help

Difference between surveillance and program evaluation

2011-11-10T10:25:00.000-08:00

What is the difference between research and program evaluation?
In public health program evaluators have only recently been introduced into the mix of public health personnel. The implementation of evaluations of public health interventions have increased over the past 5 years. There has been some confusion or confounding of the role of the public health surveillance systems for program evaluation. But this is not an effective use of the public health epidemiologist when they are called upon to provide program evaluation. Program evaluation has different purposes and stages from surveillance research activities. The following document provides a detailed distinction between research and evaluation. Although there continues to be language in this document that contributes to the confusion.

Foundational Elements for Program Evaluation Planning, Implementation, and Use of Findings, Steps to a Healthier US Cooperative Agreement Program from:

Prev Chronic Dis. 2006 January; 3(1): A19.

Published online 2005 December 15.

See: PubMed Central, Table: Prev Chronic Dis. 2006 January; 3(1): A19. Published online 2005 December 15.

The Dragonfly Effect and Evaluation

2011-11-02T02:13:00.000-07:00

The Dragonfly Effect and Evaluation

"When working to grab attention in a social media campaign, we suggest four design principles: 1)personal: create with a personal hook in mind; 2) unexpected: people like consuming and then sharing new information—draw them in by piquing their curiosity; 3) visual: show, don’t tell—photos and videos speak millions of words; and 4) visceral: design the campaign so it triggers the senses through sight, sound, hearing, or taste." Downloaded from the web on 1/9/2011 at http://www.ssireview.org/articles/entry/the_dragonfly_effect/ This morning a former student of mine sent me an email which described his fear at plunging into the Atlantic ocean in January - and his willingness to do so because he wanted to help raise funds for the Special Olympics (a program for special needs children and adults). He was successful at grabbing my attention and generated a small donation to his cause. He successfully used social media to pique my interest and compelled me to care about his cause. This approach is catching on and I expect to see more and more social programs (with psychological and healthcare programs following as well) using the "dragonfly effect" to raise funds. Evaluators will have to understand the basic principles and dynamics of this approach to successful social marketing.

9 months ago

- About The Dragonfly Effect (November 18, 2010) | Stanford ... - ssireview.org

More people using free preventive benefits provided by Affordable Care

2011-10-13T05:06:00.000-07:00

More people using free preventive benefits provided by Affordable Care

Family Physician Urges Residents, Students to Consider 'Social Determinants' of Health -- AAFP News Now -- American Academy of Family Physicians

2011-08-05T10:11:00.000-07:00

Family Physician Urges Residents, Students to Consider 'Social Determinants' of Health -- AAFP News Now -- American Academy of Family Physicians

Create Profile on GovLoop - Social Network for Government - GovLoop - Social Network for Government

2011-07-21T06:58:00.000-07:00

Create Profile on GovLoop - Social Network for Government - GovLoop - Social Network for Government

Examples of Evaluation Use

2011-02-23T06:02:00.000-08:00

The question has been asked before - Are evaluation findings and recommendations ever used by program planners and implementers? During my life as an evaluator, I have had a number of times when findings and recommendations were used by staff and leaders of the programs with whom I worked. Here are three examples, and some reflections on plausible reasons they were used.

Example 1 of Evaluation Use

Most recent example of evaluation use involves an evaluation of a youth development project in Massachusetts involving a Boys & Girls Clubs program to provide an intervention for youth who are at risk for delinquency and school behavior problems. This program was developed in a rural setting where there were previously no resources for youth in two target communities of about 30,000 residents. Police, juvenile court, school principals, and parents in a community needs assessment provided ample support for the need for a strengths-based youth development program. Steering committee members included police, court, local united way funder, bank executive, public housing coordinator, and school principal. Evaluation collected and reported on youth functional behavior (using standardized behavioral assessment) at home, school, community, social-emotional, self-injury, cognitive functioning. Also collected and reported on youth surveys on participation and satisfaction variables.

Report of results in formal report and in power presentation, and in memoranda to steering committee highlighting findings;
Results noted the positive changes in youths’ average behavior functioning from beginning of participation in home, community, but not in school;
Results of youth surveys showed that youth were not spending much time on homework and activities that were related to literacy or math;
Advisory group and program director modified activities, built in reward incentives for school work; re-trained counselors to include literacy activities within groups and presentations in fun ways.
Recent results noted shift in participation time on educational activities in the program.

Reflection on Reasons for Use

positive relationships throughout formative stages of project forged between the program evaluator and program leadership, steering committee members and staff;
use of plain English to report results, using graphics, narrative, and conversation
telling the results in easy to understand pictures and words

Example 2 of Evaluation Use

Going back to the 1990s the Head Start Bureau in Administration for Children and Families invested in a demonstration project to help transition Head Start children into the first three grades of public school: Head Start Public School Transition Demonstration Study. In all 31 sites were funded from 30 states. Each site had local evaluator and outside firm coordinated common evaluation measures across all sites. There were common measures and local measures (which varied depending on site interests and capacity). My site was most interested in incorporating Family Service Coordinators into the public school system. Data was collected on the activities of the FSC and school climate factors, including student literacy, numeracy, and social functioning.

Report of results of the FSC activities, parent satisfaction, teacher satisfaction and principal satisfaction, and changes in school climate were very positive.
Student academic progress was negligible compared to schools in the community w/o all of the services provided under the Transition Study.
FSC so positively viewed by school system and community that to this day school budget includes six FSC working across the seven schools in the community, now 15 years later.

Reflection on Reason for Use

Once principals were sold their major support was solidified; evaluation results was almost irrelevant but often quoted by principals. FSC solved their major time-killer – work with under-involved or over-involved parents;
Use of narrative stories of principals, parents and FSC provided to school committee and superintendent with graphs and bullets of findings to support the narrative

Example 3 of Evaluation Use

Evaluators do all kinds of things in the course of evaluating programs. Another example comes from evaluation of work with a community HIV/AIDS prevention program that is located in an urban city in the middle of Massachusetts – Worcester has about 180,000 people. The program serves about 1,500 individuals per year. They initiated a new project based on a yearlong needs assessment to provide prevention and referral to treatment for women of color who were at risk for HIV/AIDS and interpersonal violence. They formed a collaborative of health clinic representatives, shelter staff, domestic violence groups, and Red Cross trainers to build the program. During the formative stages, they needed to develop a standardized way to assess women who were at risk of HIV/AIDS and interpersonal violence. There were no existing standardized screening tools available at the time. The evaluators collected screening tools from each of the partners and developed a screening tool with the partners’ full participation and agreement, pilot-tested the screening tool.

After reporting on the results of the use of the screening system, project and partners incorporated it into their intake and referral system.

Reflection on Reason for Use

Evaluator built close relationships with counselors, Director of agency, and program director;
Wrote clear guidelines of how to use screening form;
Reviewed and discussed problems and exceptions throughout project implementation in regularly scheduled meetings;
Could quantify number of women with HIV/AIDS and IPV, and those women with 2, 3 or more risk factors for HIV/AIDS and for IPV separately and combined;

Racial disparities in asthma exist even among children with equal access to health care

2011-02-21T11:19:00.000-08:00

Racial disparities in asthma exist even among children with equal access to health care

Changing for Accountability and Performance

2011-02-07T07:06:00.000-08:00

I attended a conference on Saturday morning in Rhode Island focusing on how medical practices are shaping and innovating on the concept of Patient Centered Medical Home (PCMH). Not new to the American Academy of Pediatrics, PCMH is now cresting on a wave of value-added models of health care practice for both small and large practices.

The Change Curve above was presented at a workshop by Suzanne Houck (of Houck and Associates, Inc.) and Barbara C. Johnson (TransforMED) titled "Adopting a Culture of Innovation."

The takeaway for me? Change is almost always possible in every setting, and it usually follows a predictable pathway, with ups and downs, slow periods and flash flooding moments. Persist, lead and be positive and hopeful along with way to guide colleagues.

The Dragonfly Effect and Evaluation

2011-01-09T04:52:00.000-08:00

"When working to grab attention in a social media campaign, we suggest four design principles: 1)personal: create with a personal hook in mind; 2) unexpected: people like consuming and then sharing new information—draw them in by piquing their curiosity; 3) visual: show, don’t tell—photos and videos speak millions of words; and 4) visceral: design the campaign so it triggers the senses through sight, sound, hearing, or taste." Downloaded from the web on 1/9/2011 at http://www.ssireview.org/articles/entry/the_dragonfly_effect/

This morning a former student of mine sent me an email which described his fear at plunging into the Atlantic ocean in January - and his willingness to do so because he wanted to help raise funds for the Special Olympics (a program for special needs children and adults). He was successful at grabbing my attention and generated a small donation to his cause. He successfully used social media to pique my interest and compelled me to care about his cause.

This approach is catching on and I expect to see more and more social programs (with psychological and healthcare programs following as well) using the "dragonfly effect" to raise funds. Evaluators will have to understand the basic principles and dynamics of this approach to successful social marketing.

Prof. Judith H. Hibbard: The King's Fund Annual Conference

2010-12-09T03:36:00.001-08:00

Check out this SlideShare Presentation:

Prof. Judith H. Hibbard: The King's Fund Annual Conference

View more webinars from The King’s Fund.

Helping Behavioral Health Programs Succeed

2010-03-09T08:00:00.000-08:00

The following announcement was released on March 4th. For more information visit: http://www.hfcm.org/default.asp and click on the Grants tab.

Thursday, March 4, 2010

Boys & Girls Club gets grant

DUDLEY - The Health Foundation of Central Massachusetts has awarded a $175,455 grant to the Boys & Girls Club of Webster-Dudley. This grant will continue the foundation's funding through 2010 for Choices, a comprehensive program to direct at-risk young people toward positive alternatives.

"Choices serves teens by offering a wide variety of programs to teach them how to avoid negative influences and develop healthy habits in their lives. They attend life skills and Internet safety classes, participate in athletics, and receive homework help," said Tony Poti, executive director of the Boys & Girls Club of Webster-Dudley. "We are focused on helping these children develop decision-making and leadership skills, using the most effective and proven programs that have been developed by the Boys & Girls Club of America."

Dr. David S. Robinson, president of DSRobinson & Associates, has been working with the Choices program to evaluate its success and reported very positive results at the grant announcement. "I have studied the key indicators collected on youth who participate in the Choices program, I have come to agree with the Public/Private Ventures research conclusion about the high quality and benefits of Boys and Girls Clubs programs. The youth who participate in Choices programs at the Boys and Girls Clubs of Webster-Dudley show significant gains in social skills, improved attitudes toward school, and positive relationships with peers and adults. All of the Choices participants take advantage of homework help and three out of four teens report that they are doing better in school since involvement in the program," he said.

Scheduling meetings can be difficult and expensive

2009-11-24T06:33:00.001-08:00

In this period of cutbacks on funding and fiscal restraint, holding meetings can be expensive. One way to reduce the cost is to hold online meetings. But how do you schedule meetings when everyone is busy and in different locations? Here are three online meeting schedulers that may be helpful.

Tungle.com is located at www.tungle.com. It is a calendar and scheduler that you can sync with Google calendar or other online calendars.

Doodle.com is a simple online meeting scheduler in which you invite others to log on and each is polled on a list of meeting times that you set. See www.doodle.com

MeetingWizard.com is capable of more complex tasks including sending invitations to participants proposing alternative times, summarizing responses, updates to meeting organizer and sending confirmations. See www.meetingwizard.com

Evaluation Follow-up as a model of Good Evaluation Practice

2009-11-04T06:50:00.000-08:00

Michael Quinn Patton notes the importance of "walking the walk" of good evaluation practice. Here is his most recent post to Evaltalk.

from	MICHAEL PATTON
reply-to		American Evaluation Association Discussion List
to		EVALTALK@bama.ua.edu
date		Wed, Nov 4, 2009 at 9:36 AM
subject		Re: Tools for evaluating the effectiveness of consulting practice?
unsubscribe		Unsubscribe from this sender

hide details 9:36 AM (12 minutes ago)

Jane Davidson wrote: "I know MQP is big on insisting any evaluation contract includes time specifically set aside for formal follow-up. Perhaps he might share some of the questions he asks at that time?"

Here's a surprise: I focus first on issues of use. How were findings used? What has happened to recommendations (if some were generated)? What issues have surfaced in putting findings to use? (This opens up and circles back to how the evaluation was conducted and the kinds of issues Jane astutely surfaced in her report on synthesizing what happened in a set of evaluations -- very important observations, Jane).

Having dealt with use of findings and recs, I turn to process issues: What was particularly useful about the evaluation process? Not useful? Strengths, weaknesses, gaps? What, if any, impacts did the evaluation process have (separate from use of findings, i.e., process uses)?

I don't ask directly about the evaluator's or consultant's value-added, but comments on this inevitably emerge from the focus on findings and process uses.

As regards the value of doing this after the fact, I think of it as walking the talk of evaluation. If we expect programs to evaluate their outcomes and impacts, we need to model (i.e., role model) good evaluation by evaluating our own practice and work. And, of course, it informs our scholarship and inquiries into effective and useful evaluation, which in my case means I get to use the findings in my writings, an added cost-benefit of doing this. :)

Michael Quinn Patton
Utilization-Focused EvaluationSaint Paul, MN

MQPatton@Prodigy.net

Preliminary Analyses: How much to share with clients?

2009-09-30T12:44:00.000-07:00

While working on a new project, I did a preliminary analysis of pre- and post-reading test scores sent to me by examiners in 5 different sites. I ran frequencies and descriptives for each time period and noticed a large difference in only one site out of the 5. I decided to include this preliminary information in a progress report and asked the question - Did Xsite use a different test during this period? No, was the response. She checked on it and found that the one site with much lower scores had reported the data as standardized grade-level scores, not raw scores!

I typically don't report means and frequencies prior to the first draft of the report, but this example has me thinking. I have avoided a problem early on by deciding to sent the preliminary results. But I also wonder, what future problem have I created, if any? Time will tell as this evaluation case example unfolds.

Listening to Program Participants

2009-08-12T07:44:00.000-07:00

Tweeted 8-12-2009: "Evaluators: Do not assume that you know anything about what the program does. Always ask for a description of the services or activities."

Why is it important to listen to program participants describe what they do in a program?

Programs are not made up of a single component or element - a program has a set of goals and objectives (explicit or implicit) that participants are expected to follow. Each participant in a program - the implementers and the current and future beneficiaries of the program - interacts with others within a program according to the roles and expectations mutually emerging in time and guided by the goals and objectives, standards of practice as understood by the culture of the participants in the program, and according to the expectations and values of collaborating stakeholders (funders, citizen boards, communities).

When we listen to participants' narratives about their activities within a program we can hear the ways that the program is operationalized at the level of that participant's field of action. The "field of action" is the unit of interactions within a program that has the highest likelihood of managing change - and change is what programs are designed to manage for participants. Change is the core function of any social or educational program. No matter how goals and objectives are written in print, managing change toward or from some current or future behavior or condition is the critical feature of any program. Think about the budget of any program. Embedded within that budget are the concrete raw materials for making change happen. The largest share of the budget resources most often is the personnel category (or consultants if the program depends on delivery by consulting collaborators). So in terms of program resources and program goals, the management of change is the central focus of programs.

The management of change is embedded in the narrative of those participants closest to the interaction between the provider and beneficiary, and no less so with the program leaders. Each program participant carries within them a "narrative of action" that when verbalized gives the evaluator the clues to the deepest nature of a program. Unleashing these narratives through questioning and listening to participants is the beginning of a process of mutual and collaborative self-reflection that is critical in making program evaluation useful and helpful.

So do not assume to know anything about a program, but always ask each program participant you encounter to describe the work they do. Note the words and narrative revealed by the participant: In what order are activities and services described? What conditions are noted? What kinds of interactions are described? What links to the community are identified? Who is doing the action - provider or beneficiary? How is hopefulness and hopelessness conveyed? What challenges are identified? What facilitators are identified? The narrative of the participants will become an evaluator's most useful tool to make program evaluation useful and helpful.

Father's Day 2009 Post

2009-06-20T04:54:00.000-07:00

What do researchers say fathers contribute to children and child development? My take on this is - Dads, you better start talking to your babies and toddlers early and often! Here are a selection of findings for reflection.

Fathers and social development. Fathers who tended to hold their babies facing out reported that they did this because “it’s easier for the baby to see the world.” In over 20 years of studying fathers, Dr. Ross Parke states that men have a tendency to allow their infants more freedom toexplore, which appears to support the development of independence in their children. Fathers who actively play with their children appear to reinforce the notion of emotional self-control in their children and help their children learn to recognize the emotional cues of others. (http://fcs.tamu.edu/families/parenting/fathering/fathering_pdf/development.pdf)

A noted sociologist, Dr. David Popenoe, is one of the pioneers of the relatively young field of research into fathers and fatherhood. "Fathers are far more than just 'second adults' in the home," he says. "Involved fathers bring positive benefits to their children that no other person is as likely to bring."⁶ Fathers have a direct impact on the well-being of their children. It is important for professionals working with fathers—especially in the difficult, emotionally charged arena in which child protective services (CPS) caseworkers operate—to have a working understanding of the literature that addresses this impact. Such knowledge will help make the case for why the most effective CPS case plans will involve fathers. (http://www.childwelfare.gov/pubs/usermanuals/fatherhood/chaptertwo.cfm)

However, there was one surprising finding: while mothers’ depression reduced the mother-child reading activities, and similarly, fathers’ depression reduced the father-child reading activities, it was the fathers’ depression (but not mothers) that significantly affected the child’s vocabulary development at 24-months. Why? The authors suggest that even though maternal depression reduced mother-child reading time, the mother stills spends a significant amount of time with the child, so that such reduction is not likely to have a major effect on the child’s development. (http://www.child-psych.org/2009/03/baby-blues-fathers-postpartum.html)
Fathers influence child language development more than mothers: CHAPEL HILL - In families with two working parents, fathers had greater impact than mothers on their children's language development between ages 2 and 3, according to a study by the University of North Carolina at Chapel Hill's Frank Porter Graham (FPG) Child Development Institute and UNC's School of Education. Researchers videotaped pairs of parents and their 2-year-old children in their homes during playtime. The children whose fathers used more diverse vocabularies had greater language development when they were tested one year later. However, the mothers' vocabulary did not significantly affect a child's language skills. "Most previous studies on early language development focused on mothers," said Nadya Panscofar, a graduate research assistant and an author of the study. "These findings underscore that for two-parent, dual earner families, fathers should be included in all efforts to improve language development and school readiness." (http://www.unc.edu/news/archives/oct06/fpgfatherslanguage103006.htm)

Be sure to read the Boston Globe article about Helen Reinherz

2009-04-27T13:11:00.000-07:00

A lingering cloud

A study that began more than 30 years ago in Quincy shows that family arguing leaves a long-lasting imprint on children

http://www.boston.com/lifestyle/family/articles/2009/04/27/a_lingering_cloud/

Reflections on Managing Client Requests for Evaluation Data Before Contract Ends

2008-08-08T07:17:00.000-07:00

Here is what happened:

I heard from one colleague on the EVALTALK listserv who thoughtfully recommended that I revise the contract, if it did not contain language clarifying access to data by the client. I thought this was a good idea going forward anyway. Then I made a list of the benefits to the client to have access, the negatives of granting access, and the benefits and negatives of granting access for my professional obligation to my client. I drafted a memoranda in which I proposed some language that would clarify my expectations of the use of the data by my client. Here is what I said:

a. The external evaluation contract provides that analyses and reports based on findings from data collected during the program year require that the evaluation company must conduct the analyses and reporting on company computers.

b. The external evaluator could not verify that the integrity of the data and any analyses conducted by the client, and, therefore, could not use any data set prepared by the client in the external evaluation.

c. The consent form signed by providers and parents prohibits the external evaluator from providing individually identifiable information to anyone outside of the evaluators and evaluation company.

d. The client understands and agrees to the condition that the external evaluator may not use data files derived from an analyses of the (information provided) for the external evaluation, unless joint analytical plan is authorized.

e. The client will use the data provided only for internal purposes, and not submit reports derived from the analyses outside of the internal evaluation staff.

e. By agreeing to these conditions, the client will make no changes to the budget based on providing the information requested.

At a subsequent meeting with the client, after reviewing my draft proposal, the client agreed that sharing anonymous data would not help them, and that the conditions outlined by my draft proposal are appropriate. They withdrew the request, and we proceeded to move forward on the evaluation plan, amicably and on a sound relationship.

During the meeting, I also learned that the request was made to see if there was some way to assist me in making the tight deadlines that were required for the grant, and not for any other reason that may have been imagined. Because of my efforts to remain as transparent as possible - usually expressed to them through details about the status of the data collection, and my transformation of my evaluation infrastructure to comply with the evaluation requirements - they were becoming anxious about my capacity to meet the contract obligations. While they valued my transparency and honesty, I had made the client managers more anxious over time.

The lesson for me is to consider the client's capacity for reception of information (transparency) about the process of an external evaluation - while transparency models the way the evaluator wants the client to communicate, the side effect may by an erosion of confidence in the capacity to deliver on the contract.

This experience is also a confirmation of the principle that challenges are really opportunities to improve and learn to be better.

Internal & External Evaluation Collaboration Questions

2008-07-31T05:26:00.000-07:00

Dear Colleagues,

I was recently asked by a major client to provide them with the pre-test data set from an external evaluation of their program, just as I was concluding the post-test data collection. I was hired to provide an external evaluation (with contract). I am a small evaluation firm, a sole proprietor, and I did expect to provide them with all of the data after the conclusion of the study (after the evaluation reports are submitted and approved and final payment is received). I was hired to conduct the evaluation based on a federally funded grant that requires an evaluation, but not necessarily an external evaluation. The external evaluation was part of their accepted application to the federal agency. One other note, the director who hired me has recently resigned and the program coordinator with whom I have been working with now for over a year, was hired to replace the director.

The back story is that they planned to increase their research capacity, which I supported, and consequently, hired a very competent research analyst, who now has the time and competence to conduct their own internal evaluation. Without getting into the politics and personalities, which, I acknowledge plays an important part in this little drama, I am curious about the approach and opinions of other evaluators who may visit this blog.

We live in an urban environment in the Northeast where the poor outcomes of children in public education is a very serious matter, and the stakes are high for the program and the agency which runs the program. The agency has a very positive reputation in the community. With one year left on the evaluation, and with the limited structure of my research firm, I am tempted to "collaborate" (assuming I have a choice in this matter), and hand over the pre-test data set to the in-house evaluator for preliminary analyses. (There are many sub-evaluation questions we planned for in which in-house data and external data would be combined anyway.) But I also am obligated to protect the agency and program from potential criticism of the results of the study, in light of the challenge to objectivity that may emerge from an arrangement in which the in-house evaluator has prior access to pre-test data.

Assuming for a moment that the contract allowed or was passive on data access and integrity, what advice would you offer, if you were inclined to collaborate with your client. What advice would you offer if you were not so inclined? What safeguards or protections would you impose in a written agreement with your client so that some mutually beneficial compromise would emerge?

I am sending this post to the EVALTALK listserv, and later I will post some of the comments I receive.

Thank you in advance for your feedback.

David

Differentiating Program Evaluation Logic Terms

2008-04-16T06:37:00.000-07:00

Thanks to EPA Administrator and blogger, Marcus Peacock, and the EVALTALK contributors, here is an interesting and useful way to distinguish between the terms used in many common forms of logic models (The Flow of the River, http://flowoftheriver.epa.gov/my_weblog/2007/10/getting-juiced-.html)

Nectarine Juice Production Program

Performance Measurement

The inputs are nectarines and a hammer
The activity is smashing nectarines with a hammer
The output is how many nectarines got smashed
The outcome is how much juice resulted
The impact is the improved health of the people who drink nectarine juice

Program Evaluation

How well is the program producing nectarine juice using the current inputs?
Is a hammer really the most effective intervention for producing nectarine juice? What are the alternatives?
What is the nectarine juice production program’s specific contribution to the improvement of national public health?

I'm not a devoted fan of juice, but the example helps me understand the way people may use the terms. The program "Juice Production Initiative" further illustrates that programs have both resources (objects) and activities (processes) to organize and make use of for a desired end.

Selecting an Evaluator - A Primer

2006-10-24T02:42:00.000-07:00

Notes Regarding Professional Evaluation Work and Evaluation Reporting

David S. Robinson
www.evaluationhelp.com

If I were interested in employing an evaluator or evaluation firm, I would be interested in how responsive the evaluator has been to his clients with respect to evaluation needs.

Evaluator Checklist
• Can he/she complete an evaluation report in a timely manner?
• Is he/she respectful of the program and the complex environment or context in which it operates?
• Does he/she listen or is he single minded about the way evaluation must be done?
• Does he/she understand the concepts behind the program, the program theory or logical connections between the intervention and the outcomes of the intervention?
• Does he/she write well? Is he conscientious and detail oriented?
• Does he/she stay focused or wander in his questions and contributions?
• Does he/she search out seminal articles that are research oriented and share them with the program staff as appropriate?

Brief Background
I was the Director of Research and Evaluation for MSPCC for 24 years. I have been teaching research and practice evaluation since 1992 and I continue to assist the Simmons College School of Social Work in several ways. I have had a federal grant from the Head Start Bureau (1998 - 2001) and was the principal investigator on the study. I helped design the practice evaluation course at Simmons College. I consult to community organizations in the Boston area as part of the School of Social Work Research Institute. I started DSRobinson & Associates in 2004 and currently consult to medical training programs, public health initiatives, and child welfare programs in Massachusetts and Rhode Island.

My Evaluation Practice
I am trained in Empowerment Evaluation, and have been a member of the American Evaluation Association since 1990, and a current member of APA (American Psychological Association). Here are some points that I would consider noting about my experience and work.

1. Before beginning to write an evaluation plan, I tend to ask many questions, or to re-state what I have heard or read about the program or similar programs, always trying to clarify what the program director (or key leader) is attempting to do. I may offer up prior research articles in similar areas, or I may contribute program evaluation articles to clarify the kinds of things that evaluators do, define new terms, suggest articles or people working in the same or similar area. All of this preliminary work, which may take more than one meeting, is designed to explore with the director an evaluation approach that will be helpful, answer important questions, help strengthen or improve the program, contribute to scientific knowledge if the field, and be a positive learning experience for program participants.

2. After learning as much as I can about the program theory and actual intervention activities, I try to establish a logic model or conceptual model of the program graphically and in text so that all of the key participants can comment on it, revise it, think about the intervention elements that are linked to outcomes, "evaluate" how accurately I have summarized the program and context. This phase of the evaluation is participatory and often empowering for program directors, staff and participants. Sometimes this is experienced as frustrating to some – “Enough discussion, already!” may be expressed. A little bit of frustration with the speed of evaluation planning is healthy. If I hear it repeatedly, I quickly make an initial plan on paper, before I feel ready to do so, to help the program members feel that progress is being made, and to give me a little more time to assess evaluation readiness.

3. Evaluation Design. The formulation of the evaluation design is a key topic with me, and, I think, sets the tone and conditions for the most important evaluation work ahead. I know that most program directors want their program to be accepted by their colleagues as "evidence-based" and they are aware that achieving this level of reception is rare and challenging. I try to be forthright with them that achieving the status of a "science-based" program is a long and arduous road with many obstacles in the real world. Firstly, I emphasize the strongest possible experimental design that will fit the context of the program, and discuss the trade-offs that will accompany each decision to move toward more quasi-experimental or exploratory designs. Secondly, I help the program leaders think about the best design for their objectives, and try to give them the tools (words and attitudes) to help them create a culture of self-reflection and professionalism worthy of the scientific method. But I also emphasize the value of qualitative methods best used to capture the larger context of the program, and the peculiar circumstances of implementing the program in its setting. What do participants say about the program? How does the program appear to the interveners? Where did the intervention vary from the plan? These are some of the questions I raise as I incorporate the answers into the evaluation design.

4. I am careful to research standardized measures for the outcomes envisioned by the conceptual model of the program, but I am open to consider indicators suggested by the program participants. I will give my advice about why one approach is more explainable, or will be more acceptable to the scientific community, but I am flexible on this. I believe that many of the most powerful and innovative aspects of programs are not adequately captured by existing standardized instruments. I am willing to create new items and modify them according to the needs and circumstances of the setting and the characteristics of the participants.

5. I carefully develop data collection methods that fit with the participants, and make use of innovative collection methods that are consistent with the characteristics and values of the participants. I will use computer-based on-line methods or paper and pencil methods - to ensure that all participants have an opportunity to respond. I often make suggestions for incentives, because I believe that participants' time is valuable and should be compensated whenever I ask them to do some work. I am prepared to write applications for human subject protection at the sponsoring institutions (IRB Committee applications vary but essentially are consistent with the federal IRB guidelines located at http://www.hhs.gov/ohrp/ , and will write and revise informed consent documents as needed. I will prepare responses to outside commentaries by experts who have suggestions for improvement - and make those changes even at the last minute when called upon.

6. Reporting Results. I am careful to check the quality of the data before initiating analyses. I use a step-wise analytical strategy, first analyzing frequencies for each question, and then move to more and more complex analyses. I then review each evaluation question, and attempt to answer each evaluation question using appropriate statistical analyses, often combining questions (or items) into compound constructs (combined variables) to allow for more sophisticated analyses appropriate for the question. (I have found that the simplest appearing evaluation questions are often the most complex.) I write readable reports tailored to the audience, professionally documented that include background, organizational context, literature review, methods, results, conclusions and recommendations, and abstract or executive summary. I try to include text and graphic representations of the results whenever possible to make the report attractive and understandable to different kinds of readers.

Four Cornerstones of Assessing Evaluation Needs

2006-10-01T05:07:00.000-07:00

Four Cornerstones of Assessing Evaluation Needs
David S. Robinson, Ed.D.
October 1, 2006

Beginning Evaluators are often unsure about how to begin to assess an organization and determine evaluation needs. This is especially true when the student of evaluation is inside the organization - an employee of the organization that is the subject of an evaluation initiative. So what are the four keys to beginning an assessment of an organization and its evaluation needs? Here are my suggestions.

1. Be positive and confident that there are relevant articles, web sites, reports and digital information available and accessible out there to begin to explore the major mission of the organization. Many students report that after an "extensive" literature search, they cannot find anything relevant to their evaluation question or issue. My experience is that this more often results from an initial skepticism or negative attitude about the topic which is fulfilled while searching. Keep an open mind and be sure to examine the biases (through self-reflection) that reduce the likelihood of finding relevant literature about the topic. Key one: be positive and confident that you will find relevant literature about the organization and its mission, programs and services, goals and objectives.

2. Pay attention to how the organization describes itself in written and published literature, or in interviews with key stakeholders. How is the organization, program or service described by administrators, board, staff and clients? Is it only positive, sticking to the bright and polished aspects of the organization, or does it include the scuff marks and challenges? Is it balanced or one-sided? Key two: Shine or scuffed - members of an organization will tell you how ready they are to evaluate their programs and services by how open they are to objective observations.

3. Pay attention to the bright and shiny aspects of the organization, program or services that members repeatedly talk and write about. These are the best places to begin, because they are the easiest to recognize and an initial evaluation will more likely meet with openness and generous sharing in areas that are commonly recognized as positive and successful. Key three: Pay attention to the low hanging fruit.

4. Review the budget and table of organization and observe the different groupings of programs and services within an organization - the different beachheads will vary as to size and scope, and these groupings should match the budget. Pay attention to these groupings because they will lead you to ask questions like: Does the allocation of resources (budgeted programs) match the organizational goals and staffing? Do some groups receive a disproportionate amount of resources? Do the groupings and staffing resources match the strategic goals and objectives? Key four: What are the different "beachheads" and "shores" of the organization? How do those beachheads compare with the resources, staffing, mission, goals and objectives?

An evaluation initiate can use the four keys to evaluation needs and find his or her way to a beginning dialogue with leaders and colleagues thereby starting an evaluation that fits the organizational profile of evaluation readiness.

Web Sites for Learning About Best Practices

2005-05-10T14:58:00.000-07:00

I thought I'd pass on a useful list of web sites for finding out more about best practices. This list was compiled by Karl Hamner and forwarded to the Evaltalk Listserv. Hope you find it useful. David

On-Line Best Practice Resources

Mental Health & Substance Abuse: SAMHSA’s National Registry of Effective Programs (NREP) provides effective substance abuse and mental health programs for every community at http://modelprograms.samhsa.gov/.

The Department of Justice has a wide variety of youth violence prevention resources on its well-organized site at http://www.usdoj.gov/youthviolence.htm.

The National Institute on Alcohol Abuse and Alcoholism (NIAAA) is committed to helping colleges and universities reduce alcohol-related problems on their campuses, protect students from harm, and improve quality of life for the entire campus community. To achieve this, NIAAA has created the resource Reducing Alcohol Problems on Campus: A Guide to Planning and Evaluation.
This is available free at http://www.collegedrinkingprevention.gov/Reports/Handbook/default.aspx.
Other related and useful publications can be found at:
http://www.niaaa.nih.gov/publications /guides.htm.

The Department of Education (DOE) has published Identifying and Implementing Educational Practices Supported By Rigorous Evidence: A User Friendly Guide. This guide provides K-12 educational practitioners with user-friendly tools to distinguish practices supported by rigorous evidence from those that are not. It is available at http://www.ed.gov/rschstat/research/pubs/rigorousevid/index.html. More related resources from The National Center for Education Evaluation and Regional Assistance (NCEE) are at http://www.ed.gov/about/offices/list/ies/ncee/index.html.

The What Works Clearinghouse web site, http://www.whatworks.ed.gov, by the U.S. Department of Education's Institute of Education Sciences, gathers together current best practices in education. The introduction reads: “On an ongoing basis, the What Works Clearinghouse (WWC) gathers studies of the effectiveness of educational interventions (programs, products, practices,
and policies). We review the studies that have the strongest design, and report on the strengths and weaknesses of those studies against the WWC Standards so that you know what the best scientific evidence has to say.”

Karl Hamner
www.kmhconsulting.com

Do You Know What a Summative Evaluation is?

2004-12-27T10:16:00.000-08:00

Robert Stakes

"When the cook tastes the soup, that’s formative; when the guests taste the soup, that’s summative."

Summative Evaluation: The assessment or judgment of the value of a program, product, service or educational intervention in relation to similar programs, products, services or educational interventions. The judgment of value is also based on the expected changes in the target population and based on a theory about how the program, product, service or educational intervention is supposed to work. (David S. Robinson, 2004)

Short List to Build Evaluation Culture in Organizations

2004-12-05T16:56:00.000-08:00

There is nothing harder than starting out to evaluate a new program. Well, actually it is easier to evaluate a new program than to evaluate a program that has already been running for a time. Here is a short list of things to consider when beginning to evaluate a program - something to get you started:

1. Start with the leaders and gain their approval - as high up as you can go.
2. Then double back and gain approval from the line staff, the personnel who do all of the work, and who will probably provide most of the information for the evaluation.
3. Learn as much as you can about the work culture and the details of the activities that people do.
4. Use some form of structured logic modeling to organize what you learn about the organization, and detail the goals and objectives of the program, the resources used, the activities enacted to meet the goals and objectives, the direct products (or outputs) of the activities, and the immediate, intermediate and long-term outcomes.
5. Brainstorm what would be an observable indicator for each objective and outcome.
6. For each indicator establish a data source and collection method with a schedule and responsible personnel identified.
7. Share with leaders and line staff and revise, revise, and revise until everyone is sick of it or agrees it is the best plan.
8. Congratulations, you are well on your way if you have gone this far.