Metrics versus Stories: What Matters Most?

In my work with clients on large projects, I've seen the importance of stories over and over again. It's always hard to break down how we should use stories in our work, but they are everywhere—in the way you communicate with your team, with your clients and outsiders, and even internally within your company. 

  What do metrics measure? 

 Metrics are a way to measure the progress of a project. They can be quantitative or qualitative, and they're often used to measure the effectiveness of a project. 

  Quantitative metrics – These are numbers that describe how something is doing (like changes in standardized clinical assessments). 

 Qualitative metrics – These are words that describe how something is doing (like levels or scores on satisfaction questionnaires). 

 Qualitative metrics can be useful for determining the effectiveness of a project, but they're often subjective and difficult to compare with one another. Quantitative metrics are better for measuring progress because they allow researchers to see how something is doing over time and use objective criteria like whether a patient meets certain clinical standards. 

  What does the story tell? 

 As an agency, you’re going to have the opportunity to tell stories. Stories are about people, not just numbers—they're about the impact of your project and how you overcame challenges. A good story can do more than just impress potential clients and funders; it will also help them understand what makes you different from other agencies providing similar services.. 

We've found that when we share our successes with clients or funders or policymakers, they're much more likely to trust us enough to want us as their partner on future projects (and vice versa). If you’re looking for a way to stand out from the crowd, then it's time to start telling stories. And don't forget: if you're going to tell your story, make sure it's memorable! Putting it together. You can use metrics to tell the story of how your project is doing, and stories to tell the readers of its success. 

The best way to understand this is with an example: imagine you’re working on a new treatment approach for substance use disorders. You've trained your clinicians on the new approach, tested it with potential clients, and have seen great results so far—but there's one problem: the community is not making referrals to your new program! 

If you were using storytelling as your method for communicating this information with others in the organization (and perhaps even outside), here are some ways we might do things differently: 

We could use metrics to tell the story of how our new service is doing. For example, we might say: "Our new approach has increased client engagement and satisfaction with services by 20% since we began using this new approach." This is useful information for many people in the organization, but it doesn't explain why or how that happened. 

We could use stories to describe how our new product is successful. For example, we might say: "Satisfaction with our new approach to SUD treatment has increased by 20% since we started using the new approach—and all because of one client who loved it so much he wrote a letter to his friends telling them about how much better it made their lives." This tells us not only what happened (increased satisfaction) but also why and how (this one client spreading the word). 

Both metrics and stories are important to tell the story of success for your project. Both metrics and stories are important to tell the story of success for your project. Metrics show how much progress you are making toward your goals, while stories convey what those goals mean in practical terms. The difference is that metrics are quantitative, while stories are qualitative. 

Both metrics and stories should be used to maintain focus on what matters most: How can we create something valuable for our clients? What makes us unique? When you are struggling to get work done, it can be tempting to focus on metrics. After all, metrics are easy to measure and track—they give us a clear idea of what we’ve accomplished. But metrics can also distract us from the larger picture; they can make us think that our project is further along than it actually is. 

We know that there’s a lot to cover as far as metrics and stories go, but we hope this post has helped you understand the importance of both. For example, if you’re using a story to explain your project or business model, consider using numbers as well so that people can see concrete evidence of success in their own lives.

David S. Robinson, Ed.D (Master of Education: Harvard Graduate School of Education; Doctor of Education: Harvard Graduate School of Education)

 Contact:

Mobile: 617-733-5979

Email: davidr46@gmail.com

Address: 11 Round About Way, Greenville, SC 29609

Slide Presentation: https://www.slideshare.net/davidr46

Ask A Question Here: https://forms.gle/JuwQiW1gAmGJ3huD7

Program Evaluation in an Era of Equity

Program Evaluation in the Era of Equity: What this means for your work

Image Source: FreeImages

‍

The equity agenda has the potential to shape how we think about and evaluate programs for years to come. In response, program evaluators are asking new questions, developing novel techniques, and adopting new measures of success that align with equity principles. Program evaluation is a set of processes used to assess the effectiveness of programs or interventions in achieving their intended outcomes. Program evaluation can help us understand what is working and what isn’t working so that we can make informed decisions about whether to continue or end the program or take other actions. Programs often struggle with these questions: Are we reaching the people who need our services? Are they using them? Are they seeing positive impacts as a result? And if not, why not?

What is equity?

Equity is the idea that all people deserve to be treated fairly, with dignity and respect, regardless of their differences (e.g., income, race, ethnicity, gender, sexual orientation, language, religion, immigration status, disability status, etc.). In the context of program evaluation, equity is the extent to which a program is meeting the needs of the people it intends to serve or the degree to which a program is having positive impacts on its intended beneficiaries. Equity is a core principle developed for the field of evaluation by the W.E.B. Du Bois Institute for Social Research at Harvard University and the Evaluation & the Public Program (EPP) at the University of Washington. Equity is a broader framework for evaluating programs that goes beyond questions of impact and effectiveness to also include issues of fairness, inclusivity, and participation.

Why is equity important in program evaluation?

Programs that are truly successful expand beyond their initial reach and impact. For this to happen, we need to know who is being left out, why, and what we can do about it. Program evaluation can help answer these questions. But to do so, we must ensure that the evaluation is designed with equity principles in mind. Equity-oriented program evaluations include methods that enable us to identify and understand who is being left out and why, and also provide information on how to better include those people. A common problem in program evaluation is that people who are not served by a program are less likely to participate in the evaluation. This means that their perspective and experiences go missing from the evaluation, resulting in an incomplete and potentially biased picture of the program. There is growing recognition that many of the traditional methods used in program evaluation create significant barriers for people who are left out by the process. Equity-oriented program evaluation methods are designed to reduce or remove these barriers.

Key questions for evaluating equity-oriented programs

- In whose name are we doing this work? Whose experiences and perspectives are shaping our ideas, discussions, and decisions? Whose needs and experiences are we trying to address? - For whom is this work missing? Who is not being reached by the program? Who is not participating in the evaluation? Why are they missing? - What are the barriers that prevent people from participating? Are there ways to remove the barriers to participation?

Novel methods for evaluating equity-oriented programs

- Focus groups with people who are not participating in the program evaluation: One method for addressing the first question above is to invite people who are not participating in the program evaluation to participate in a focus group discussion. This provides an opportunity to learn more about their experiences and perspectives. People who have not been participating in the evaluation are often those who are being left out. - Mixed methods approaches: Mixed methods approaches combine quantitative and qualitative methods, such as surveys and interviews, to provide a more complete and holistic understanding of program impacts and outcomes. - Engagement-oriented evaluation approaches: Engagement-oriented evaluation approaches involve meaningful participation from program participants in designing, conducting, and analyzing the evaluation. Through these approaches, people who are served by the program play a greater role in the evaluation, creating a more comprehensive and complete picture of the program’s impacts and outcomes. - Creative and participatory methods: Creative approaches to program evaluation expand the ways in which the evaluation is done, including the use of art, poetry, and other creative methods. Participatory methods involve participants in designing and conducting the evaluation. Both creative and participatory methods are intended to broaden the reach of the evaluation by making it more accessible and open to a broader range of people and perspectives.

Conclusion

Programs are most effective when they reach a broad range of people and address their needs and experiences. Equity-oriented program evaluation can help us identify who is being left out and why, and suggest ways to more fully include these people. As the field of program evaluation evolves, we are seeing increasing interest in equity-oriented evaluation approaches. This is welcome and necessary because the equity agenda has the potential to shape how we think about and evaluate programs for years to come. In response, program evaluators are asking new questions, developing novel techniques, and adopting new measures of success that align with equity principles.