Local Wisdom, Lasting Change: Empowering Communities Through Grounded Program Evaluation

Abstract

A local craftsman built a pavilion from local storm-felled trees, demonstrating resourcefulness and sustainability. This pavilion serves as a metaphor for grounding program evaluation in local resources and knowledge. Utilizing local stakeholders in program evaluation taps into contextual knowledge often missed by external evaluators. Empowering community members to conduct evaluations fosters ownership and engagement with the process. Effective evaluations, like the pavilion, acknowledge limitations and build on lessons learned. Strategies for enhancing program evaluation include engaging stakeholders, utilizing local resources, and focusing on sustainability. Building evaluation capacity within communities creates a lasting impact and ongoing improvement. The post advocates for valuing local wisdom and resilience in program evaluation, leading to more meaningful and sustainable outcomes.

Introduction

During a recent visit to Low Country, South Carolina, I stumbled upon an unexpected lesson in resourcefulness and sustainability. Amidst the moss-draped oaks and tidal marshes, I met a local polymath—an engineer by training but a craftsman by passion. On his property outside of DC stood a magnificent pavilion, its wooden beams rising gracefully toward the sky. What made this structure remarkable wasn't just its aesthetic appeal, but its origin story: the entire pavilion was constructed from storm-felled trees, local timber milled on-site, and assembled with ingenuity born from necessity.

As I admired the pavilion's elegant design and sturdy construction, I couldn't help but see parallels to my work in program evaluation. Too often, we rely on imported frameworks and external expertise when the most sustainable and impactful solutions might be growing right around us. This pavilion—born from local resources, skills, and knowledge—offers a powerful metaphor for how we might approach program evaluation in a more grounded, sustainable, and ultimately more effective way.

The Pavilion Story – A Lesson in Ingenuity and Sustainability

The polymath brother (as his family affectionately called him) wasn't just any engineer. With degrees in mechanical engineering and architecture, and years spent working on complex infrastructure projects, he approached problems with both technical precision and creative vision. When Hurricane Matthew swept through the area in 2016, leaving dozens of mature hardwood trees scattered across a local popular park, he saw not devastation but opportunity.

Instead of hiring contractors to remove the fallen trees and import materials for a new gathering space, he set up a portable sawmill and began transforming chaos into creation. Oak became support beams, cypress was milled for weather-resistant flooring, and various hardwoods were carefully selected for different structural elements based on their natural properties. The pavilion took shape over months, designed to withstand future storms while honoring the character of the landscape it came from. Local neighbors came together to collaborate on and help rebuild the pavilion.

What struck me most was how this approach—using what's available, respecting local conditions, and applying appropriate expertise—created something more harmonious with its environment than any prefabricated structure could have been. The pavilion wasn't just built on the land; it was built of the land, embodying a profound lesson in working with, rather than imposing upon, the existing environment.

Drawing Parallels to Program Evaluation

This pavilion-building approach mirrors what the most effective program evaluations strive to achieve. Instead of imposing standardized evaluation frameworks that often fail to capture local nuances, what if we invested in developing evaluation skills among program participants and community members?

When we empower local stakeholders to design and conduct evaluations, we tap into invaluable contextual knowledge. These individuals understand the subtle cultural factors, historical contexts, and community dynamics that external evaluators might miss. Just as the polymath brother knew which wood would resist local insects and which would withstand humidity, community members know which questions matter most and how to interpret responses correctly.

Building evaluation capacity from the ground up also fosters ownership. When participants see evaluation as their tool—not something imposed from outside—they're more likely to engage meaningfully with the process and act on the findings. The evaluation becomes not just an assessment but an integral part of the program's growth and development, much like the pavilion became not just a structure but an expression of resilience and adaptation.

Lessons from the Pavilion Builder – Accomplishments, Errors, and Reflections

The finished pavilion stands as a testament to what locally-sourced solutions can accomplish. It has weathered subsequent storms, hosted countless gatherings, and become a landmark that visitors admire. Its functionality exceeds what commercially available options might have provided, precisely because it was designed with intimate knowledge of local needs and conditions.

Yet the builder was quick to point out his missteps: "I didn't account for how the different woods would expand at different rates," he admitted, showing me where he'd made subsequent adjustments. "And this section took twice as long as necessary because I was learning the technique as I went."

These admissions weren't signs of failure but rather reflections of a healthy learning process—one that parallels effective evaluation practices. Good evaluations acknowledge limitations, document lessons learned, and build on previous experiences. They recognize that perfection isn't the goal; improvement is.

Perhaps most insightful was his reflection on the process: "The best part wasn't finishing it—it was figuring out how each unique piece of wood could contribute to the whole." This perspective mirrors the value of inclusive evaluation approaches that recognize how diverse perspectives combine to create a more complete understanding of program impacts.

Practical Strategies for Enhancing Program Evaluation Using Local Knowledge

How can we bring this pavilion-building metaphor to program evaluation? Several strategies stand out:

Engage Stakeholders Meaningfully: Involve program participants not just as data sources but as evaluation designers and analysts. Their questions often lead to the most relevant insights.

Utilize Local Resources: Map existing skills and knowledge within your community before seeking external expertise. Sometimes the perfect evaluation "timber" is already in your backyard.

Focus on Sustainability: Design evaluation practices that can be maintained beyond initial implementation. Simple, repeatable methodologies often yield more consistent insights than complex approaches that collapse without expert guidance.

Embrace Flexibility: Allow your evaluation framework to adapt to changing circumstances, just as the pavilion builder adjusted his techniques to work with the unique properties of each tree.

Building Something That Lasts

The pavilion stands as a physical manifestation of sustainable design—a structure that will serve its purpose for generations because it was built with deep understanding of local conditions and materials. Similarly, evaluations built on local knowledge and skills create lasting impact.

When we invest in building evaluation capacity within communities and organizations, we don't just get better data for one assessment—we create an evaluation mindset that continues to generate insights long after external consultants have moved on. Program participants empowered with evaluation skills become agents of ongoing improvement and adaptation.

Like the storm-felled trees transformed into a beautiful pavilion, challenges in programs can become opportunities for growth when viewed through an evaluative lens that values local wisdom and resilience.

Conclusion

The polymath brother's pavilion reminds us that our best resources often lie right beside us in the neighborhood—we need only the vision to recognize them and the skills to transform them. In program evaluation, this means looking first at the knowledge, experiences, and capabilities within the communities we serve before importing external frameworks or expertise.

By building evaluation capacity locally, tailoring approaches to context, and valuing diverse perspectives, we create evaluations that are not just more accurate but more meaningful and sustainable. Like a well-built pavilion, these evaluations stand the test of time, providing shelter for better decision-making and program improvement for years to come.

The next time you approach program evaluation, ask yourself: What trees have already fallen that I might build with? What local knowledge might I mill into something useful? The answers might just lead to evaluation practices as beautiful and enduring as a hand-crafted pavilion rising from the South Carolina lowlands.

Difference between surveillance and program evaluation

What is the difference between research and program evaluation?
In public health program evaluators have only recently been introduced into the mix of public health personnel. The implementation of evaluations of public health interventions have increased over the past 5 years. There has been some confusion or confounding of the role of the public health surveillance systems for program evaluation. But this is not an effective use of the public health epidemiologist when they are called upon to provide program evaluation. Program evaluation has different purposes and stages from surveillance research activities. The following document provides a detailed distinction between research and evaluation. Although there continues to be language in this document that contributes to the confusion.

Foundational Elements for Program Evaluation Planning, Implementation, and Use of Findings, Steps to a Healthier US Cooperative Agreement Program from:

Prev Chronic Dis. 2006 January; 3(1): A19.

Published online 2005 December 15.

See: PubMed Central, Table: Prev Chronic Dis. 2006 January; 3(1): A19. Published online 2005 December 15.

The Dragonfly Effect and Evaluation

The Dragonfly Effect and Evaluation

"When working to grab attention in a social media campaign, we suggest four design principles: 1)personal: create with a personal hook in mind; 2) unexpected: people like consuming and then sharing new information—draw them in by piquing their curiosity; 3) visual: show, don’t tell—photos and videos speak millions of words; and 4) visceral: design the campaign so it triggers the senses through sight, sound, hearing, or taste." Downloaded from the web on 1/9/2011 at http://www.ssireview.org/articles/entry/the_dragonfly_effect/ This morning a former student of mine sent me an email which described his fear at plunging into the Atlantic ocean in January - and his willingness to do so because he wanted to help raise funds for the Special Olympics (a program for special needs children and adults). He was successful at grabbing my attention and generated a small donation to his cause. He successfully used social media to pique my interest and compelled me to care about his cause. This approach is catching on and I expect to see more and more social programs (with psychological and healthcare programs following as well) using the "dragonfly effect" to raise funds. Evaluators will have to understand the basic principles and dynamics of this approach to successful social marketing.

9 months ago

- About The Dragonfly Effect (November 18, 2010) | Stanford ... - ssireview.org

The Dragonfly Effect and Evaluation

"When working to grab attention in a social media campaign, we suggest four design principles: 1)personal: create with a personal hook in mind; 2) unexpected: people like consuming and then sharing new information—draw them in by piquing their curiosity; 3) visual: show, don’t tell—photos and videos speak millions of words; and 4) visceral: design the campaign so it triggers the senses through sight, sound, hearing, or taste." Downloaded from the web on 1/9/2011 at http://www.ssireview.org/articles/entry/the_dragonfly_effect/

This morning a former student of mine sent me an email which described his fear at plunging into the Atlantic ocean in January - and his willingness to do so because he wanted to help raise funds for the Special Olympics (a program for special needs children and adults). He was successful at grabbing my attention and generated a small donation to his cause. He successfully used social media to pique my interest and compelled me to care about his cause.

This approach is catching on and I expect to see more and more social programs (with psychological and healthcare programs following as well) using the "dragonfly effect" to raise funds. Evaluators will have to understand the basic principles and dynamics of this approach to successful social marketing.

Listening to Program Participants

Tweeted 8-12-2009: "Evaluators: Do not assume that you know anything about what the program does. Always ask for a description of the services or activities."


Why is it important to listen to program participants describe what they do in a program?

Programs are not made up of a single component or element - a program has a set of goals and objectives (explicit or implicit) that participants are expected to follow. Each participant in a program - the implementers and the current and future beneficiaries of the program - interacts with others within a program according to the roles and expectations mutually emerging in time and guided by the goals and objectives, standards of practice as understood by the culture of the participants in the program, and according to the expectations and values of collaborating stakeholders (funders, citizen boards, communities).

When we listen to participants' narratives about their activities within a program we can hear the ways that the program is operationalized at the level of that participant's field of action. The "field of action" is the unit of interactions within a program that has the highest likelihood of managing change - and change is what programs are designed to manage for participants. Change is the core function of any social or educational program. No matter how goals and objectives are written in print, managing change toward or from some current or future behavior or condition is the critical feature of any program. Think about the budget of any program. Embedded within that budget are the concrete raw materials for making change happen. The largest share of the budget resources most often is the personnel category (or consultants if the program depends on delivery by consulting collaborators). So in terms of program resources and program goals, the management of change is the central focus of programs.

The management of change is embedded in the narrative of those participants closest to the interaction between the provider and beneficiary, and no less so with the program leaders. Each program participant carries within them a "narrative of action" that when verbalized gives the evaluator the clues to the deepest nature of a program. Unleashing these narratives through questioning and listening to participants is the beginning of a process of mutual and collaborative self-reflection that is critical in making program evaluation useful and helpful.

So do not assume to know anything about a program, but always ask each program participant you encounter to describe the work they do. Note the words and narrative revealed by the participant: In what order are activities and services described? What conditions are noted? What kinds of interactions are described? What links to the community are identified? Who is doing the action - provider or beneficiary? How is hopefulness and hopelessness conveyed? What challenges are identified? What facilitators are identified? The narrative of the participants will become an evaluator's most useful tool to make program evaluation useful and helpful.

Reflections on Managing Client Requests for Evaluation Data Before Contract Ends

Here is what happened:

I heard from one colleague on the EVALTALK listserv who thoughtfully recommended that I revise the contract, if it did not contain language clarifying access to data by the client. I thought this was a good idea going forward anyway. Then I made a list of the benefits to the client to have access, the negatives of granting access, and the benefits and negatives of granting access for my professional obligation to my client. I drafted a memoranda in which I proposed some language that would clarify my expectations of the use of the data by my client. Here is what I said:

a. The external evaluation contract provides that analyses and reports based on findings from data collected during the program year require that the evaluation company must conduct the analyses and reporting on company computers.

b. The external evaluator could not verify that the integrity of the data and any analyses conducted by the client, and, therefore, could not use any data set prepared by the client in the external evaluation.

c. The consent form signed by providers and parents prohibits the external evaluator from providing individually identifiable information to anyone outside of the evaluators and evaluation company.

d. The client understands and agrees to the condition that the external evaluator may not use data files derived from an analyses of the (information provided) for the external evaluation, unless joint analytical plan is authorized.

e. The client will use the data provided only for internal purposes, and not submit reports derived from the analyses outside of the internal evaluation staff.

e. By agreeing to these conditions, the client will make no changes to the budget based on providing the information requested.

At a subsequent meeting with the client, after reviewing my draft proposal, the client agreed that sharing anonymous data would not help them, and that the conditions outlined by my draft proposal are appropriate. They withdrew the request, and we proceeded to move forward on the evaluation plan, amicably and on a sound relationship.

During the meeting, I also learned that the request was made to see if there was some way to assist me in making the tight deadlines that were required for the grant, and not for any other reason that may have been imagined. Because of my efforts to remain as transparent as possible - usually expressed to them through details about the status of the data collection, and my transformation of my evaluation infrastructure to comply with the evaluation requirements - they were becoming anxious about my capacity to meet the contract obligations. While they valued my transparency and honesty, I had made the client managers more anxious over time.

The lesson for me is to consider the client's capacity for reception of information (transparency) about the process of an external evaluation - while transparency models the way the evaluator wants the client to communicate, the side effect may by an erosion of confidence in the capacity to deliver on the contract.

This experience is also a confirmation of the principle that challenges are really opportunities to improve and learn to be better.

Internal & External Evaluation Collaboration Questions

Dear Colleagues,

I was recently asked by a major client to provide them with the pre-test data set from an external evaluation of their program, just as I was concluding the post-test data collection. I was hired to provide an external evaluation (with contract). I am a small evaluation firm, a sole proprietor, and I did expect to provide them with all of the data after the conclusion of the study (after the evaluation reports are submitted and approved and final payment is received). I was hired to conduct the evaluation based on a federally funded grant that requires an evaluation, but not necessarily an external evaluation. The external evaluation was part of their accepted application to the federal agency. One other note, the director who hired me has recently resigned and the program coordinator with whom I have been working with now for over a year, was hired to replace the director.

The back story is that they planned to increase their research capacity, which I supported, and consequently, hired a very competent research analyst, who now has the time and competence to conduct their own internal evaluation. Without getting into the politics and personalities, which, I acknowledge plays an important part in this little drama, I am curious about the approach and opinions of other evaluators who may visit this blog.

We live in an urban environment in the Northeast where the poor outcomes of children in public education is a very serious matter, and the stakes are high for the program and the agency which runs the program. The agency has a very positive reputation in the community. With one year left on the evaluation, and with the limited structure of my research firm, I am tempted to "collaborate" (assuming I have a choice in this matter), and hand over the pre-test data set to the in-house evaluator for preliminary analyses. (There are many sub-evaluation questions we planned for in which in-house data and external data would be combined anyway.) But I also am obligated to protect the agency and program from potential criticism of the results of the study, in light of the challenge to objectivity that may emerge from an arrangement in which the in-house evaluator has prior access to pre-test data.

Assuming for a moment that the contract allowed or was passive on data access and integrity, what advice would you offer, if you were inclined to collaborate with your client. What advice would you offer if you were not so inclined? What safeguards or protections would you impose in a written agreement with your client so that some mutually beneficial compromise would emerge?

I am sending this post to the EVALTALK listserv, and later I will post some of the comments I receive.

Thank you in advance for your feedback.

David