Reflections on Managing Client Requests for Evaluation Data Before Contract Ends

Here is what happened:

I heard from one colleague on the EVALTALK listserv who thoughtfully recommended that I revise the contract, if it did not contain language clarifying access to data by the client. I thought this was a good idea going forward anyway. Then I made a list of the benefits to the client to have access, the negatives of granting access, and the benefits and negatives of granting access for my professional obligation to my client. I drafted a memoranda in which I proposed some language that would clarify my expectations of the use of the data by my client. Here is what I said:

a. The external evaluation contract provides that analyses and reports based on findings from data collected during the program year require that the evaluation company must conduct the analyses and reporting on company computers.

b. The external evaluator could not verify that the integrity of the data and any analyses conducted by the client, and, therefore, could not use any data set prepared by the client in the external evaluation.

c. The consent form signed by providers and parents prohibits the external evaluator from providing individually identifiable information to anyone outside of the evaluators and evaluation company.

d. The client understands and agrees to the condition that the external evaluator may not use data files derived from an analyses of the (information provided) for the external evaluation, unless joint analytical plan is authorized.

e. The client will use the data provided only for internal purposes, and not submit reports derived from the analyses outside of the internal evaluation staff.

e. By agreeing to these conditions, the client will make no changes to the budget based on providing the information requested.

At a subsequent meeting with the client, after reviewing my draft proposal, the client agreed that sharing anonymous data would not help them, and that the conditions outlined by my draft proposal are appropriate. They withdrew the request, and we proceeded to move forward on the evaluation plan, amicably and on a sound relationship.

During the meeting, I also learned that the request was made to see if there was some way to assist me in making the tight deadlines that were required for the grant, and not for any other reason that may have been imagined. Because of my efforts to remain as transparent as possible - usually expressed to them through details about the status of the data collection, and my transformation of my evaluation infrastructure to comply with the evaluation requirements - they were becoming anxious about my capacity to meet the contract obligations. While they valued my transparency and honesty, I had made the client managers more anxious over time.

The lesson for me is to consider the client's capacity for reception of information (transparency) about the process of an external evaluation - while transparency models the way the evaluator wants the client to communicate, the side effect may by an erosion of confidence in the capacity to deliver on the contract.

This experience is also a confirmation of the principle that challenges are really opportunities to improve and learn to be better.

Internal & External Evaluation Collaboration Questions

Dear Colleagues,

I was recently asked by a major client to provide them with the pre-test data set from an external evaluation of their program, just as I was concluding the post-test data collection. I was hired to provide an external evaluation (with contract). I am a small evaluation firm, a sole proprietor, and I did expect to provide them with all of the data after the conclusion of the study (after the evaluation reports are submitted and approved and final payment is received). I was hired to conduct the evaluation based on a federally funded grant that requires an evaluation, but not necessarily an external evaluation. The external evaluation was part of their accepted application to the federal agency. One other note, the director who hired me has recently resigned and the program coordinator with whom I have been working with now for over a year, was hired to replace the director.

The back story is that they planned to increase their research capacity, which I supported, and consequently, hired a very competent research analyst, who now has the time and competence to conduct their own internal evaluation. Without getting into the politics and personalities, which, I acknowledge plays an important part in this little drama, I am curious about the approach and opinions of other evaluators who may visit this blog.

We live in an urban environment in the Northeast where the poor outcomes of children in public education is a very serious matter, and the stakes are high for the program and the agency which runs the program. The agency has a very positive reputation in the community. With one year left on the evaluation, and with the limited structure of my research firm, I am tempted to "collaborate" (assuming I have a choice in this matter), and hand over the pre-test data set to the in-house evaluator for preliminary analyses. (There are many sub-evaluation questions we planned for in which in-house data and external data would be combined anyway.) But I also am obligated to protect the agency and program from potential criticism of the results of the study, in light of the challenge to objectivity that may emerge from an arrangement in which the in-house evaluator has prior access to pre-test data.

Assuming for a moment that the contract allowed or was passive on data access and integrity, what advice would you offer, if you were inclined to collaborate with your client. What advice would you offer if you were not so inclined? What safeguards or protections would you impose in a written agreement with your client so that some mutually beneficial compromise would emerge?

I am sending this post to the EVALTALK listserv, and later I will post some of the comments I receive.

Thank you in advance for your feedback.

David

Differentiating Program Evaluation Logic Terms

Thanks to EPA Administrator and blogger, Marcus Peacock, and the EVALTALK contributors, here is an interesting and useful way to distinguish between the terms used in many common forms of logic models (The Flow of the River, http://flowoftheriver.epa.gov/my_weblog/2007/10/getting-juiced-.html)

Nectarine Juice Production Program

Performance Measurement

• The inputs are nectarines and a hammer
• The activity is smashing nectarines with a hammer
• The output is how many nectarines got smashed
• The outcome is how much juice resulted
• The impact is the improved health of the people who drink nectarine juice

Program Evaluation

• How well is the program producing nectarine juice using the current inputs?
• Is a hammer really the most effective intervention for producing nectarine juice? What are the alternatives?
• What is the nectarine juice production program’s specific contribution to the improvement of national public health?

I'm not a devoted fan of juice, but the example helps me understand the way people may use the terms. The program "Juice Production Initiative" further illustrates that programs have both resources (objects) and activities (processes) to organize and make use of for a desired end.