Seeing is Believing

When evaluating a program or service, nothing beats a site visit.  Yes, it’s important to review the numbers, look at the logic model, quantify outcomes, and gather customer/client satisfaction data.  These fundamental sources of information are essential to painting the evaluation picture.  But the heart and soul of an evaluation comes from face to face meetings, observations, and ‘walking around’ a program.

I will be doing three site visits in September – three very different agencies in very different parts of Wisconsin, requiring a lot of travel and a lot of time.  So why not just interview people over the phone or do a ‘Go To Meeting’ virtual meeting?

Here’s the answer:  I can’t tell if there’s a ‘there’ there unless I go see.  Seriously, the ability of executive directors to describe their programs in glowing terms is legendary.  If so inclined, an enthusiastic executive director can turn tens of participants into hundreds, good outcomes into astonishing accomplishments, well, you get the idea.  If I’m evaluating a program, I need to make sure the program is operating as described, the participants are really present and engaged and the outcomes are legitimate.

In my experience, these are the things that make for a great site visit:

1.  Genuine welcome:   This begins at the front door.  Do people know I’m coming?  Are they gracious and friendly?  Are the people I need to see available?  Does it appear that the evaluation site visit is a priority?

2.  Openness:  Do people appear to be sharing information freely?  Or are they guarded in what they share?  Does everyone in a group discussion speak or just the executive director?  Are people nervous about sharing or eager to tell their story?

3. Confidence and pride: Are people proud of their organization and happy to tell their story?  Are they willing to share war stories, to describe barriers or problems encountered and how they were overcome?

4. Inclusiveness: Does the executive director leap up to go find “Mary” who is the expert in a particular area or call in a client waiting at the front desk to relate his experience with the program?  In other words, does the executive director or program staff want to include others in explaining the program? 

5. Real Deal Feel:  When I leave, do I feel like I saw the real deal or a show staged for my benefit?  There’s no way to quantify this, but an experienced evaluator can sense an artificiality in the site visit that lets her/him know that the real program wasn’t shared (and may not actually exist).

These are the things I’ll be looking for in September as I travel around Wisconsin.  What about you?  Done evaluation site visits?  Been site visited?  What have been your experiences?  What can we learn from you?