Was It Worth It? How to Create Metrics for Events

Events are terrific.  If you’ve worked for a nonprofit organization, you have probably been involved in planning, staffing or cleaning up after an event.  It could be a neighborhood clean-up or back to school fair.  An event can be a promotion for a new program or a way to identify potential clients. 

When I managed Community Involvement at the Social Development Commission, we put on a slew of events, gave away thousands of hotdogs and neighborhood swag, like tote bags and refrigerator magnets with community service phone numbers on them. One communitywide planning event had an auditorium full of neighborhood residents doing a conga line through the aisles to the beat of an Indian drum. This made us all feel terrific.

But what did it really mean?  Most of the time, event organizers/sponsors use three metrics to decide if an event was worth the investment: 1) number of participants; 2) number of problems with the event; and 3) how happy we feel.  An event that fills the room, doesn’t have a catastrophe like running out of food, and leaves us humming while we clean up is an absolute success.

Is it possible to do a better, more substantive evaluation of an event?  Absolutely!

Here are some ideas to consider:

1.  Survey participants.  Yes, I know.  No one wants to interrupt the Kumbaya moment with a clipboard and a checklist.  A quick, post-card survey with 3 to 5 questions can provide actual data that can be used to determine what participants thought was valuable, what other information or resources they might like, and what potential impact the event will have on their lives.  The West Allis Health Department conducts an annual event called Two for the Show which is a developmental screening with various ‘stations’ to assess toddlers’ speech, large and small muscle development, and other developmental milestones.  This is one of the Health Department’s primary ways of identifying children in need of Birth to Three services so they are able to track identification of children with developmental challenges as they show up in the Birth to Three program.  Over and above that, however, the Health Department conducts a survey of each Two for the Show parent. Very smart strategy – makes funders happy and helps shape the next event.

2.  Service utilization. This is a variation of ‘tell them Fred sent you.’  Since many events, like the events we used to hold at SDC, are geared toward encouraging program participation in programs like Head Start or energy assistance, it is very helpful to connect participants’ attendance at the event with the eventual enrollment in a program.  This can be as simple as handing someone a card and asking them to mention the event when they call the program and offering some benefit for doing so.  This could be expedited enrollment or a small premium like a McDonald’s gift certificate.  Anything that helps you as the event organizer connect your event to later program participation is a big plus as you seek support for next year’s effort. 

3.  Tracking What Happens Next.  Events are often the vehicle for addressing a community need or problem.  Generally, the event creates several work groups with the hopes that when people leave they are willing to work on specific tasks in order to achieve an agree-upon goal.  Very often, big community organizing events are unable to translate to dynamic, robust work groups and so the energy and promise of the event just dissipates.  Vision Sherman Park, a tremendously inspirational community planning event that brought together observant Jews, African American, and White residents for a day of planning and dialogue, had a less vibrant transition to work groups.  That experience toward me that assessment of the follow-up is critical.  What happened afterward?  Who stayed involved?  Who didn’t?  What can be improved next time?

Ultimately, it’s all about a critical eye that looks beyond the momentary happiness of a ‘successful’ event to examine its true value and impact.  It’s not difficult, but it takes take planning and commitment.  When your next event rolls around, try to take a harder look at the issue of metrics.  I think it will pay off for you.

Seeing is Believing

When evaluating a program or service, nothing beats a site visit.  Yes, it’s important to review the numbers, look at the logic model, quantify outcomes, and gather customer/client satisfaction data.  These fundamental sources of information are essential to painting the evaluation picture.  But the heart and soul of an evaluation comes from face to face meetings, observations, and ‘walking around’ a program.

I will be doing three site visits in September – three very different agencies in very different parts of Wisconsin, requiring a lot of travel and a lot of time.  So why not just interview people over the phone or do a ‘Go To Meeting’ virtual meeting?

Here’s the answer:  I can’t tell if there’s a ‘there’ there unless I go see.  Seriously, the ability of executive directors to describe their programs in glowing terms is legendary.  If so inclined, an enthusiastic executive director can turn tens of participants into hundreds, good outcomes into astonishing accomplishments, well, you get the idea.  If I’m evaluating a program, I need to make sure the program is operating as described, the participants are really present and engaged and the outcomes are legitimate.

In my experience, these are the things that make for a great site visit:

1.  Genuine welcome:   This begins at the front door.  Do people know I’m coming?  Are they gracious and friendly?  Are the people I need to see available?  Does it appear that the evaluation site visit is a priority?

2.  Openness:  Do people appear to be sharing information freely?  Or are they guarded in what they share?  Does everyone in a group discussion speak or just the executive director?  Are people nervous about sharing or eager to tell their story?

3. Confidence and pride: Are people proud of their organization and happy to tell their story?  Are they willing to share war stories, to describe barriers or problems encountered and how they were overcome?

4. Inclusiveness: Does the executive director leap up to go find “Mary” who is the expert in a particular area or call in a client waiting at the front desk to relate his experience with the program?  In other words, does the executive director or program staff want to include others in explaining the program? 

5. Real Deal Feel:  When I leave, do I feel like I saw the real deal or a show staged for my benefit?  There’s no way to quantify this, but an experienced evaluator can sense an artificiality in the site visit that lets her/him know that the real program wasn’t shared (and may not actually exist).

These are the things I’ll be looking for in September as I travel around Wisconsin.  What about you?  Done evaluation site visits?  Been site visited?  What have been your experiences?  What can we learn from you?

Ask the Consultant: Evaluating a Program You Don’t Like

What do you do as an evaluator when you really don’t like or support the program approach you are evaluating; say, it’s something contrary to your principles or beliefs?

This was a question asked by an Alverno University student of me and several evaluation colleagues who were speaking to her class last week. One colleague recounted a major evaluation focused on a teen pregnancy prevention approach he couldn’t endorse.  I recalled instances where, in the course of an evaluation, I encounted agency practices with clients that made me uncomfortable, even angry.  We all agreed that this problem comes up a lot for evaluators since, being human beings, we have often have very strong personal beliefs.

When this happens, though, there is an enormous risk of one’s personal beliefs influencing the objectivity of the evaluation.  This can happen in such subtle ways that even the evaluator isn’t aware that his/her biases are shading everything – the construction/selection of evaluation instruments, the content of interviews, and the interpretation of observed activity. While it is far better and a lot more fun for an evaluator to evaluate an approach he/she fundamentally endorses, the opposite is often true.  When in this situation, a couple of possible strategies might be useful.

First, one of the evaluators on the panel reminded us all that every program deserves a decent evaluation, sort of on the order of everyone accused of a crime is entitled to legal counsel.  Good thing to keep in mind.  Every program approach benefits from a thorough, well-conceived and implemented process and outcome evaluation.

Second, when an evaluator is put in a position of having to fairly evaluate a program approach he/she doesn’t like, the bottom line is sticking with the process.  This means evaluating a program based on its program design/logic model.  Period.  This means not letting alternative or more philosophically attractive approaches enter into the analysis as implicit or explicit points of comparison.  This is tough, but essential.

Third, the evaluator simply must keep her/his biases in check and be extra vigilant about avoiding any opportunities to go looking for evidence to support those biases.  Because an evaluator often has a lot of control over how success is defined and measured, this can be extremely challenging.  Basically, to do right by the evaluation, the evaluator has to put on and keep wearing the mantle of objectivity even when it chafes.

 These are some ideas about handling this thorny situation.  In future blog posts, I’ll be tackling other questions that have been posed to me about planning, grantwriting, collaboration, and professional ethics.  If you have a question, let me know.  Be glad to take a crack at answering!