For many people the idea of pulling out evaluation ideas seems more painful than having a root canal done. In preparation for the ‘Building a Mental Health & Addictions Charter for Niagara’ working session held on September 18, 2013, I joined a working group that agreed to take on this often tedious task. Our goal was to identify how we would evaluate the process of bringing people together to create the Charter. We chose a process that allowed us to ease the work into a one-hour activity by dividing our ideas into three categories:
How much did we do?
- # of
- #
How well did we do?
- % of
- %
Is anyone better off?
- % increase in
- %
Using these categories we were able to quickly capture a wealth of ideas on paper rather than talking in circles about the intent of our evaluation. The process also pushed us to go beyond the easy number counting data to identify measures of how well the work of building the Charter is done (or the quality of the changes the development of the Charter helps us to realize) and ultimately if the process of creating the Charter leads to anyone being better off (i.e. people who are involved with building the Charter).
At a second meeting, we added to and refined the list of indicators. Then we chose the ones that we believed would be most meaningful to those working on building the Charter. For example, if someone was asked to justify their time spent developing the Charter, what data would help them to make this case? What data would help us to know if we were staying on course with the overall goal of the Charter?
Next we presented our ideas to the 55 people from 37 agencies attending the September 18th working session to ensure we didn’t miss anything, and that participants saw the value in the evaluation measures we had identified.
Here is what we came up with:
- % of initial forum participants that signed a Call to Action card to build the Charter
- % of forum participants who join a working group (and the % of agencies present that this represents)
- % of agencies or networks that can describe their role in building the Charter (a measure of engagement)
- % of agencies or networks that have informed their Board/planning committees about the work of building the Charter
- % of participating agencies or networks that are connected to 1 or more agencies (to be measured by network mapping)
- % of people who help build the charter who report positive connections (increased quality of communications) and/or a change in experience in communicating with representatives from other agencies.
The next step is to work with academic researchers to turn these ideas into evaluation questions, collect baseline data and determine when we will collect follow-up data. At a later date we will work through the process again, to identify indicators to measure the implementation of the Charter and the ultimate impact of the Charter on residents in Niagara.
The process described above is based on the work of Mark Friedman’s Results-based Accountability. For more information about using this process visit: www.resultsaccountability.com.
Lisa Gallant
Healthy Living Niagara Coordinator
lisa.gallant@niagararegion.ca
