By December Lange Treacy, MS
Assistant Project Director and Program Evaluator at STEPs at the University of Nebraska at Omaha
Support and Training for the Evaluation of Programs (STEPs) evaluates the Rape Prevention & Education (RPE) program in Nebraska, which is funded by the Centers for Disease Control and Prevention. The contents of this post are solely the responsibility of the author and do not necessarily represent the official views of the Centers for Disease Control and Prevention or the Department of Health and Human Services.
Recently, a prevention educator and I were talking about evaluating community-level work. She told me, “I just feel like I am bobbing out at sea. I can see the shore, but I don’t feel like I am moving any closer.” When I first started learning about evaluating coalition building and community mobilization efforts, I felt like the preventionist bobbing out at sea. I had a vague idea of where I wanted to end up, but I was overwhelmed in a sea of information. To manage this information overload, the STEPs team collaborating with Nebraska RPE (including myself, Claire Rynearson, and Dr. Jeanette Harder), waded in deep and started to organize information in what ended up becoming the Coalition Building and Community Mobilization Measurement Tools for Sexual Violence Prevention with Nebraska RPE. This “measurement tool menu,” as we have come to call it, can serve several purposes.
First, the “measurement tool menu” can help those working to prevent sexual violence start thinking about coalition building and community mobilization as long-term, developmental processes. That is, there are several stages through which coalitions progress as they build themselves and mobilize their communities. We use the terms Formation, Implementation, and Impact in our menu, based on Goodman et al.’s (1996) developmental phases. I have found this to be especially helpful for setting feasible expectations for the types of outcomes, evaluation questions, and measurement options based on the coalition’s stage of development.
Since coalition and community mobilization are still relatively new approaches in sexual violence prevention, many programs are likely in the first stage (Formation) of coalition building. In this stage, the focus is on building the coalition’s membership, developing formal structures with established rules and procedures, and identifying both community needs and potential solutions.
We have worked with prevention teams who are doing the hard work of the Formation stage to ensure they have a strong foundation on which to build their coalition. However, they often tell us that it is hard to know if this is “real work” or if it “counts.” The measurement tool menu offers a resounding yes! This work is not only legitimate, but it is vital to the coalition building process.
Once we understand where we are at in the coalition building process, we can start to ask, “What can we start to evaluate at this stage?” While we may be eager to skip ahead and start looking at the community-level indicators to evaluate the impact of our coalition, we want to ensure what we propose to measure are realistic and feasible given the developmental stage of our coalition.
For example, because a key activity in the Formation stage is recruiting coalition members, one output to consider measuring may be the number of meetings held with community members. One of the RPE-funded programs in Nebraska currently keeps a meeting log where they track all of the meetings they hold with potential coalition partners. This is one way to track an output of the coalition-building process in alignment with the Formation stage.
In addition to outputs, you can also start identifying outcomes associated with the work in each stage. Based on the activities typically conducted in the Formation stage, you may look at outcomes around member participation and engagement, the formality of the organizational structure and procedures, organizational climate, or community partnerships and collaboration.
The “measurement tool menu” is organized by both stage of development and potential outcomes. For example, if your coalition is in the Formation stage and you have an outcome related to formalizing the structure of your coalition, you might consider the Coalition Self-Assessment Tool, the Community Action Program Institutionalization Scale, Levels of Collaboration Survey, or the Plan Quality Index.
For each measurement tool, there are recommendations for when to use the tool, how to score the tool, the citation for the tool, and if they have been evaluated for reliability.
As an external evaluator, while I am often in the boat alongside those working to prevent sexual violence, I am not the one setting the course or choosing where to drop the anchor. Instead, I think of my role as providing resources like an oar or a compass so that the prevention team can successfully navigate rough waters to get where they want to go together. I hope you check out one of these resources, the Coalition Building and Community Mobilization Measurement Tools for Sexual Violence Prevention with Nebraska RPE, and it helps you and your team move a little bit closer to shore.
For more tools and information on evaluating partnerships, visit the NSVRC Evaluation Toolkit.