Innovations in Evaluation

A Report on Evaluation in the Field of Sexual Violence Prevention

[Cover image shows a road leading into the distance.]

Suggested citation: Townsend, S. M. (2017). Innovations in evaluation: 
A report on evaluation in the field of sexual violence prevention. Enola, PA: 
National Sexual Violence Resource Center.

© National Sexual Violence Resource Center 2017. All rights reserved. 
This document was supported by Cooperative Agreement #5UF2CE002359-04 
from the Centers for Disease Control andPrevention (CDC). Its contents are solely 
the responsibility of the authors and do not necessarily represent the official views 
of the CDC.The content of this publication may be reprinted with the following 
acknowledgement: This material was reprinted, with permission, from the National 
Sexual Violence Resource Center’s publication entitled Innovations in Evaluation: 
A Report on Evaluation in the Field of Sexual Violence Prevention.
This report is available by visiting www.nsvrc.org

Innovations in Evaluation

A Report on Evaluation in the Field of Sexual Violence Prevention

By Stephanie M. Townsend, PhD

Executive Summary

In recent years, there has been an increasing emphasis on evaluating efforts to prevent
sexual violence. Meeting evaluation requirements has been a challenge for the field
due to the complexity of evaluating primary prevention programs and the lack of evaluation
capacity within organizations. This report highlights six state- and local-level approaches
to evaluation and explores organizational and individual factors of evaluation capacity.
Building capacity for evaluation and evaluative thinking requires both organizational
and individual change. Organizational factors include having:

• a culture of learning, leadership that supports evaluation
• resources and processes that support evaluation
• active communication about evaluation

Individual factors include:

• being aware of the benefits of evaluation
• being motivated to engage in evaluation
• evaluation-specific knowledge
• evaluation-specific skills

To do evaluation work that produces useful findings, organizations need to strengthen
their capacity. There are multiple approaches to doing this. Strategies for building evaluation
capacity should be implemented in a way that facilitates clear and ongoing communication,
multi-year initiatives, and active practice of skills. When selecting individuals to lead
these efforts, attention should be paid to evaluation expertise, understanding of
sexual violence prevention, and an ability to translate evaluation principles and findings
into plain language. Four issues were recurrent challenges for the programs highlighted in this
report: allocation of time, overemphasis on numbers and monitoring, misalignment of roles
and skills, and distrust of evaluation.

Recommendations for responding to these challenges include contributions
that can include building relationships with funders, public health agencies, technical
assistance providers, and local programs. It is recommended that funders align evaluation
requirements with the organization’s capacity, provide dedicated resources to evaluation,
fund multi-year evaluation initiatives, and bring greater clarity to the different roles of
monitoring versus evaluation. 

Public health agencies are encouraged to give careful attention to the understanding of
sexual violence when hiring staff evaluators and contractors, align outcomes with the nature and
intensity of the intervention and the expected process of change, and invest in both evaluation
and research.

Technical assistance providers should not rely solely on evaluation workshops to build
capacity, but also provide hands-on technical assistance, accompany program staff through
the evaluation process, and build communities of learning and practice around evaluation
through in-person and online communication, and create user-friendly tools for data analysis.

Local programs are advised to build on the strengths and capacities of their staff
and volunteers, dedicate time for evaluation, learn from peers, and actively seek help with
evaluation work.

Evaluation in the Field of Sexual Violence Prevention

Evaluation of Sexual Violence Prevention Programs

Interest in evaluation of sexual violence prevention programs was first sparked by
funders who had long required the reporting of performance indicators such as the number
of people served, presentations made, etc. To fulfill their obligations as stewards of public and
donated resources, funders began to ask for more evidence that the work they were funding
was having the kind of outcomes promised. Early on, this included requiring logic models to show
how the proposed activities could achieve the intended outcomes. With time, it grew into a
requirement for demonstrating outcomes and a preference for evidence-based practices.

This move toward increased evaluation requirements is found in a broad array of
social services and education fields, not only the prevention of sexual violence. However,
evaluating the prevention of sexual violence poses unique challenges. First, there are few
sexual violence prevention strategies that have been deemed “evidence-based.” A systematic
review by the Centers for Disease Control and Prevention (n.d.) has identified only three
programs that have been evaluated rigorously enough and demonstrate sufficient outcomes to
be considered effective programs: Safe Dates, Shifting Boundaries, and Real Consent. They
have also identified four promising programs: Green Dot, Second Step, Coaching Boys Into
Men, and Bringing in the Bystander. One reason these programs have been sufficiently evaluated
and researched is that most are in some way connected with university researchers who
either worked collaboratively with practitioners to develop the programs or developed the
programs as part of the research project itself. There may be dozens, if not more, highly
effective, locally developed programs that would show effective outcomes if they could access the
kinds of resources required for both a large scale implementation and rigorous evaluation.

[Image: Stack of folders.]

Many local programs prefer to use their own approaches to sexual violence prevention or to
modify and combine strategies developed by others. One common reason is that programs
want to use strategies and curricula that reflect their community’s values and use language and
examples participants can relate to. Additionally, none of the effective or promising programs
identified by the CDC use an anti-oppression or social justice framework nor are they culturally specific.
These are values held strongly by many programs in the field. The practitioners
and evaluators interviewed for this report all described how context and values play a large
role in the decision by programs to rely mostly on their own curricula and strategies. For example:

	“The programs in our state rely heavily on anti-oppression, anti-racism, and social justice
	work as an integral part of prevention…Because of the diversity of our state and its history of 
	colonization, this is very important. It’s found not only in sexual violence prevention, but also in 
	a lot of other public health work in the state that works from a social justice model.” 

	“Our state is very locally driven. Each program makes their own decisions about what fits their 
	community best based on the local needs assessments they do. This applies to both their 
	prevention programming and how they evaluate.”

	“Programs in our state want to have a lot of autonomy in their work. They know their 
	communities best and need the freedom to do what will work there. Especially because the 
	state is so big with so much diversity between communities, having all of the programs do the 
	same prevention program would never work.”

Therefore, staff often combine activities from different programs and supplement with their
own ideas to craft a prevention strategy they think will work in their community. Or, they
develop their own programs from scratch based on their understanding of the social-ecological
model of sexual violence. The widespread use of locally developed prevention programs
means the task of showing evidence falls on the program. Even when an effective or promising
program is used, many funders still require local data to show it is being effectively implemented
by the grantee.

Almost all local sexual violence prevention programs are being asked to engage in
evaluation work. However, it is the rare program that has staff who are trained in evaluation.
Although there is a preference in the field for formal, higher education credentials, program
evaluation is not an essential curricular component of undergraduate or even most
graduate programs in the social sciences. Even those staff who have had some training in
research methods and data analysis often do not have training and experience in designing
evaluations or measurement development to prepare them for this task.

In order to move evaluation forward in the field, it is necessary to build evaluation capacity.
A common thread among those interviewed for this project is that they all have sought ways to
do that. Whether at the local or state levels, they have made intentional efforts to foster learning
and to create systems and structures that make quality evaluations possible.

Purpose of This Report

State sexual assault coalitions, administrators of Rape Prevention Education funds, and local
prevention programs have responded to the call for outcomes evaluations in a variety of
ways. This report focuses on how programs at the local and state levels have worked to build
evaluation capacity and for thinking evaluatively (i.e., making data-informed decisions that lead
to action). It highlights examples from the field that represent forward thinking approaches
to this task. Specifically, the report explores organizational and individual components
of evaluation capacity building that were repeatedly highlighted by these innovators as
necessary to their evaluation efforts.

This project was not a systematic testing of an evaluation capacity building model. Rather, it
uses models and theories of evaluation capacity building to document and better understand
what is happening in the field. Coalitions, administrators, local programs, and technical
assistance providers can learn from these programs about the approaches they are taking,
how those approaches developed, and promising next steps.

Moving Forward in Evaluation

For this report, evaluation partners from three state level and three local initiatives were
interviewed about their work to evaluate sexual violence prevention programs and to build
evaluation capacity within their own and other organizations. These programs were selected
because of the notable successes they are having and the fact they have each taken a different
approach. Collectively, they illustrate that there are many ways to engage in this work and that
success rests in part on choosing an approach that fits the needs, values, and context where the
work is happening. A brief summary of each of these evaluation initiatives is provided below.

New Mexico

In 2014, New Mexico’s Department of Health created a staff evaluation position in their Office
of Injury Prevention. With this position, the state launched the development and implementation
of a statewide evaluation of sexual violence prevention programs. Although each program
was implementing their own prevention activities, they were all to be evaluated using the
same outcome measure.

The evaluator began by meeting with the programs and learning about what they were
doing. They found that while all programs were evaluating their work in some way in
order to meet funders’ requirements, many of those evaluations could benefit from stronger
measurement and design. With the evaluations that were being used at that time, it was not
possible to compare progress over time or across programs. Therefore, the state embarked
on a one-year process of developing common outcome measures.

That development process included extensive input and collaboration with the field. Together
with the evaluator, priority outcomes were identified that cut across the common goals
most of the programs shared. The evaluator then combed through the research literature to
find measures with established reliability and validity that matched those goals. It was quickly
realized that those measures each contained too many questions to be completed during
programming and, in some cases, required too high a level of literacy for the middle school and
high school audiences who were the primary population served by the programs. Therefore,
a consensus process was used in which each program identified the most relevant items
from the measures and additional modifications to language were made, resulting in a nineitem
pre-post survey to measure changes in attitudes about rape myths, gender norms, and
acceptance of interpersonal violence, and a six-item measure of bystander intentions.

Additionally, a qualitative student roundtable, teacher interview, and facilitator interview
protocols were developed. To help carry out this work, an evaluation guidance document was
provided to programs. This guidance was critical to the success of the project. It included all of
the measures, protocols, explanations for how to use them, and expectations for reporting data.

In the second year of the project, the survey and roundtable protocol were used by six
prevention programs in New Mexico with more than 3,000 students. Staff in the programs were
trained to enter their data into a spreadsheet. The data were then sent to the Department of
Health where the evaluator analyzed the data. Program-specific reports of the local findings
were prepared for each program and they met with the evaluator to discuss the findings.
Additionally, the evaluator and programs are working together to find ways to share the
findings with the schools and students that participated in the prevention programs, and the
evaluator is working on a statewide report with aggregated data.

In addition to this common measure and roundtable protocol, separate measures were
also developed for an LGBTQ-specific program, students with disabilities, and a bystander
efficacy measure for school and college staff.

[Image: Road leading into the distance.]

Ohio

The Ohio Department of Health, Ohio Alliance to End Sexual Violence, and an evaluation
contractor have been working for eight years to create a package of evaluation resources
that local programs can use to evaluate their prevention initiatives. Like in New Mexico, each
program uses their own unique set of locally driven prevention activities. In Ohio, however,
they opted not to develop a single outcome measure that would be used by all programs.
Rather, aligned with a locally-driven approach, the evaluation resources provide a “menu” from
which programs can choose.

The package of resources currently includes four major tools:

1. Needs and Resources Assessment: This tool allows programs to assess
primary prevention needs within their organizations. The tool operates at
multiple levels including individual staff (e.g., understanding of prevention theory,
intersections between sexual violence and other forms of oppression, programming
and evaluation skills, etc.), prevention activities (e.g., time spent working at
different levels of the spectrum of violence), and organizational needs (e.g., training
and technical assistance needs, preferred modes of delivery, etc.).

2. Primary Prevention Capacity Assessment: This tool is used by programs
to rate their prevention activities in areas such as: principles of primary prevention,
matching the program to the participants, implementation, evaluation, and barriers/
lessons learned.

3. Menu of Outcomes: This list includes prevention outcomes the programs can
choose from based on which ones best fit their needs and prevention activities. The
outcomes chosen form the focus of the program’s evaluation.

4. Item Bank: For each outcome, there are survey items the programs can choose
from to use in pre-post surveys. The items are divided into eight content areas: sexual
assault, coercion and consent; interpersonal violence and sexual violence attitudes;
bystander behavior; communication; gender stereotyping in media and culture; violence
and oppression; healthy relationships; and conflict management. Additionally, the
items measure outcomes at three different levels: knowledge, attitudes, and skills.

With this package of resources, programs can tailor their evaluations to the outcomes
they are targeting in their prevention activities. Additionally, to the extent that multiple
programs use the same outcomes and items, data can be aggregated to gain a broader 
statewide perspective.

New resources that are in the process of being developed include model policies for schools,
employers, and health associations; resources for activity-based evaluation to incorporate
into community events; and tools for using social media analytics to measure behavior change.
In addition to resource development, the evaluation team provides training and technical assistance 
through regional meetings and with individual programs.

A final aspect of the evaluation team’s work is developing a community of learning among programs for 
both prevention practice and evaluation. They do this in two ways. First, when providing technical
assistance, the team connects program staff with others who are doing similar work and facing
similar challenges. Second, they developed a directory of the risk and protective factors each
program is addressing in their prevention and evaluation work. This directory has been shared
with all programs in the state to make their own connections with their peers.

Texas

The Texas Association Against Sexual Assault (TAASA) recently added an Evaluation Manager
to their staff. This full-time position is designed to internally evaluate the work being done by
TAASA and to provide external support to local programs around evaluation. 

In addition to being one of the few coalitions who have funded a full-time evaluation position,
TAASA did not fund the position with Rape Prevention Education money. Although much
of the evaluation work being done within TAASA and among its member programs is of
prevention work, this staff position is not limited to prevention. Not only does this make the
Evaluation Manager available to more types of projects, it also means there is more freedom
to take diverse evaluation approaches that are tailored to what the local programs are most
interested in learning.

Another characteristic of this position is the promotion of a social justice approach to
evaluation. This commitment is expressed through a clear commitment to evaluation
benefitting the people who are most involved in and affected by the program being evaluated. 
Some of the evaluation methods used at the local level are closely tied with programs’ social 
justice and empowerment commitments, such as PhotoVoice (a participatory action research
method in which participants collect data by photographing their own communities and
collectively analyze the photos for themes that relate to the research or evaluation questions).
TAASA is also looking to promote more developmental evaluation, an approach that is
particularly suited to innovation, radical program redesigns, complexity, and crisis (Patton, 2006,
2010). Developmental evaluation documents patterns in real-time interactions, rather than
relying on data that are collected separate from the activity itself (Patton, 2006, 2010).

This program-specific approach to evaluation is currently carried out through intensive technical
assistance provided by the Evaluation Manager to local programs and TAASA staff. Extensive
time is spent building relationships with local programs in the space where they are more
comfortable and where the evaluator can get to know the staff and their programs.

[Image: Students sitting in a circle, holding a discussion.]

Day One

Day One in Providence, Rhode Island, is the rape crisis program that serves all six counties
in the state. For years the program had been doing a primary prevention program in schools
that they developed, called Your Voice, Your View. The program consists of four 50-minute
sessions grounded in social norms and bystander intervention theories. Additionally, students
participate in a social marketing campaign and teachers participate in a brief training. The major
goals of the program are to unearth healthy social norms and promote bystander interventions.

In partnership with Lindsay Orchowski, PhD, at the Alpert School of Medicine at Brown
University, Your Voice, Your View is currently being evaluated in a four-year, randomized
control trial in 30 schools. The first year of the evaluation consisted of development work,
including: making revisions to the program so it could be implemented in a large number
of schools, developing a program manual, developing competency measures to ensure
consistent implementation across facilitators and schools, training facilitators, developing
outcome measures, gaining access to the schools, and planning for sustainability. The
evaluation includes a pre-survey and a postsurvey six months later. Data collection and
analysis is currently underway.

Blackburn Center

The Blackburn Center in Greensburg, Pennsylvania, has worked in partnership with Seton Hill College
for many years. The focus of their work is social transformation. In addition to individual activities
such as workshops, staff trainings, and community events, the program has worked to build a
sustained institutional engagement with primary prevention. They have also taken on the broader
issue of gender equity including incorporating the issue into curricula and establishing institutional
policies that support equity. More recently, this work has been expanded to the University of
Pittsburgh at Greensburg.

Recognizing the long-term process of social transformation, the Blackburn Center
and its college partners have embarked on a longitudinal evaluation of their social
transformation work. They have developed a survey that is given to all students during their
first year on campus and again toward the end of their fourth year.

The campus assessment teams at each college have collaborated on the survey design and
data analysis. They work together to analyze and interpret the data at their respective
campuses and look for patterns across the two communities. Support for this work includes
course releases for two faculty members. Rather than an individual faculty member being
the collaborating partner based on their own interests, the colleges have made an institutional
commitment to working with the Blackburn Center on social transformation and evaluation
of those efforts.

[Image: Adult students in a class, filling out paperwork.]

Pittsburgh Action Against Rape

Pittsburgh Action Against Rape (PAAR) has committed to evaluation in all of its prevention
initiatives, regardless of the funding source or requirements. They have leveraged connections
with independent evaluators and university faculty to carry out those evaluations. When
working with external partners, PAAR remains in a leading role by defining the outcomes that
need to be measured and putting the needs of the community at the forefront of the evaluation
design. They emphasize finding ways to measure behavioral change and using evaluation
strategies that offer participants meaningful opportunities to give voice to their experiences.

PAAR takes a long-term view on evaluation work and is leveraging opportunities to
contribute to the evidence base for primary prevention. For example, in 2011 they developed
a series of four workshops for parents to prevent child sexual abuse. In collaboration with an
evaluator, they used a process measure to assess the level of participation in the workshops; a
pre-post survey to measure changes in attitudes, behavioral intents, and actual behaviors; and
brief qualitative interviews to document the social impact and diffusion of the intervention.

The results of that evaluation indicated the workshops were worth continuing with
modifications to enhance particular outcomes. PAAR secured funding to implement a revised
version of the workshops in six communities in Pennsylvania in collaboration with other
sexual violence prevention programs. That implementation was evaluated, using a modified
version of the pre-post survey, in collaboration with university partners. Simultaneously, two
other states used the curriculum and evaluation tools to implement the workshops in multiple
communities in their states.

PAAR is now working with the researchers and evaluator to pool the data across those various
implementations and analyze the aggregated data with the hopes of publishing the findings
in a peer-reviewed journal. This will be the first step to establishing a credible evidence base
for the workshops. This is a vital step in bridging the gap between research and practice and
building an evidence base for a much broader set of strategies than currently is available.

Multiple Approaches

These six examples illustrate that there are multiple approaches to building evaluation
capacity and evaluating primary prevention strategies. In considering these examples, two
initial commitments by the programs influenced how they proceeded and succeeded. First, would
the evaluation be internally or externally driven? Second, would the evaluation be tailored to the
specific program or uniform across programs? Using these dimensions, we can place each of
these examples in one of four quadrants (see Figure 1).

Figure 1. Example programs placed into four quadrants based on approaches to evaluation.

[Figure shows the programs organized based on these approaches: uniform, tailored, internal
and external. New Mexico is located between uniform and external. Blackburn Center is located 
between uniform and internal. PAAR and Texas are both located between internal and tailored.
Ohio and Day One are both located between tailored and external.]

These dimensions are not mutually exclusive of one another. For example, while New Mexico
developed a uniform measure for most of their prevention programs, that measure was developed 
with extensive input from programs into the content of the measure. Consequently, the uniform 
measure used in the state aligned with the goals of those programs. Additionally, there were unique 
evaluations developed for programs working with targeted populations.

Evaluation Capacity Building and Evaluative Thinking

These six approaches to evaluation demonstrate the development of evaluation capacity.
Evaluation capacity building is, at its most basic, any effort that tries to strengthen the
understanding of evaluation concepts in a way that leads program staff to apply those concepts to
their work. Underlying this is the goal of evaluation becoming routine within the organization
(Stockdill, Baizerman, & Compton, 2002).

Evaluation capacity building requires both organizational and individual change (Preskill
& Boyle, 2008; Taylor-Ritzler, Suarez-Balcazar, Garcia-Iriarte, Henry, & Balcazar, 2013).

• Organizational components include:
	○ An organizational climate that values learning
	○ Leadership that supports evaluation
	○ Resources that support evaluation
	○ Processes that support evaluation
	○ Active use of evaluation findings
	○ Active communication about evaluation
• Individual components include:
	○ Being aware of the benefits of evaluation
	○ Being motivated to engage in evaluation
	○ Evaluation-specific knowledge
	○ Evaluation-specific skills

While these components make effective and meaningful evaluation possible, a larger shift in
organizational culture is also necessary. That is, programs that engage in evaluation capacity
building should develop a mindset of evaluative thinking. Evaluative thinking (Buckley, Archibald,
Hargraves, & Trochim, 2015) includes:

• Critical thinking
• Curiosity
• A valuing of evidence
• Identifying assumptions
• Asking thoughtful questions
• Reflection

Through intentional reflection on questions and evidence, an organization that engages
in evaluative thinking will make data-informed decisions that lead to action (Buckley et al., 2015).

[Image: Arrow pointing forward.]

Summary

There are multiple ways prevention programs can move forward in evaluation. Efforts to build
evaluation capacity should begin by reflecting on what approaches best fit the current
capacity, needs, and values of the program(s). There is no one approach to evaluation that will
fit all programs. Building evaluation capacity requires addressing both organizational and
individual factors.

Organizational Factors

Building evaluation capacity is not merely a matter of teaching individuals about what
evaluation is and how to do it. Evaluation capacity occurs within organizations and is about:
	“sustainable evaluation practice – where members continuously ask questions that
	matter, collect analyze, and interpret data, and use evaluation findings for 
	decisionmaking and action. For evaluation practice to be sustained, participants must be
	provided with leadership support, incentives, resources, and opportunities to transfer their
	learning about evaluation to their everyday work.” (Preskill & Boyle, 2008, p. 444)

Individuals can be catalysts for change that promote evaluation capacity and use. The
catalyst must be an individual who is in a position to facilitate change in other members’
knowledge and skills and to mainstream evaluation practices in the organization
(Garcia-Iriarte, Suarez-Balcazar, Taylor-Ritzler, & Luna, 2011). This underscores, again, that
building evaluation capacity is about changing organizations, not merely training individuals.

Culture of Learning

Not all organizations are ready to engage in building their evaluation capacity. The organization
has to demonstrate a value of and capacity for learning. Elements of organizational capacity for
learning include (Preskill & Boyle, 2008):

• Information is freely shared among members of the organization
• Members trust one another
• Questions are consistently asked
• There is a willingness to take risks
• Leaders seek information when making decisions
• Leaders are open to feedback
• Employees are supported in and rewarded for learning and professional development

Recent documentation of organizational change processes in the Sexual Assault
Demonstration Initiative (Townsend, National Sexual Assault Coalition Resource Sharing
Project, & National Sexual Violence Resource Center, 2017) underscored the critical role that
a culture of learning can play in organizational change. Based on organizational change
efforts over a four-year period, it was clear that organizations demonstrating the most
consistent structures and practices to support learning were able to accomplish the greatest
organizational change.

The programs interviewed for this report all described elements of learning organizations
in their own institutions and grantee programs. The following characteristics were named as
important to moving forward with evaluation:

• Having a sincere desire to know what is working and what is not
• Being curious and asking good questions
• Being interested in learning
• Actively participating in workshops and regional meetings
• Asking for feedback and continually striving to improve practice
• Reflecting on the bigger picture of what the organization does and why
• Being willing to look at issues from different perspectives and to draw insights and
techniques from different disciplines

Leadership That Supports Evaluation

Leadership that supports evaluation is a critical component of successfully integrating
evaluation processes and evaluative thinking into an organization. This does not mean that
strengthening evaluation work has to be a top–down process. However, because evaluation
capacity requires organizational resources and commitments, at some point the leadership of
the organization must support the allocation of those resources – including staff time,
infrastructure, and fiscal support.

[Image: A group of women looking over paperwork.]

Designation of Leaders

The programs interviewed about their approaches to evaluation all identified leadership
and organizational commitment to evaluation as necessary. Notably, the state coalitions and
Departments of Health had all allocated funding to support either full-time staff positions or
contract positions to lead evaluation work in the state. These staff and contractors were
charged with the design and implementation of evaluations and providing support and technical
assistance to local programs. At the local level, leadership came internally from executive and
program directors who saw evaluation as an opportunity to build their programs.

Supportive Leadership
 
A notable characteristic of this leadership is that it emphasizes supporting local program staff
rather than imposing requirements on them. As one staff evaluator explained:
	“I can work in a way that isn’t imposed, but that builds on what they [local programs]
	are thinking and doing and what they are interested in learning…I tell programs, ‘Think
	of me as a free consultant.’ That allows them the freedom to make our relationship be
	about what they need and want versus about what they need to report.”

Accountability

Another striking characteristic of the leadership approach taken by these organizations is how
they think about accountability. Because of the requirement to engage in evaluation by funders
(in some states evaluation plans are used in the scoring of funding proposals and/or included in
funding contracts) it is easy for programs to see evaluation as accountability to funding agencies.
However, the programs interviewed for this report described evaluation as accountability to
the people they serve:
	“Funders need to stop thinking of evaluation as monitoring. The funder is not the one
	the evaluation is for. It’s for the people in the program to know if what they are doing
	is worth it…What gets people thinking evaluatively is the downward accountability.
	It’s about making sure they’re doing right by the people they serve.”

	“Because our programs are committed to a social justice model in their work, it was
	important that the evaluation include ways for those who are most impacted by the work,
	the students and teachers, to have input…Evaluation shouldn’t be only about funders.
	It’s about those who are most impacted by the work.”

	“Evaluation isn’t about what the funders want. We would be doing this anyway,
	even if no funder required it. We have to evaluate because how else will we know if
	what we’re doing is working? It’s part of our responsibility to the community and to the
	people we serve.”

Evaluation Leaders as Role Models

Evaluation leaders must intentionally act as role models. All of the state organizations
interviewed for this report talked about how they have to model evaluation for local programs.

This was described in greatest detail by one evaluation team: “Local programs have really
appreciated the [tools], in part because they see us using them in our work.” This has included
using the assessment data to:

• Plan the content of state and regional trainings
• Inform the content of technical assistance resources that are developed
• Inform the delivery methods of technical assistance
• Determine topics for regional meetings

The key is that the evaluation team uses data to inform their own work and talk with programs
about how evaluation informed the training, technical assistance, and/or regional meeting.
That way, programs “…see evaluation being used.” This models the value of evaluation.

Resources and Processes That Support Evaluation

In addition to the human resource of leadership, evaluation also requires fiscal resources and
organizational processes. These make a systemic approach to evaluation possible.

Funding Sources

The funding of staff positions and evaluation contracts facilitated the advances in evaluation
made by the programs interviewed for this report. It is notable that for one of the state
coalitions and two local programs, the primary source for funding evaluation work was not CDC/
RPE funds. Instead, their evaluation work was primarily being funded by other state sources
or by foundations. These programs described having more freedom in their approaches to
evaluation. For example:
	“It helps that the funding for [the evaluation] position is not from RPE or CDC
	who have very specific ideas about what they want measured and how they want it
	measured. Instead, we have freedom to take other evaluation approaches that rape crisis
	centers are more comfortable with. We can do evaluation in a way that isn’t imposed, but
	builds on what they are thinking about, doing, and interested in learning.”

In contrast, programs that rely on CDC/RPE funds described tailoring their evaluation work
to what that particular funder is looking for. For example: “We try to stay within the framework of
where the CDC is going with looking more at risk and protective factors.”

Allocation of Time

In addition to allocating funds, the dominant theme among these programs when talking about
the resources and processes that have supported their evaluation work was allocating time to
evaluation. Designing meaningful evaluations that are methodologically sound and supported by the
programs who will be implementing them takes time. The kind of time needed is best exemplified
by two projects:

In New Mexico, local programs had some evaluations already in place when the Department
of Health hired a statewide evaluator. However, the measures being used were locally developed
and did not adhere to principles of effective measurement, nor were they sufficiently validated
to allow for comparing progress across time or across programs. Therefore, the first year of the
evaluation process was spent:

• Building relationships between the evaluator and programs
• Informally assessing what was happening in the programs and what values were important
to them in their work and in evaluation
• Educating programs about the importance of validated and uniform measures
• Through a consensus process between all of the programs and the evaluator, identifying
key outcomes that were in common across all of the programs
• Reviewing the literature for existing measures that assess those outcomes
• Identifying and modifying items from validated measures that would
best measure those outcomes in the communities the programs were working in

Similarly, in Rhode Island, the partnership between Day One and a research team at Brown
University spent their first year:

• Developing the program manual for the curriculum to be evaluated
• Making revisions to the curriculum
• Developing adherence and competency measures to ensure the curriculum was
implemented with fidelity
• Developing a training protocol for curriculum facilitators
• Developing outcome measures
• Gaining access to the schools where the curriculum would be implemented
and evaluated
• Planning for sustainability

Although these were large-scale and statewide evaluations, even at the local level more time
must be allocated to developing programs and evaluations than is often supported by funders.
As staff from one local program explained, “You cannot develop, implement, and evaluate a
program in one fiscal year. But that’s what looks good to funders so it’s hard not to promise them
what they want.” Conversely, another program emphasized the power of the foundation funding
they receive for their social media work, which is a three-year, renewable grant:
	“When we were able to go to a three-year grant cycle, it allowed us so much more time
	to really think about what we were doing and do it well. Social transformation takes time.
	Jumping to measuring outcomes every year is premature and it can make it look like you
	are not achieving your goals when, in fact, there is deep change happening.”

Evaluation Tools and Reporting

A trend was seen in the state-level evaluation work of the programs where practical tools
were developed to facilitate data collection and analysis. For example:

[Image: Charts and graphs.]

In New Mexico, after the common pre-post attitudes measure was developed, a MS Excel
spreadsheet was created for programs to enter their data into. They then send their spreadsheets
to the Department of Health where the evaluator analyzes the data. An evaluation report is created
for each program, specific to their findings. The evaluator meets with program staff in person to
talk through the findings and discuss how they can be used. The evaluator is also working with
programs to find meaningful and relevant ways to share the local results back with the schools and
students in the communities.

In Ohio, two common measures were developed and implemented in each program:
(1) the Needs and Resources Assessment that assesses the needs in the community and (2)
the Primary Prevention Capacity Assessment that assesses the capacity of the program to do
primary prevention work. Those assessments are used by the local programs to guide their
prevention planning and used at the state level to inform training and technical assistance.
Additionally, a menu of outcomes and a bank of items for use in pre-post surveys has been
developed. Local programs use these tools to develop their local outcomes surveys.

In Texas, a common outcomes tool for primary prevention was developed. That initial tool is
now being revised based on input from the field about how well it is working in their communities
and how well it matches the nature of their prevention programming. In addition to that
common tool, the coalition’s Evaluation Manager works individually with programs to develop
more innovative approaches to evaluation that match their organizational goals, philosophies,
and resources.

At the local level, we also see tool development as a key evaluation activity that occurs at
the organizational level. This often involves collaborative work with consultants or
community partners. For example:

[Image: Group of college students talking.]

Pittsburgh Action Against Rape originally contracted with an evaluation consultant to
develop both process and outcomes measures for their parent workshops. Those tools were
included in the curriculum package and later used in modified versions in the larger scale
implementation of the program in six sites in Pennsylvania and in multi-site implementations
done in two other states.

Blackburn Center’s campus transformation initiatives are being evaluated by campus
assessment teams at each of the two colleges where they are working. In addition to their
campus-specific efforts, the two assessment teams are also working together on
measurement development and sharing findings. This assessment is a longitudinal evaluation of
campus-level change. In addition to evaluating individual activities such as workshops and
campus events, the assessments teams have developed a pre-post survey of attitudes and
behaviors that is administered to all students in their first year and again in their fourth year of
enrollment. This will allow for measuring change over time.

Day One’s program is being evaluated through a combination of process measures to verify
adherence to program fidelity and competent facilitation, and outcome measures to assess
changes in knowledge, attitudes, and behaviors. The measures were designed based on research
principles of measurement and also with the perspective of practitioners regarding relevance
and suitability to the audience. 

It is notable that all of these organizations had existing survey tools that did not fit the
programs or populations. At both the state and local levels, the staff of these programs are
keenly aware of the importance of matching the goals and activities of the prevention initiatives
to the outcomes measured. In some cases, relevant measures do not exist in the research
literature. In other cases, they are not a suitable fit for the population due to literacy levels,
cultural relevance, or practical length. This speaks to the need in the field for tools based on
sound measurement principles that are relevant to the goals and populations of sexual violence
prevention programs.

While most of these programs rely on surveybased measures, there are other types of tools
being used. For example, New Mexico’s evaluation includes a qualitative assessment through
student roundtables conducted at the end of the programs. In Texas, some programs have
explored the use of PhotoVoice as an evaluation method. This underscores the need for qualitative
guidance and tools. Qualitative methods are particularly suited to going beyond the counting
model of performance monitoring and relying on the active listening skills of advocates
and community educators. More importantly, they may better reflect the social justice and
community building values of some programs.

Collaborative Efforts

Finally, it is important to stress that these programs have all been collaborative efforts.
Collaboration has taken a variety of forms in these programs, including:

• Partnerships among local programs, coalitions, Departments of Health,
evaluators, colleges, and/or researchers
• Statewide primary prevention and/or evaluation teams that meet regularly to
identify needs, plan evaluation strategies, and develop evaluation resources
• Repeated opportunities through regional and other meetings and electronic
communication for local programs to provide input on evaluation design and measures
• Shared interpretation of evaluation findings 

Building collaborative relationships is a process. As one prevention director explained:
	“At first I was very protective of the program. I was unsure how working with a
	researcher would impact it. At the first site visit it was overwhelming to be the sole
	practitioner in a room full of researchers. But being treated as a true partner alleviated
	those concerns.”

One key to building a collaborative relationship is having genuine respect for one another. This
was described by all of the sites interviewed for this report. The mutual respect for one another’s
expertise was poignantly illustrated by Day One’s prevention director who described the different
roles the partners play with implementing the prevention program:

Because the randomized control trial involves 30 schools, there was a need for more program
facilitators than Day One had the capacity to provide. Therefore, part of the research team
was trained to implement the program in the schools in collaboration with Day One. As
co-facilitators, the research partners and Day One staff run the sessions with students. “We
have so much more with our partnership than we [prevention program] or the researchers would
have alone. It was tricky to train our research partners to implement the program, but they
are open to direction. It’s not just about teaching them the content, but how to effectively
facilitate a primary prevention program.” This partnership relies on the research team bringing
their expertise on research and evaluation and the prevention staff bringing their wealth
of experience on facilitation and community responses to sexual violence. The prevention
staff also ground the project in values of advocacy, healing, and action.

Active Use of and Communication About Evaluation

Although some of the approaches to evaluation reflected in this report are still early in their
implementation, all programs reported ways they are using and communicating about evaluation.
These include:

• Discussion and training about evaluation at regional meetings
• Preparing and sharing reports on local evaluation findings
• Preparing and sharing reports on statewide evaluation findings
• Exploring ways to share findings with community partners and participants
• Ongoing discussion of evaluation needs and planning in primary prevention and/or
evaluation teams and workgroups

A particular way of communicating about evaluation that may serve as a model for other
states is the fostering of social networks happening in Ohio. The evaluation team has
developed a directory that identifies which risk and protective factors are being measured by
each program in the state. That directory has been shared with the programs so they can
network with other programs measuring similar outcomes around what is working, trouble
shooting, lessons learned, etc. In addition to that public resource, the evaluation team also
frequently facilitates individual connections between programs. Instead of the evaluation
team being in a position of providing all of the answers and guidance, they connect programs
to one another for assistance. These strategies are being used “to create a community of
learning” in the state. This approach can also promote shared leadership for evaluation
and sustainability of evaluation capacity building efforts. In the event that there are
major reductions in staffing at the state level, programs will be well positioned to continue
providing assistance to one another through their own peer network.

Communication about evaluation can also occur informally within an organization. For
example, one coalition’s prevention team was already experienced and actively engaged with
evaluation when an evaluation position was created. Through informal communication within
the coalition, other teams have become more interested in enhancing their own evaluation
work as well. Rather than imposing specific expectations on the teams, they were allowed to
move at a pace that was comfortable for them and met their needs.

Summary

Building evaluation capacity is a process of organizational change. As such, before
addressing the training needs of individuals, it is important to determine that the organization
itself has sufficient commitment to evaluation and is ready to engage in change. The
organization does not need to have elaborate systems and structures in place before it begins
to engage in evaluation. However, it must be open to creating systems and structures that
support evaluation as the needs emerge.

Individual Factors

Individuals within organizations must also be prepared with the knowledge and skills
to engage in evaluation work and evaluative thinking. In the field of sexual violence
prevention, building evaluation capacity at the individual level typically requires intentional
and sustained efforts. Although there has been a trend toward hiring college-educated staff,
especially from the fields of social science and social work, it is the rare staff member who
will have been trained and had experience in program evaluation. They may have a general
orientation toward reading and understanding data, but designing, implementing and analyzing
evaluation projects requires specific skills.

This is not to say that program evaluation is outside the scope of practice for coalitions
or local programs. Given sufficient training and supports, programs are able to engage in
meaningful evaluations. However, the training and support that is needed should not be
underestimated if programs are expected to design and carry out evaluations from scratch.

Attitudes Toward Evaluation

The first individual factor that comes into play is the attitudes staff and others involved in the
evaluation have about evaluation itself. Attitudes that support moving forward with evaluation
include (Preskill & Boyle, 2008):

• Assuming that evaluation is a good thing to do
• Thinking that evaluation can contribute to effective decision making
• Believing that evaluation and evaluative thinking add value to the organization’s work
• Believing that the staff are capable of learning how to design and carry out evaluations

[Image: Several lightbulbs hanging from the ceiling with one of them lit up.]

Because the programs interviewed for this report have demonstrated forward-thinking
practice in their evaluation work, it is not surprising that they hold positive attitudes toward
evaluation. However, those working at the state level were keenly aware of the attitudinal barriers
that exist in the field. The description from one coalition staff member was echoed by others:
	“Programs have been burned by evaluation in the past. They have been asked to do
	things without the skills, resources, money, or time it takes to do them…You have to help
	them see how evaluation can actually be useful to them and not to see it as just one
	more thing a funder is asking them to do.”

This is not to say that there is necessarily strong resistance to evaluation. Another
evaluator working at the state level was told when hired that she would need to build
relationships with the local program partners and to expect resistance from them. However,
she found little resistance. Program staff were concerned that a statewide evaluation would
require them to abandon their locally-developed programs. However, through their relationships
with the evaluator they were reassured that she understood their social justice commitments
and that the evaluation could give them more leverage for supporting that work. As a result,
local support was generated for designing and implementing the statewide evaluation.

Similarly, in another state, the local programs have responded very favorably. They appreciate
the tools provided and value seeing data used at their regional meetings and in the training
and technical assistance they receive. Although programs acknowledge the time consuming nature
of doing evaluation, they see how the outcomes can be used to guide their work in meaningful ways
and to gain more funding and support.

Knowledge and Skills for Evaluation

Evaluators have identified specific knowledge and skills that are needed to design evaluations
from scratch and implement them through data collection, analysis, interpretation, and use. For a
list of evaluation knowledge and skills, see Table 1.

Table 1:
Evaluation Knowledge:
What evaluation involves
Terms and concepts
How evaluation can contribute to decision making
Strengths and weaknesses of different approaches and data collection methods
How politics can affect evaluation
Importance of culturally appropriate methods
What makes for ethical evaluation practice
Importance of differing opinions and perspectives
Relationship between goals, objectives, activities, and outcomes
What to look for when hiring an evaluator

Evaluation Skills:
How to develop a logic model
How to develop evaluation questions and plan
How to design data collection instruments
How to choose appropriate data collection methods
How to do basic data analysis — quantitative and qualitative
How to interpret results and draw conclusions
How to develop an evaluation budget
How to communicate evaluation processes and findings
How to manage the evaluation process
(Preskill & Boyle, 2008; Taylor-Ritzler et al., 2013)

In contrast to this comprehensive list, the programs interviewed for this report were more
selective in choosing which of these knowledge and skills local programs most need to carry out
meaningful and effective evaluation. While all of these knowledge and skill areas are present in
the big picture of their evaluation efforts, some are contributed by the external evaluator or
evaluation team and some are contributed by the local programs, leading to a shared evaluation
capacity. See Figure 2 for a representation of the shared evaluation capacity between evaluators and
local programs.

Figure 2. Shared evaluation capacity between evaluators and local programs

[Venn diagram showing the attitudes, knowledge, skills, resources, insights, experiences, and relationships
of both evaluator and local programs. Where the diagram overlaps is labeled "shared evaluation capacity."]

Thinking about evaluation capacity as shared, rather than as residing in only the evaluator or the
local programs, can focus capacity building efforts and make the work more manageable. Rather than
needing to develop the ability for local programs to design and implement evaluations from scratch
and with complete independence, efforts can focus on developing specific knowledge and skills. This
takes fewer resources and places fewer demands on the local programs.

Building Individual Attitudes, Knowledge, and Skills

A wide variety of strategies have been identified and used to build evaluation capacity at the
individual level. They include (Preskill & Boyle, 2008; Garcia-Iriarte et al., 2011):

• Written materials
• Online resources
• Training
• Technical assistance
• Meetings
• Communities of practice
• Coaching or mentoring
• Collaborative involvement in evaluation projects
• Internships

While the approach is often to involve as many staff as possible in these strategies, there is
also evidence that working closely with one or a small number of staff and supporting them as
catalysts for change in their organizations can also be effective (Garcia-Iriarte et al., 2011).

Regardless of the strategy that is used, a number of factors can influence how much
capacity is built. Some factors to consider during implementation include (Preskill & Boyle, 2008):

• Communication: Clear, consistent, frequent, and widespread communication about
what is happening in regard to evaluation and evaluation capacity building can affect
individuals’ level of interest in, engagement with, and commitment to evaluation.

• Timing: When working with programs that provide crisis response who are often
under resourced, there is no perfect time to increase evaluative work. However,
care should be given to timing by avoiding initiating evaluation work during months
when other major initiatives are happening (such as Sexual Assault Awareness Month),
when it is known that major organizational changes will be happening (such as a
transition in Executive Directors), and when the program is in an active state of
organizational crisis (such as a major and unanticipated loss of funding).

[Image: Clock]

• Facilitator’s expertise and effectiveness: Any outside facilitator of evaluation work
must be seen as credible. This is particularly a challenge when evaluating sexual violence
prevention because an evaluator may have excellent technical expertise in evaluation,
but lack understanding of the unique aspects of sexual violence prevention.

• Frequency and quality of participation: Developing evaluation knowledge and
skills takes time and opportunities to apply and practice what is learned. This calls
not only for some frequency in evaluation capacity building opportunities but also
for those opportunities to include explicit practice. Whether it is through hypothetical
cases or applying knowledge and skills to work the program is presently doing, skill
development requires active practice with opportunities to explore the complexities
that come up and gain feedback.

These factors are all found in the work done by the programs interviewed for this report.
Whether a statewide approach to evaluation was taken or local programs were supported
in designing and carrying out their own evaluations, communication and frequency and
quality of participation were both identified as key facilitators of success. For example:

• In New Mexico, Ohio, and Texas, regional meetings and trainings are regularly held.
These gatherings have specific topics and goals as well as provide opportunities for
networking among programs.
• In New Mexico, the program-specific results are shared through multiple modes of
communication, including both a written report and a face-to-face meeting with the
evaluator to discuss the findings and how to use them.
• The university-program collaborations at Day One and the Blackburn Center both
involve weekly or monthly meetings of the evaluation teams for collaborative planning,
troubleshooting, and discussion of findings.

The time needed for effective evaluation was also emphasized by the programs. All of
these efforts were designed from the outset to be multi-year efforts. That commitment and
investment from the beginning was identified as critical to success. This is particularly a challenge
when evaluation and evaluation building efforts are funded on an annual cycle. Programs
expressed a need for more multi-year funding cycles that allow them to engage in long-term
efforts with more intentionality and consistency.

Finally, the expertise and credibility of evaluators supporting and leading these efforts
was described as important. No one interviewed for this project expressed concerns about
technical expertise in terms of evaluation design and methodology. Evaluators, whether staff or
contractors, were generally seen as bringing valuable skills and expertise for evaluation.
However, attention to credibility in regard to sexual violence is needed. This was expressed at
both the state and local levels:

In describing her own position at the state level, one evaluator emphasized how important
it was that she developed trusting relationships with the local programs. When programs felt
like she “got it” in regard to their values and approach, they were relieved and engaged
collaboratively with the evaluation process. As someone who shared a similar perspective
but who had not previously worked in sexual violence prevention, she spent a lot of time at
the beginning learning about sexual violence, asking questions about what local programs
were doing, and listening to them.

A program that has worked with numerous contract evaluators has had a wide range of
experiences. While some of them clearly knew what they were doing in terms of data analysis
and produced what the program staff trusted were sound results, at times they found the
reports of limited use because they were heavy with statistical details. In contrast, they have also
worked with contract evaluators who understand the field and are able to translate evaluation
results into plain language that can be used for program improvement. Those evaluations
have been much more useful, both for program improvement and for building the staff’s own
understanding of evaluation.

There was a common theme among these programs that not all evaluators are equal
when it comes to evaluating sexual violence prevention. While it is easy to suggest that a
program partner with faculty at a nearby college or university, doing that does not necessarily
lead to successful evaluations. The evaluator needs to have both technical expertise as well
as an understanding of (or at least willingness to learn about) the unique issues related to sexual
violence prevention.

Summary

To do evaluation work that produces useful findings and to be active partners in evaluation,
local program staff need to develop knowledge and skills. There are multiple approaches
to achieving this goal. The strategies used should be implemented in a way that facilitates
clear and ongoing communication, multi-year initiatives, and active practice of skills. When
selecting individuals to lead these efforts, attention should be paid to evaluation expertise,
understanding of sexual violence prevention, and ability to translate evaluation principles and
findings into plain language.

Responding to Challenges

As the programs highlighted in this report moved forward in their evaluation work, they
encountered numerous challenges that had to be addressed. Looking across the programs, four
issues were recurrent themes for both state-level and local programs.

[Image: Charts and notebooks lying on a table.]

Allocation of Time

The time it takes to design a meaningful evaluation is often underestimated by funders
and by programs. A key factor that allowed these programs to move forward in their
evaluation work was that they were able, either through funding or other strategic
planning, to allocate time to evaluation design and measure development. Staff from the
highlighted programs emphasized the need to allocate sufficient time — ideally one year — to
evaluation design and measure development, especially when working across multiple
programs or sites.

The allocation of time is especially important for a field that is frequently drawn into crisis
mode. Due to the emotional effect of doing crisis intervention, the frequency of community crises
such as high profile cases, and organizational crises such as funding cuts, it is difficult for
programs in this field to create time and space for the type of reflection, learning, and planning
required to build evaluation capacity. Scheduling dedicated time for this work requires that
programs not be forced into compressed timelines. 

Overemphasis on Numbers/Monitoring

Although evaluation is talked about as a tool primarily for program improvement and gaining
useful insight, it is often tied to monitoring and accountability. This is to the detriment of quality
evaluations. Programs need the freedom to design evaluations that provide accurate and
meaningful data. That requires clarity about whom the evaluation is for — the consensus
across these interviews was that evaluation is for the people most affected by the program,
including both the program staff and the community.

While monitoring and accountability are important, those goals are best met through
other mechanisms. For example, some funders use evaluation to try and ensure
fiscal accountability. However, efficient fiscal management and detection of inappropriate use
of funds are better assessed through financial and operational audits.

Besides being ineffective, what funders ask programs to count sends messages about
what is deemed important. Local programs still struggle with the mixed messages the
reporting requirements send. On the one hand, they are told that primary prevention requires
multiple dosages and community saturation, that it may be best achieved by intensive work
with a smaller number of people, and that community mobilization (which may have no
or few “participants”) is important. On the other hand, funders continue to ask for regular
reports on the number of participants in prevention programs and the number of distinct
presentations or workshops held while providing virtually no mechanism for telling about the
intensive hours and informal networking that may go into community mobilization
efforts. This misalignment between what is promoted as “good” prevention work and what
programs must report pushes both prevention programming and the outcomes that are
evaluated into constrained parameters.

Misalignment of Roles and Skills

While all staff can build their individual skills for evaluation and make meaningful contributions,
part of the challenge at the local level is the misalignment between the preventionist/
educator and evaluator roles. When hiring prevention staff, the emphasis is on hiring
people who have excellent group facilitation, presentation, and networking skills. Because
most evaluation requirements are still tied to prevention programs (with little being required
in direct services), evaluation tasks are typically assigned to the preventionist/educator. However,
the skill set required for evaluation is different. As one person explained:
	“It’s like you’ve hired an actor to be a director. The facilitator needs to respond to
	what’s happening in the moment and to think session to session about what that particular
	group needs. But the evaluator needs to think about the big picture and long-term outcomes.
	They need to have program management skills, the ability to think about data, an
	educational background that is relevant to program evaluation, and significant experience
	in the field outside of presentations.”

The misalignment of skills is exacerbated by the fact that, in many programs, preventionist/
educator roles are entry-level positions. Therefore, often times it is the people in the
organization who have the least experience in sexual violence intervention and prevention
who are being asked to do some of the most complex thinking about how to measure what
the program is achieving. Because prevention funding requires evaluation but provides no
additional or designated funds for it, most programs do not allocate the resources for a
designated evaluation position.

Distrust of Evaluation

The challenges described above add up to many staff, both at the local and state levels,
distrusting and even fearing evaluation. This may include concerns that:
• Evaluation requirements and/or results will require abandoning current
prevention strategies
• Evaluation results will lead to a loss of funding
• Sharing evaluation challenges with funders will jeopardize funding
• Time spent on evaluation will take away from the prevention work itself and that if
a reduction in programs offered or people participating in them decreases as a result,
there will be a loss of funding
• Resources allocated to evaluation will take away from direct services such as
counseling and advocacy
• There is no way to measure community level work

These concerns, combined with work overloads, can lead to feelings of frustration and
burnout. As one person described it, programs are left “feeling like evaluation is crammed down
their throats.” This thwarts evaluation capacity building and can foster resistance to evaluation
requirements and evaluative thinking.

Recommendations for Practice

Funders

Funders have advanced the practice of evaluation in prevention programs. It was
largely under their leadership that evaluation was prioritized as a necessary and useful tool.
It has been approximately 10 years since the push toward evaluation began. It is a good
time to take stock of the impacts, intended and unintended. This report was not designed
as a comprehensive assessment of the impact of funders’ requirements on programs and
coalitions. Therefore, it is all the more striking how many connections between evaluation
capacity and funding were made by the staff who were interviewed.

The basic message they sent was that more dedicated resources are needed if evaluation is
to move forward in meaningful and useful ways. Strategies that may be worth considering include:

• Align evaluation requirements with the capacity of the organization. Organizations that lack a 
culture of  learning or do not have leadership that supports evaluation are not ready to engage 
in the same depth of evaluation as organizations that are strong in those areas. Requiring
different levels of evaluation and allocating funding for evaluation accordingly may
allow for more targeted evaluation capacity building and yield more robust evaluations.

• Provide dedicated resources to evaluation. This can counterbalance
the tension between funding evaluation versus funding programs and services.
If those resources are sufficient, it will shift evaluation from being an unfunded
mandate to a valued activity.

• Fund multi-year evaluation initiatives. Multi-year funding can provide a structure
that supports the relationship building, input, and consensus processes necessary
for designing evaluations and developing measures that are relevant to the
programs being evaluated and to their community contexts. It also allows for
measuring longer-term outcomes that are more indicative of the types of changes
in behavior, social norms, and culture reflective of primary prevention.

• Bring greater clarity to the different roles of monitoring versus evaluation.
Periodically reviewing the type of monitoring data being collected and how it
is used can minimize unnecessary reporting requirements. Additionally, ensure that
monitoring and evaluation reports are kept as separate as possible. Make the process
of funding determinations as transparent as possible to minimize the unintended effects
reporting can have on driving programming. It may also alleviate concerns programs
have about losing funding.

Public Health Agencies

[Image: Building blocks on a table.]

Because most prevention funds come to communities through public health agencies
and primary prevention often uses public health models, it is vital that public health
administrators support sexual assault-specific approaches to evaluation. Public health models
and professionals have much to offer to the field of sexual violence prevention. However, some
caution should be exerted:

• Careful attention should be paid when hiring evaluators. While an applicant or
contractor may have excellent technical skills, if they lack an understanding of the
social context of sexual violence, it may be difficult for them to design evaluations
that are relevant to primary prevention. Evaluating primary prevention of other
health issues does not necessarily transfer to the complexities of sexual violence. This
can make it difficult to develop effective indicators of behavioral change and can
result in choosing indicators that will provide an inaccurate view of program effectiveness.

• Outcomes should be aligned with the nature and intensity of the intervention
and the expected process of change. While this is a fundamental concept in
evaluation, the expectation that programs demonstrate effectiveness in achieving
primary prevention goals can lead them to measure long-term goals prematurely

• Invest in research. Often times, public health agencies evaluate sexual violence
programs under the rubric of program improvement. This avoids the necessity of
obtaining approval from an Institutional Review Board (IRB). However, it also puts
severe limitations on the ability to share the evaluation results at conferences
and to publish it in research journals. The field needs more strategies that have
practice-based evidence to support their efficacy. Otherwise, the requirement to use
evidence-based strategies will continue to constrain programs to only a few options.
Ensuring that the evaluation work you are doing can be shared publicly is a simple
step toward broadening that evidence base.

Technical Assistance Providers

State and national technical assistance providers can play a vital role in building evaluation
capacity in the field. This may be best achieved by reframing what technical assistance looks like:

• Workshops are not sufficient by themselves. The expected outcomes of
evaluation workshops need to match their intensity. Short, one-time workshops,
such as occur at conferences, may be an effective way to introduce ideas and to
build interest in a particular evaluation approach. However, they generally are
not effective for building skills. Longer or multi-session workshops allow for practice
and development of skills and receiving immediate feedback. However, they do
not necessarily equip participants for the unexpected quirks and challenges that will
come up when they are evaluating their own programs.

• Hands-on technical assistance that accompanies program staff through the
evaluation process is needed. In this way, technical assistance is less about
disseminating information and more about coaching or mentoring program
staff through the evaluation process. It may require assisting the program with
successive evaluation projects, with the program taking on more independence
each time.

• Building peer networks and communities of learning/practice can expand the resources
in the field. The frequency of program-toprogram networking varies across states and
should not be presumed. Actively facilitating connections between individuals and creating
settings that promote networking and peer learning can enhance the support and
capacity building process.

• Creating tools for data analysis can be an efficient way of building capacity for
local programs. One of the areas where local programs face the greatest challenges
is analyzing quantitative data. From learning how to use spreadsheets and how to run
simple statistical analyses, data analysis can seem insurmountable. Providing simple,
user-friendly tools can minimize the number and complexity of skills individuals need
to develop. For example, surveys can be accompanied by pre-programmed MS Excel
spreadsheets that only require local programs to enter the data and then automatically
generating means, graphs, and statistical tests of change on pre-post surveys.

Local Programs

Local programs can successfully design and implement program evaluations. However, it
should be remembered that while it is easy to write a survey, it’s not necessarily easy to
write a good one. The same can be said of any evaluation method. A few basic strategies can
facilitate the process and, in the end, provide meaningful information that can be used.

• Play to your strengths. Consider the skills, interests, and experiences of your staff,
board, and volunteers. Do not presume that the prevention staff are the best
suited for evaluation work. You may have other human resources. Similarly, not all
evaluation tasks need to be completed by the same person. For example, a person
who is good with details may be the best suited for entering data, while a person
who is good at explaining numbers in plain language may be the best suited for
summarizing the results of the evaluation. Involving more than one or two staff
members in the process of evaluation may lead to more buy-in and sustainability of
prevention efforts.

• Dedicate time. Organizations that do crisis intervention tend to operate in a crisis
mode, even when there is no crisis at hand. Dedicating time for reflection, strategic
planning, and other organizational tasks is good practice. Include evaluation in that
cycle. Whether it is one day per month or an afternoon every week, schedule the time and
make it the priority. Otherwise, evaluation will always be put on the back burner.

[Image: Group of coworkers writing on a whiteboard.]

• Learn from others. Talk with other programs in your area. Go to evaluation
workshops at conferences. Ask your coalition to connect you with people who
are doing similar prevention work and talk with them about how they are evaluating it.
Connect with people who are using similar evaluation methods and help one another
problem solve.

• If you get stuck, ask for help. Whether it is from a colleague, another program,
your coalition, or another technical assistance provider, reach out to
someone. Don’t stay stuck.

• Plan for sustainability from the beginning. Especially when an evaluation project
brings in temporary resources such as data experts or additional facilitation support,
it is important to plan ahead for how that work will be continued when the evaluation/
research is completed. The evaluation project should not set up expectations for
long-term activities if the resources to carry out those activities will not be available in
the long-term. That may mean scaling back on some aspects of the evaluation.

Funders, public health agencies, coalitions, technical assistance providers, and local programs
each have unique roles to play in building the evidence base for the primary prevention of
sexual violence. By building evaluation capacity within the field, multiple goals can be achieved.
Rather than being merely a task required by funders, evaluation can become an integral part
of how programs operate and think and it can benefit the community.

Methodological Notes

[Image: A row of checkboxes, with one checked off.]

Evaluation Design and Methodology Interviews can provide a rich understanding of
participants’ experiences and beliefs. Because they are conducted on a one-on-one basis, it
is possible to go in more depth and to explore experiences and issues that an individual
might be reluctant to share in a group setting. Because of their in-depth and interactive nature,
interviews are an effective way of checking the validity of conclusions that the evaluator may
draw from other sources of data (Singleton & Straits, 2005).

For this report, interviews were used to explore how state-level and local programs
were moving forward with the evaluation of primary prevention. Taking a case study
approach, organizations that were known to have made particular programs in evaluation
were interviewed to better understand what has supported their evaluation work and how
they solved any problems or challenges they encountered.

Procedures

The interview procedures and protocol were developed collaboratively by the National
Sexual Violence Resource Center (NSVRC) and the evaluator. Programs were selected using a
uniform nomination form and rating criteria. Nominations of programs were solicited in
three ways. First, an announcement was posted on the NSVRC’s prevention email list soliciting
suggestions from the field (including selfnominations). Second, NSVRC staff identified
programs based on their knowledge of and work with programs throughout the country. Third,
programs that presented on evaluation topics at the 2016 National Sexual Assault Conference
were considered.

This process resulted in eight programs identified as having made notable advances
in the area of evaluation. All programs were contacted by email and six (75%) responded
and agreed to be interviewed and/or to have materials from their evaluation work included.
Interviews were conducted in September and October 2016 via telephone. Interviews lasted
between 58 and 65 minutes.

Interviews

The interview protocol was semi-structured (Bernard, 1995). This method is ideal in situations 
where the evaluator anticipates having only one opportunity to interview an individual. The
protocol includes specific areas to be covered and questions to be asked, but the evaluator is
able to probe for more detail, to pursue lines of inquiry that spontaneously emerge, and to allow
for a conversational tone. This method introduces a structure to the interview while still allowing
people to express themselves in their own terms. It also allows for unanticipated experiences to be
raised and explored.

The protocol covered four main areas:
• What the organization is doing to evaluation primary prevention programs, including:
	○ Evaluation design, methodologies, and measures
	○ Roles different people play
	○ Skills required
	○ Necessary resources
• How the evaluation developed, including:
	○ The catalyst for the current approach
	○ How people responded to the idea
	○ Changes made along the way
• What it has been like to implement the evaluation, including:
	○ Challenges and how they were handled
	○ Types of evidence that are most valuable
	○ Lessons learned
• Influence the evaluation work is having, including:
	○ Successes with evaluation capacity building
	○ How evaluation findings have been used
	○ Other consequences of the evaluation

Data Analysis

Analytic induction was used to analyze the approaches to and processes of evaluation work
among these programs. This analytic technique emphasizes the development and testing
of explanatory assertions (Erickson, 1986). The evaluator developed a preliminary set of
assertions based on the data. Those assertions were then tested against the data, looking for
five types of evidentiary adequacy:

1. Is there adequate evidence to support this assertion?
2. Is there enough variety in the kinds of evidence that support this assertion?
3. Are there any doubts about the accuracy of the data?
4. Was any evidence collected that could disconfirm or negate the assertion?
5. Do any cases exist that are contrary to the assertion?

Assertions were then revised or eliminated based on their evidentiary adequacy until
there was a well warranted set of assertions. Those final assertions and evidence were then
presented as the findings.

Selected Resources

For programs wanting to learn more about program evaluation and the evaluation of
primary prevention programs, the following resources may be useful:

• Primary Prevention and Evaluation Resource Kit. This four-volume toolkit is
available from the Pennsylvania Coalition Against Rape. Unlike other evaluation
handbooks, this kit is specific to sexual violence prevention and so may be easier to
apply than generic guides on evaluation. 
	○ Volume 1 Choosing Prevention Strategies introduces primary prevention 
	and factors	to consider when choosing prevention strategies. 
	http://www.pcar.org/resource/primary-prevention-and-evaluationresource-
	kitchoosing-prevention-strategies

	○ Volume 2 Evaluating Prevention Strategies explains basic concepts in
	program evaluation, summarizes the current findings from research on
	primary prevention of sexual violence, and walks through six basic steps for
	evaluating your programs. http://www.pcar.org/resource/primary-preventionand-
	evaluation-resource-kit-evaluatingprevention-strategies

	○ Volume 3 Analyzing Evaluation Data provides a step-by-step guide of how to
	analyze quantitative data (such as from surveys) using MS Excel. 
	http://www.pcar.org/resource/primary-preventionand-evaluation-resource-kit-
	analyzingevaluation-data

	○ Volume 4 Analyzing Qualitative Data offers guidance on how to analyze
	qualitative data (such as data from interviews and focus groups). 
	http://www.pcar.org/resource/primary-preventionand-evaluation-resource-kit-
	analyzingqualitative-data

• Listening to Our Communities: Assessment Toolkit. This short toolkit is
available from both the National Sexual Violence Resource Center and the Resource
Sharing Project. A product of the National Sexual Assault Demonstration Initiative,
this toolkit was designed to help programs assess the climate in their communities
for sexual assault services and prevention. It provides an overview of community
assessment, measurement options, guides for doing focus groups and interviews,
and a guide on analyzing qualitative data. http://www.nsvrc.org/publications/
nsvrc-publications-toolkits/listening-ourcommunities-assessment-toolkit

• AEA365. This is a free blog and/or email subscription published by the American
Evaluation Association. Written by and for evaluators, the daily posts provide short,
practical tips on evaluation. It is great for getting new ideas and links to resources. You
can find the blog at: http://aea365.org/blog/

• AEA Coffee Break Webinars. These are weekly 20-minute webinars held
on Thursdays at 2:00 p.m. Eastern. Each webinar demonstrates a specific
evaluation technique. The schedule of upcoming webinars is regularly updated
and can be found at: http://comm.eval.org/ coffee_break_webinars/coffeebreak. 
Some resources at AEA365 are free to the public; others require a membership.

References

Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and
	teaching evaluative thinking: Insights from research on critical thinking. 
	American Journal of Evaluation, 36, 375 – 388. doi:10.1177/1098214015581706

Centers for Disease Control and Prevention (2016). Sexual violence: Prevention strategies.
	Retrieved from https://www.cdc.gov/violenceprevention/sexualviolence/prevention.html

Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.),
	Handbook of research on teaching (3rd ed. pp. 119-162). New York, NY: Macmillan Publishing.

Garcia-Iriarte, E., Suarez-Balcazar, Y., Taylor-Ritzler, T., & Luna, M. (2011). A catalyst-forchange
	approach to evaluation capacity building. American Journal of Evaluation, 32,
	168 – 182. doi:10.1177/1098214010387114

Patton, M. Q. (2006). Evaluation for the way we work. The Nonprofit Quarterly, 13(1), 28-33.

Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance
	innovation and use. New York, NY: Guilford Press

Preskill, H., & Boyle, S. (2008). A multidisciplinary model of evaluation capacity
	building. American Journal of Evaluation, 29, 443 – 459. doi:10.1177/1098214008324182

Singleton, R. A., & Straits, B. C. (2005). Approaches to social research (4th ed.). New
	York, NY: Oxford University Press.

Stockdill, S. H., Baizerman, M., & Compton, D. W. (2002). Toward a definition of the
	ECB process: A conversation with the ECB literature. New Directions for Evaluation
	2002(93), 7–26. doi:10.1002/ev.39

Taylor-Ritzler, T., Suarez-Balcazar, Y., Garciar-Iriarte, E., Henry, D. B., & Balcazar, F. E. (2013).
	Understanding and measuring evaluation capacity: A model and instrument validation
	study. American Journal of Evaluation, 34, 190–206. doi:10.1177/1098214012471421

Townsend, S. M., National Sexual Assault Coalition Resource Sharing Project, & National Sexual
	Violence Resource Center. (2017). Sexual Assault Demonstration Initiative: Final report. Retrieved
	from the National Sexual Violence Resource Center: 
	http://www.nsvrc.org/sites/default/files/publications_nsvrc_reports_sadi-final-report.pdf