Evaluation Approaches and Orientations

Evaluation involves more than setting outcomes and determining data collection methods to measure the achievement of those outcomes. Just as prevention programming can be driven by one or more theories and approaches – and the programming will vary depending on the theories and approaches chosen – evaluation also varies depending on the approach or framework chosen to guide it.

The following section describes different paradigms and approaches to evaluation. Most of these approaches vary somewhat in their purpose or overarching theoretical basis, but in practice, they draw from the same or similar methods for collecting data and answering evaluative questions.

The approach chosen by preventionists will depend upon

the purpose of the evaluation,
the skills of the evaluators,
the agency’s philosophical orientations, and
the resources available for implementing the evaluation.

The following approaches are relevant to sexual violence prevention work. Although the approaches have different names, some of them overlap or can be used in conjunction with each other. Additionally, the primary duties or level of involvement of the evaluator varies depending on the approach. Having a general understanding of these different approaches to evaluation will help you make informed decisions about the best approach for your own evaluation needs at various times.

Click on the links below to learn more about each approach:

Empowerment Evaluation
Utilization Focused Evaluation
Transformative Mixed Methods Evaluation
Actionable Evaluation
Activity-Based Evaluation
Participatory Evaluation
Developmental Evaluation
Principles Focused Evaluation

Empowerment Evaluation

Empowerment evaluation focuses primarily on building the capacity of individuals and organization to conduct their own evaluations. The evaluator’s role includes carrying out particular evaluation tasks and facilitating capacity building within an organization and among stakeholders. Among the principles that drive empowerment evaluation are community ownership, inclusion, democratic participation, and social justice.

For more information on Empowerment Evaluation:

The Principles of Empowerment Evaluation (Recorded Webinar) This webinar features David Fetterman giving an overview of Empowerment Evaluation.

Evaluation for Improvement: A Seven-Step Empowerment Evaluation Approach For Violence Prevention Organizations (PDF, 104 pages) This guide from the CDC includes extensive, step-by-step guidance on hiring an empowerment evaluator and is specifically geared toward organizations engaged in violence prevention work.

Empowerment Evaluation: Knowledge and Tools for Self-Assessment, Evaluation Capacity Building, and Accountability (Book$) This book by David M. Fetterman, Shakeh J. Kaftarian, and Abraham Wandersman provides an in-depth look at empowerment evaluation and includes models, tools, and case studies to help evaluators with implementation.

The Ohio Primary Prevention of Intimate Partner Violence & Sexual Violence Empowerment Evaluation Toolkit. (Online Toolkit) The Ohio Domestic Violence Network, in partnership with the Ohio Department of Health (ODH) Sexual Assault and Domestic Violence Prevention Program, is pleased to release a Primary Prevention of Sexual and Intimate Partner Violence Empowerment Evaluation Toolkit. This toolkit was developed by Sandra Ortega, Ph.D., and Amy Bush Stevens, MSW, MPH, and supported through funding by the Centers for Disease Control and Prevention's Domestic Violence Prevention Enhancement and Leadership Through Alliances (DELTA) Program and Rape Prevention Education Program. The intended audience for this toolkit is local primary prevention providers, particularly those who are beginners or who have intermediate level skills in program evaluation. This toolkit could also be used by evaluation consultants as a source of training and technical assistance materials. It includes tools on needs and resource assessments, logic models, evaluation methods, data collection and analysis, and presenting and using evaluation findings.

Utilization-Focused Evaluation

Utilization-Focused Evaluation puts evaluation use at the center of all aspects of planning and implementing an evaluation. Early stages of the evaluation process involve identifying both the intended users and the intended uses of the evaluation. Evaluators working from this approach are responsible for coordination and facilitation of various processes. As a framework, Utilization-Focused Evaluation can be implemented in a variety of different ways.

For more information on Utilization-Focused Evaluation:

Utilization-Focused Evaluation: A Primer for Evaluators (PDF, 132 pages) In this document, Ricardo Ramirez and Dal Brodhead provide evaluators with guidance on how to implement a 12-step evaluation guided by a utilization-focused framework. Examples are included for each step and case studies are provided in the appendix to give more substantial information about Utilization-Focused Evaluation in practice.

Utilization Focused Evaluation (Online Article) This article on the BetterEvaluation website provides a brief overview of UFE, including both a 5-step and a 17-step framework, an example, and advice for conducting such an evaluation.

Essentials of Utilization-Focused Evaluation (Book$) This book by Michael Quinn Patton gives extensive guidance on implementing UFE.

Transformative Mixed Methods Evaluation

Transformative Mixed Methods is a framework for both research and evaluation that focuses on centering the voices of marginalized communities in the design, implementation, and use of evaluation (Mertens, 2007). This framework requires evaluators to be self-reflective about their own positions and identities in relation to program participants and to prioritize the evaluation’s impact in the direction of increased social justice.

For more information on Transformative Mixed Methods:

Transformative Paradigm: Mixed Methods and Social Justice (PDF, 16 pages) This article by Donna Mertens outlines the basic assumptions, approaches, and social justice implications of Transformative Mixed Methods.

Transformative Research and Evaluation (Book$) This book by Donna Mertens provides an extensive overview of the theory and practice of Transformative Mixed Methods, including examples and guidance on analyzing and reporting data from transformative evaluations.

Actionable Evaluation

Actionable evaluation is an approach that focuses on generating clear, evaluative questions and actionable answers to those questions (Davidson, 2005). This approach recommends the use of rubrics to track and measure criteria related to outcomes rather than focusing on distinct indicators of outcome achievement. This approach is especially useful for tracking similar interventions or different interventions with similar outcomes that are implemented in various contexts and sites. An evaluator using this approach would facilitate the process of rubric design, collect and analyze the data, and assist the organization with using the data. They might also build the organization’s capacity to continue using the rubrics on their own.

For more information on Actionable Evaluation:

Actionable Evaluation Basics (Minibook, various formats$) This accessible and useful evaluation minibook by E. Jane Davidson gives an overview of actionable evaluation and guidance on how to implement it.

Activity-Based Evaluation

Activity-Based Evaluation (or activity-based assessment) focuses on integrating data collection into existing curriculum-based efforts to assess learning integration at distinct points throughout the intervention (Curtis & Kukké, 2014). Through this approach, facilitators can get real-time feedback to make improvements in the intervention. An evaluator using this approach might design the data collection materials and assist with data collection, analysis, and use, or they might facilitate building the organization’s capacity to use this methodology.

For more information on Activity-Based Evaluation:

Activity-Based Assessment: Integrating Evaluation into Prevention Curricula: (PDF, 32 pages) This toolkit from the Texas Associating Against Sexual Assault and the Texas Council on Family Violence provides an introduction to integrating evaluation into an educational curriculum, gives examples of data collection tools to use, and makes suggestions for analysis and use of the data.

Human Spectrogram Online Course (15 minutes) This free online interactive learning tool (NSVRC, 2019) walks participants through how to implement an activity-based evaluation through the use of the human spectrogram. Information covered includes an overview of the human spectrogram, a preparation checklist, and additional resources from the NSVRC Evaluation Toolkit. Creating a free account with NSVRC's online campus is required.

Participatory Evaluation

Participatory approaches to evaluation focus on engaging program recipients and community members in evaluation planning, implementation, and use (Guijt, 2014). These approaches can increase community buy-in for evaluation and make the evaluation more credible to the community through mobilizing various community stakeholders and program participants in the evaluation process. An outside evaluator would support participatory evaluation through facilitating the various processes involved in planning, implementation, and use of the evaluation.

For more information on Participatory Evaluation:

Participatory Approaches: (PDF, 23 pages) This Methodological Brief from UNICEF provides a very accessible and detailed introduction to participatory program evaluation

Participatory Evaluation: (Online Article) This page on BetterEvaluation gives a brief overview of participatory evaluation.

Putting Youth Participatory Evaluation into Action: (Video Presentation) This video presentation by Katie Richards-Schuster explains the process and benefits of engaging youth in evaluation work.

Developmental Evaluation

Developmental Evaluation focuses on initiatives implemented in complex environments and tracks the development of these initiatives that must be dynamic and responsive to their environments. (Gamble, 2008). Under the Developmental Evaluation framework, the evaluator is a more active part of the program team, providing real-time feedback to impact the intervention and engaging stakeholders in making meaning of the data collected.

For more information on Developmental Evaluation:

A Developmental Evaluation Primer (PDF, 38 pages) This short guide provides an overview to developmental evaluation and guidance on conducting evaluations through the developmental approach.

Developmental Evaluation for Equity-Focused and Gender-Responsive Evaluation (Recorded Webinar) This webinar features evaluator Michael Quinn Patton giving an overview of developmental evaluation.

Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use (Book$) This book by Michael Quinn Patton provides a thorough introduction to developmental evaluation theory and practice. It is written in an accessible manner and includes real work examples.

Principles-Focused Evaluation

Principles-focused evaluation, a type of Developmental Evaluation, measures a program against a set of evidence-based principles that should drive programming and lead to improved outcomes. This means that programs following similar principles can be evaluated on similar criteria even when their programming or implementation looks different.

light bulb

Watch this 44 minute NSVRC Evaluation Toolkit webinar which introduces preventionists to principles-focused evaluation (PFE) and how it has been used to support sexual violence prevention programs. Viewers will learn how to incorporate their own sexual violence prevention effectiveness principles into their evaluation approach. This learning will be accompanied by examples from Washington state’s experience of engaging preventionists in the process of identifying principles – a project supported by the State Department of Health. If you are short on time, check out this overview on our podcast.

For more information on Principles-Focused Evaluation:

Principles-Focused Evaluation: The GUIDE (Book$) This book by Michael Quinn Patton provides a complete overview of Principles-Focused Evaluation and a guide for implementation.

References

Curtis, M. J., & Kukké, S. (2014). Activity-based assessments: Integrating evaluation into prevention curricula. Retrieved from the Texas Association Against Sexual Assault: http://www.taasa.org/wp-content/uploads/2014/09/Activity-Based-Assessment-Toolkit-Final.pdf

Davidson, E. J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.

Gamble, J. A. A. (2008). A developmental evaluation primer.. Retrieved from Better Evaluation: https://www.betterevaluation.org/sites/default/files/A_Developmental_Evaluation_Primer_-_EN.pdf

Guijt, I. (2014). Participatory approaches (Methodological brief #5). Retrieved from the United Nations Children’s Fund (UNICEF) Office of Research: https://www.participatorymethods.org/files/Participatory_Approaches_ENG%20Irene%20Guijt.pdf

Mertens, D. M. (2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 1(3), 212-225. doi:10.1177/1558689807302811