English French German Italian Portuguese Russian Spanish

Types of evaluation

Evaluation can be characterised as being either formative or summative (see Table below). Broadly (and this is not a rule), formative evaluation looks at what leads to an intervention working (the process), whereas summative evaluation looks at the short-term to long-term outcomes of an intervention on the target group.

Formative evaluation

Formative evaluation takes place in the lead up to the project, as well as during the project in order to improve the project design as it is being implemented (continual improvement). Formative evaluation often lends itself to qualitative methods of inquiry.

Summative evaluation

Summative evaluation takes place during and following the project implementation, and is associated with more objective, quantitative methods. The distinction between formative and summative evaluation can become blurred. Generally it is important to know both how an intervention works, as well as if it worked. It is therefore important to capture and assess both qualitative and quantitative data.

Types of evaluation


Formative
Summative
Type of Evaluation
ProactiveClarificative InteractiveMonitoring Outcome

When to use

Pre-project

Project development

Project implementation

Project implementation

Project implementation and post-project

Why use it?

To understand or clarify the need for the project

To make clear the theory of change that the project is based on

To improve the project’s design (continual improvement) as it is rolled out

To ensure that the project activities are being delivered efficiently and effectively

To assess whether the project has met its goals, whether there were any unintended consequences, what were the learnings, and how to improve

Source: Owen & Rogers (1999)

 

Goals-based evaluation

Evaluation has typically involved measuring whether predetermined targets have been met. You may be familiar with the term ‘SMART’ (Specific, Measurable, Attainable, Relevant, and Timely) targets. These type of targets fall under the banner of goals-based evaluation.

Goals-based evaluations have objectively-set targets usually determined by people responsible for the funding or implementation of the project.  Goals-based evaluation does not question whether the selected goals are valid, nor whether appropriate measures of effectiveness are being assessed.
Recent behaviour change and research and evaluation practice proposes the use of goals-free evaluation.

Goals-free evaluation

Goals-free evaluation looks at emergence and unintended consequences of a project, by looking at the actual effects without pre-empting what these may be. This type of evaluation focuses on the change process within the target group. The goals-free evaluation parallels a social learning approach to evaluation which can be desirable where the goal of a project seeks to build capacity of people to manage a complex issue. In such cases, relying solely on a goals-based evaluation may not be appropriate as it can be difficult to set targets and quantify a process of change, as well as capture change which one may not anticipate.

A key point to consider in setting goals and indicators of success is the validity of the indicator. This is especially important when you are looking to measure a largely “unobservable” behaviour (for example, those that happen within the home, and cannot be readily objectively observed). In such cases, proxy indicators are often used (for example, household electricity use) but these indicators may be gross measures that consist of many varied behaviours, of which only one may be of interest. The question is how to establish meaningful indicators?

 
Donate Now

Please make a donation to upgrade the Evaluation Toolbox.

The Evaluation Toolbox is maintained by Damien Sweeney and Martin Pritchard from PREA as their in-kind contribution back to the community. The Toolbox now needs several weeks of work to review and upgrade all the contents and add new content for you. Work has begun and we are seeking your donation (big or small) to help support this major upgrade. Email us to indicate what you want to be included in the Toolbox.

case study

What is the Toolbox?

The toolbox aims to provide a one-stop-site for the evaluation of community sustainability engagement projects that aim to change household behaviours. Through the toolbox you can learn how to conduct your own evaluation of a behaviour change project using the guides and templates provided.

see more...

Why Evaluate?

Conducting an evaluation is considered good practice in managing a project.

The monitoring phase of project evaluation allows us to track progress and identify issues early during implementation, thus providing and opportunity to take corrective action or make proactive improvements as required.

see more...

 

DSE
City of Whitehorse City of Whitehorse
Accord