English French German Italian Portuguese Russian Spanish
Tools Outcome Hierarchy

Outcome Hierarchy

Outcome Hierarchy (OH) is an often-used process in project design as a way to clarify the program logic. As an evaluation tool, OH can be used to identify  what capacity that has been created by the project. The information provided can be used for formative evaluation for future project design, as well as outcome evaluative, to show how participation has shifted worldviews in the interests of the project’s intended goals. It allows participants to assess the project’s success, and provide a participant’s view as to the potential for long-term change. OH also provides a means for participants to review their participation against the project’s design, thereby providing an assessment as to whether the planning was accurate.

OH as formative evaluation

Allows stakeholders (including target group) to participate in the project design and provide input as to the activities, objectives and goals of a project. This can be undertaken from scratch, or by reviewing participants’ views of a prior project.

OH as summative evaluation Allows stakeholders (including target group) to report on the projects’ activities and goals from their worldview and interests and provide a grounded assessment of the success and learnings, and follow-up initiatives.

OH is different to measuring workshop satisfaction or quantifying impacts. It rather focuses on reflective thinking on the process of change, and how a project can be built on to further strength or embed change for sustainability.

The OH process can be time consuming, and preferably requires someone with skills in facilitating group interaction. The OH process can also be modified to limit the reflection to the project’s activities, and skip the long-term visioning.

OH can assist participants to evaluate the process, impact and outcome of a project, especially with a focus on long term consequences.

Pros and Cons of Outcome Hierarchy

Pros Cons

Provides valuable information from context of participants (and stakeholder) experiences

Requires good facilitation skills

Structured thinking that focuses on personal reflection of the project from participant viewpoint

Participants need to have a clear understanding of the process, including need for personal reflection

Provides contextual data for future project improvements or developing new initiatives

Can be time consuming, especially if undertaken in full

 

A case study on the use of the outcome hierarchy process to evaluate the Bayside REAP project is available here.

How to Conduct a Post-Project Outcome Hierarchy Workshop

The Outcome Hierarchy (OH) works as a staged process to evaluate the change that results from the implementation of a project, as well as to design a new project, or follow up project.  It works by getting project stakeholders (participants, or others) to establish a baseline state (how things are currently). This is followed by stakeholders agreeing on a project goal or targets. Participants are then able to determine the gap or need(s) between the current baseline and intended goal. The need is then responded to with agreed actions that would allow the need to be met and the goal to be realised.


The OH exercise tracks through the likely consequences of goal attainment. It carries out this exercise in three stages with each new set of consequences building on an imagined accomplishment of the previous set. The exercise concludes with an “ultimate outcome” and the attribution of indicative timelines for each stage.


The following generic template describes an OH. It is read from the baseline up, in order to assist with the inductive movement of envisioned thinking.

The Outcome Hierarchy Map

Ultimate Outcome

Time:

What change is likely to happen as a consequence of the intermediate outcome?

Intermediate Outcome

Time:

What change is likely to happen as a consequence of the immediate outcome?

Immediate Outcome

Time:

What change is likely to happen as a consequence of the impacts?

Immediate Impact

Time:

What difference would we see at the time that our goal or target is achieved?

Action

 

What actions do we need to take to meet the need and realise the goal or target?

Need

 

What is the gap that currently exists to achieve the goal?

Project goal or target

 

What is the Project aiming to deliver?

Current State

 

What is the current situation or capacity?

Requirements

  • A facilitator to guide the discussion, prompt questions, summarise themes, and ensure everyone has a say
  • Small squares of paper (with sticky tape or reusable adhesive, eg. blu-tack) or post-it notes
  • A wall, or “sticky wall”  -  A sticky wall is a vinyl material that has been coated with repositionable spray adhesive. These can be purchased from certain suppliers, or you can make one yourself. For instructions on how to make your own, see page 22 of Developing and Using Program Logic.

Developing and using program logic user guide

  • Set up chairs in a U-shape with the  out or whiteboard at the front
  • About 3.5 hours, and some refreshments

A sticky wall used in the Outcome Hierarchy process for the Bayside REAP project

Sticky wall

Process

To help understand the process, it is advised that you refer back to the OH Map, as well as an example of a completed OH (link to below) as a guide.

Step 1- provide an introduction to the OH process

The facilitator gives an overview of the OH and how it fits in the evaluation of the project. It is advisable to explain its purpose, and the intended audience of the evaluation report.

It is important to emphasise that the OH process requires participants to reflect on their personal experience and requesting evaluative judgements from this standpoint.

Step 2- setting the baseline

The first stage of the OH process develops data for setting the baseline of the hierarchy – in this instance, the capacity that the program has built for each participant (if the OH is being used in a formative evaluation, it would be the existing capacity in the community). Participants are invited to think about what difference the project has made for them and to describe a specific event that illustrates this difference being realised in their experience of the initiative.


This exercise can be done through experiential learning practices. For example, participants can use mapping  and story boards to assist them to think through their narratives.  This can be done in pairs or groups of three, to help each other tell their story and document it (storyteller plus documenter). It is important to remind and encourage participants to speak from the first person about a specific real event. The common tendency is to talk about generalised experience rather first person action and impression in a local, real situation.

A Storyboard used in the Bayside REAP project

Story board

A Story Map used in the Bayside REAP project

Story Map

The second stage is to share the stories and analyse them in a collective manner to identify the built capacity in each person’s case. The maps and storyboards are placed on a wall and the participants gather around, listening to each person tell their story. The facilitator assists with clearly identifying the built capacity, and teasing out any capacities that may be implied through the stories but not necessarily noticed by the participants.

The participants then gather around a large group map of a blank Outcome Hierarchy (link to OH Map). This can be drawn up on a whiteboard, a “sticky wall” or using butcher’s paper. Participants are then invited to write their new capacities on their individual pieces of paper (eg. post-it notes), and placed them on the wall.

Getting participants to write on separate pieces of paper overcomes the potential for collusive thinking that can take place in groups, where people may replicate the thoughts others rather than engage in self-reflection. The different thoughts are then viewed by all and grouped under common themes to create shared themes or strategies. The grouped themes are then placed in the base rung of the hierarchy (Current State).

Step 3- determining the goal

Participants are then asked to think about the overarching issue (eg. greenhouse gas reduction) and either adopt pre-existing project goal(s) or target(s) or propose new ones themselves. There may be more than one goal to arrive at what the project needs to achieve for the issue to be addressed. The goal(s) or target(s) are written on a piece of paper (one piece per goal or target) and placed in the next line of the OH (Project Goal or Target).
Step 4- identifying the need and actions
The participants are then invited to consider the gap that exists between the new current capacity and the intended goal, to identify the need that the project needs to meet for the goal to be met. The ways in which the needs are met are understood as actions. Each need should have at least one action. As before each element is given its own piece of paper and placed in the appropriate location on the Hierarchy.

While the Hierarchy can be worked with in a linear manner, each capacity leading to a potential goal, with a need and action to meet it, group discussion following completion of each row can critique the inter-connectedness between items to produce an integrated rather than linear analysis. This however takes significantly more time and is more appropriate for a whole day event rather than a several hours.

Step 5- indemnifying the impact

The participants are then encouraged to imagine that the actions have been taken, the needs met and the goal attained. They are then asked to describe what difference would be seen at the time that the goal is met, write this on individual pieces of paper which are the grouped in like themes. This becomes a description of impact, and if done well can provide good indicator data for evaluation and monitoring of future projects.

The following steps can be difficult for participants, and require encouragement through good facilitation.  Alternatively, you may decide to complete the exercise at the Impact level.

Step 6- Visioning the long-term future

Once the impact is written up, participants are invited to change gears, and enter into an envisioning state of mind. The exercise is less concerned with the practicalities and logistics of project delivery and now envisions the meaningful consequences of successful delivery as a values-based measurement system.

Participants are invited to image what change might come about as a consequence of the project’s agreed impacts. This can be a general discussion about “change” or a more specific discussion about particular aspects of change such as environmental, social, economic, cultural or governance qualities of change for example.

The participants notate their ideas and they are placed or written on the OH at the immediate outcome line. They are then invited to image that these outcomes have taken place within an “immediate” time of the program being completed (say, 3-6 months depending on the scale of the program), and to then imagine what change would be likely to come about as a consequence of the immediate outcomes taking place. Their ideas are placed or written onto the OH at the intermediate outcome line and an estimated time agreed. If there is a significant variation of the estimated time, an item may be moved up to the ultimate outcome line of the OH.

The process is repeated a final time to reach the ultimate outcome line, and a time attributed to it (this can be a year or two up to decades or longer, depending on the focus of the program).

 

Step 7 (this can be undertaken following Step 5 if Step 6 is not undertaken)

In closing, the participants are asked to make recommendations about changes to the implementation or what worked really well, given the potential of the capacity that it has built for them. The Facilitator supports participants to limit their recommendations to the information they have developed in the Outcome Hierarchy rather that introduce new information of a different nature at this stage.

Step 8- Evaluating the OH process

On completing the OH, participants can be asked to reflect on their experience of completing the OH. This can be done by placing comments down anonymously on individual notes for people to read, or have an overt conversation depending on the time and feeling of the event.
This evaluation of the evaluation provides a critical review of the quality of the content that has been developed during the process, and helps inform the final report.


Example Of Completed Oh (Bayside Reap- Residents)
Comleted outcome hierarchy for Bayside REAP project


Tips to Using an Outcome Hierarchy for Evaluation

  • Prepare participants for the process by outlining what the OH is and its use as an evaluation tool. This will help avoid the potential for participants’ confusion that evaluation is necessarily quantitative and reductive.
  • Allow for a minimum of 3.5 hours so that adequate reflection on experience and story telling can take place. This process is time-expensive but essential to allow participants to self-determine their capacity building on which the Hierarchy depends.
  • When participants turn their attention from the narratives to the OH provide some reorientation exercise about the relationship between the present and the future. This will help participants understand the value of their thinking.
  • Encourage the envisioning of both positive and negative outcomes throughout the OH exercise to more adequately balance the data against idealism or despair.
  • Allow for group discussion about each stage of OH completion to afford critical thinking (questioning of assumptions, offering of alternatives).

 

Case Study

A case study on the use of the outcome hierarchy process to evaluate the Bayside REAP project is available here. The outcome hierarchy evaluation and case study for Bayside was prepared by Dr Susan Goff of Cultureshift.

 
Donate Now

Please make a donation to upgrade the Evaluation Toolbox.

The Evaluation Toolbox is maintained by Damien Sweeney and Martin Pritchard from PREA as their in-kind contribution back to the community. The Toolbox now needs several weeks of work to review and upgrade all the contents and add new content for you. Work has begun and we are seeking your donation (big or small) to help support this major upgrade. Email us to indicate what you want to be included in the Toolbox.

case study

What is the Toolbox?

The toolbox aims to provide a one-stop-site for the evaluation of community sustainability engagement projects that aim to change household behaviours. Through the toolbox you can learn how to conduct your own evaluation of a behaviour change project using the guides and templates provided.

see more...

Why Evaluate?

Conducting an evaluation is considered good practice in managing a project.

The monitoring phase of project evaluation allows us to track progress and identify issues early during implementation, thus providing and opportunity to take corrective action or make proactive improvements as required.

see more...

 

DSE
City of Whitehorse City of Whitehorse
Accord