This case study reviews the evaluation of the Knox Greenleaf residential sustainability workshops using the ORID (Observation, Reflection, Insights, Decisions) process. These ORID focus group sessions were conducted on the 13th and 20th August 2009, and were undertaken as part of Knox’s participation in the Local Sustainability Accord-funded project to develop an evaluation toolkit for community engagement projects.

The Greenleaf workshops have been internally evaluated by Council staff through the use of post-workshop evaluation dartboards, as well as keeping track of number of attendees. The Council has also requested participants to provide copies of utility bills in order to look at changes in resource consumption.
Evaluation is a valuable exercise in order to assess the effectiveness of the project and to reflect on the lessons learned. Evaluation is a process to understand whether the theory of change that underpins an initiative such as Greenleaf holds true, and whether there are unintended consequences.

Evaluation can be considered in several ways:

• Performance evaluation measures variables such as the number of people attending workshops,
• Impact evaluation measures variables such as responses to surveys, number of light bulbs installed,
• Outcome evaluation measures the change in behaviour or resource use in relation to the target or goal of the project. Without thorough outcome evaluation, it is not possible to demonstrate whether a behaviour change project has had the desired effect.

The ORID method used to evaluate the Greenleaf project looks at outcome evaluation, in that it seeks to reveal the process of change based on the participants’ viewpoint. The ORID process is an example of a qualitative method, whereby the process of change is the focus, rather than a change in numbers. In comparison, the collection of utility bills is a quantitative method to evaluate project outcome, and the dartboard methods can be seen as means to evaluate both workshop performance and impact. The ORID methods can also be considered as a ‘goals-free’ evaluation, as it does not pre-empt the consequences. In comparison, the collection of utility bills is a ‘goals-based’ evaluation method as targets (such as 10% reduction in energy use) are determined prior to the project implementation.

This case study will be presented in the ORID format, in that it will review observations of the process, provide reflection and insights, and offer decisions as to its usefulness and applicability in a local government context. This is preceded by an explanation of the ORID process below.

The ORID process

ORID stands for Observation-Insights-Reflection-Decision. It is a structured discussion process that can be used for small groups (up 12 people). The first step in the ORID process is to ask the participants for their observations about the project (see Figure 1, Step 1). Once all the observations have been listed and grouped into like- themes, participants are asked to reflect on the first observation and state what they liked or did not like. Based on the reflections participants are then asked to provide insights into what this means for the project. The decision step provides some concrete and focussed actions that the Council (or organisation) can take away to improve projects or plan strategies and future initiatives. The process is repeated for each observation (Figure 1). Prompt questions to elicit responses are provided in Table below.

The ORID process

Prompt questions for the ORID steps

ORID of the ORID

DISCUSSION

The ORID process provides valuable qualitative information on the strengths and weaknesses of a project, based on the viewpoint of the participants (or other stakeholders). As such, the process can provide some valuable learnings to improve further phases of a project, or to design a new project or vision for future initiatives.

The strength of the ORID process is in its structure. The structured conversation allows observations to be teased out into what their meaning and implications are for the project. This allows participants to collectively make decisions on what works, what does not, and how things can be changed. However, the structured way of thinking does not come easily, and can be awkward for all involved. Having the facilitator explain the process up front, and having a prompt reminding participants about the structure may assist in keeping people on track.

The greatest difficulty in maintaining the structure was the seeming similarities between ‘reflections’ and ‘insights’. Following on from observations, it was found that participants often leapt into providing insights as to why they found something good or not. One way to overcome this would be for the Council or organisation running the session to agree to remove the ‘reflection’ component and run an ‘observation’ – ‘insights’ – ‘decision’ (OID) focus group instead. Reflecting back on the two sessions that were held in Knox, it is likely that the same level of detail would have come out of the participants had this been the case.

Lessons

• The facilitator needs to explain the ORID process up front and be confident to manage conversations so that they keep to the structure. This includes explaining a few rules upfront, such as listening to other people, respecting everyone’s opinion, and not having one person dominate the conversation. The facilitator needs to have the confidence to know when to step in and when to let conversations flow.
• It is also helpful to have a highly visible prompt reminding everyone of the ORID structure- this could be a powerpoint slide or writing on a whiteboard or the like.
• Facilitator needs to have a way to capture the stories. Recording the ORID session (with participants permission) and taking down notes/observations will help to achieve this.
• The Council or organisation running the session could consider removing the ‘R’ and run an ‘OID’ focus group.