English French German Italian Portuguese Russian Spanish
Tools Metering

Using the ORID focus group method with the City of Knox Greenleaf program

This case study reviews the evaluation of the Knox Greenleaf residential sustainability workshops using the ORID (Observation, Reflection, Insights, Decisions) process. These ORID focus group sessions were conducted on the 13th and 20th August 2009, and were undertaken as part of Knox’s participation in the Local Sustainability Accord-funded project to develop an evaluation toolkit for community engagement projects.


The Greenleaf workshops have been internally evaluated by Council staff through the use of post-workshop evaluation dartboards, as well as keeping track of number of attendees. The Council has also requested participants to provide copies of utility bills in order to look at changes in resource consumption.
Evaluation is a valuable exercise in order to assess the effectiveness of the project and to reflect on the lessons learned. Evaluation is a process to understand whether the theory of change that underpins an initiative such as Greenleaf holds true, and whether there are unintended consequences.


Evaluation can be considered in several ways:

  • Performance evaluation measures variables such as the number of people attending workshops,
  • Impact evaluation measures variables such as responses to surveys, number of light bulbs installed,
  • Outcome evaluation measures the change in behaviour or resource use in relation to the target or goal of the project. Without thorough outcome evaluation, it is not possible to demonstrate whether a behaviour change project has had the desired effect.


The ORID method used to evaluate the Greenleaf project looks at outcome evaluation, in that it seeks to reveal the process of change based on the participants’ viewpoint. The ORID process is an example of a qualitative method, whereby the process of change is the focus, rather than a change in numbers. In comparison, the collection of utility bills is a quantitative method to evaluate project outcome, and the dartboard methods can be seen as means to evaluate both workshop performance and impact. The ORID methods can also be considered as a ‘goals-free’ evaluation, as it does not pre-empt the consequences. In comparison, the collection of utility bills is a ‘goals-based’ evaluation method as targets (such as 10% reduction in energy use) are determined prior to the project implementation.


This case study will be presented in the ORID format, in that it will review observations of the process, provide reflection and insights, and offer decisions as to its usefulness and applicability in a local government context. This is preceded by an explanation of the ORID process below.


The ORID process

ORID stands for Observation-Insights-Reflection-Decision. It is a structured discussion process that can be used for small groups (up 12 people). The first step in the ORID process is to ask the participants for their observations about the project (see Figure 1, Step 1). Once all the observations have been listed and grouped into like- themes, participants are asked to reflect on the first observation and state what they liked or did not like. Based on the reflections participants are then asked to provide insights into what this means for the project. The decision step provides some concrete and focussed actions that the Council (or organisation) can take away to improve projects or plan strategies and future initiatives. The process is repeated for each observation (Figure 1). Prompt questions to elicit responses are provided in Table below.

The ORID process



Prompt questions for the ORID steps

ObservationsReflectionsInsightsDecisions

“What is your most significant observation about the project?”

“What do you remember?”

“What did you hear or see?”

“Was this exciting or surprising?”

“Did this engage your interest or empower you to become more sustainable?”

“Did you like this or not?”

“Did this frustrate you?”

“Why did this aspect work or not work?”

“What does this mean for the project?”

“What are the implications?”

“What are the options?”

“What changes are needed?”

“How would you prioritise changes?”

“What would you say about the project to someone who was not there?”

“Having experienced and reflected on this project, what are your next steps?”

ORID of the ORID

ObservationsReflectionsInsightsDecisions

Thinking in the ORID framework can be awkward.

This can be challenging if the group starts getting off track or very engaged in a topic

Learnings can be hard to capture if the structure is not kept to.

Facilitator needs to explain the ORID process up front and be confident to manage conversations so that they keep to the structure.

The difference between ‘reflection’ and ‘insights’ can be difficult for participants, and the facilitator to get their head around.

This can be frustrating as it is not the ‘normal’ way of thinking for many people.

The structure of the ORID process is its strength as it can allow lessons to be captured for each observation in a clear manner, but it often prevents the normal flow of a conversation. By getting participants to think about observations first, and then pooling like ones together, the facilitator can then proceed through the “RID” stages. Alternatively, it may be possible to agree beforehand to skip ‘reflection’ and go from ‘observation’ to ‘insight’ then ‘decision’.

It is important for the facilitator to explain the ORID process. It is also helpful to have a highly visible prompt reminding everyone of the ORID structure- this could be a powerpoint slide or writing on a whiteboard or the like.

The Council or organisation running the session could alternatively agree to remove the ‘R’ and run an ‘OID’ focus group.

Some participants can dominate the conversation.

This can be challenging for the facilitator and other participants.

It is important for the facilitator to be confident enough to steer the conversation to other people’s points of view, and to keep to the structured process.

Facilitator needs to explain a few rules upfront, such as listening to other people, respecting everyone’s opinion, and not having one person dominate the conversation

The participants provide very interesting and meaningful observations about the project delivery and results.

This type of information is really valuable in terms of understanding how a project leads to change.

The participants’ stories provide valuable lessons for the project implementation or further design and delivery.

Facilitator needs to have a way to capture the stories. Recording the ORID session (with participants permission) and taking down notes/observations will help to achieve this.

Having like-minded people together in a room can create a ‘buzz’.

The excitement created once conversations start helps participants draw out new observations and discussion.

Facilitator needs to be able to encourage conversations and maintain the buzz, all at the same time as keeping the structure of the process.

The facilitator needs to have the confidence to know when to step in and when to let conversations flow.



DISCUSSION

The ORID process provides valuable qualitative information on the strengths and weaknesses of a project, based on the viewpoint of the participants (or other stakeholders). As such, the process can provide some valuable learnings to improve further phases of a project, or to design a new project or vision for future initiatives.


The strength of the ORID process is in its structure. The structured conversation allows observations to be teased out into what their meaning and implications are for the project. This allows participants to collectively make decisions on what works, what does not, and how things can be changed. However, the structured way of thinking does not come easily, and can be awkward for all involved. Having the facilitator explain the process up front, and having a prompt reminding participants about the structure may assist in keeping people on track.


The greatest difficulty in maintaining the structure was the seeming similarities between ‘reflections’ and ‘insights’. Following on from observations, it was found that participants often leapt into providing insights as to why they found something good or not. One way to overcome this would be for the Council or organisation running the session to agree to remove the ‘reflection’ component and run an ‘observation’ – ‘insights’ – ‘decision’ (OID) focus group instead. Reflecting back on the two sessions that were held in Knox, it is likely that the same level of detail would have come out of the participants had this been the case.

Lessons

  • The facilitator needs to explain the ORID process up front and be confident to manage conversations so that they keep to the structure. This includes explaining a few rules upfront, such as listening to other people, respecting everyone’s opinion, and not having one person dominate the conversation. The facilitator needs to have the confidence to know when to step in and when to let conversations flow.
  • It is also helpful to have a highly visible prompt reminding everyone of the ORID structure- this could be a powerpoint slide or writing on a whiteboard or the like.
  • Facilitator needs to have a way to capture the stories. Recording the ORID session (with participants permission) and taking down notes/observations will help to achieve this.
  • The Council or organisation running the session could consider removing the ‘R’ and run an ‘OID’ focus group.
 
Donate Now

Please make a donation to upgrade the Evaluation Toolbox.

The Evaluation Toolbox is maintained by Damien Sweeney and Martin Pritchard from PREA as their in-kind contribution back to the community. The Toolbox now needs several weeks of work to review and upgrade all the contents and add new content for you. Work has begun and we are seeking your donation (big or small) to help support this major upgrade. Email us to indicate what you want to be included in the Toolbox.

case study

What is the Toolbox?

The toolbox aims to provide a one-stop-site for the evaluation of community sustainability engagement projects that aim to change household behaviours. Through the toolbox you can learn how to conduct your own evaluation of a behaviour change project using the guides and templates provided.

see more...

Why Evaluate?

Conducting an evaluation is considered good practice in managing a project.

The monitoring phase of project evaluation allows us to track progress and identify issues early during implementation, thus providing and opportunity to take corrective action or make proactive improvements as required.

see more...

 

DSE
City of Whitehorse City of Whitehorse
Accord