Formative Evaluation of the Living Green in Gannawarra & Surrounds Program
Introduction
The Shire of Gannawarra and its stakeholder partners are in the process of finalising the design of a community engagement and behaviour change program funded under the Victorian State Government’s Local Government Sustainability Accord. The program, called Living Green in Gannawarra & Surrounds (LGG), is a two year sustainable living program that is being delivered in targeted communities in the Shires of Gannawarra, Buloke, Loddon and Swan Hill. The project intends to build on existing sustainable living programs (Green Homes, Sustainable Homes Program etc) but use existing networks and community groups to target the participants, rather than have open workshops. This approach has been identified as a pilot concept for these Shires because the project partners have identified limitations of success with workshops with an open invitation. Preliminary surveys found the idea of ‘attending additional’ meetings did not appeal to residents in these rural communities and so the idea of presenting to existing groups was borne.
The LGG program intends to build the capacity of communities to attain information and incentives to encourage long term behaviour change, by working with Community Resource Centres and Neighbourhood Houses and their partnerships with local organisations and other key resource providers through existing networks. The project is planned to be piloted over 2 years, and is designed to:
• Understand the barriers to environmental behaviour change in rural communities
• Understand the opportunities that exist to educate rural communities about sustainable living
• Provide practical advice and resources to help households live more sustainably
• Achieve reductions in water and energy use and increase participation in recycling and sustainable consumption
The objectives of the LGG program, as outlined in the funding proposal, are to:
• Connect and engage with the rural community through existing networks
• Understand the barriers to environmental behaviour change in rural communities
• Understand the opportunities that exist to educate rural communities about sustainable living
• Utilise modified resources (eg ACF’s Greenhomes) for rural communities
• Provide practical advice and resources to help households live more sustainably
• Achieve reductions in water and energy use and increase participation in recycling and sustainable consumption using a suitable evaluation tool to measure improvement
The project stakeholders propose to meet the LGG objectives through the delivery of workshops on three themes:
• Energy conservation
• Water conservation
• Waste & recycling.
Formative evaluation of the project design was undertaken prior to its rollout in order to clarify the logic of the intervention. This included a Logical Framework Approach (LFA) workshop, followed by a Program Logic workshop. The LFA workshop consisted of undertaking a stakeholder analysis and problem tree prior.
Logical Framework workshop
A LFA workshop was held in Bendigo on 28 January 2010. The workshop was facilitated by the National Centre for Sustainability, Swinburne University of Technology, and attended by project stakeholders from Shire of Gannawarra and Regional Waste Management Group. A representative from the City of Whitehorse, overseeing the Accord-funded Evaluation Toolbox project, also attended the workshop.
The LFA provides a tool to clarify the issue that an intervention seeks to resolve by outlining the core issue, and the chain of causes and effects that lead to the issue occurring. Though the LFA represents a linear logic model, it provides an important tool to assess how an intervention, and its theory of change, will overcome a particular chain of cause and effects. This helps clarify the likelihood that an intervention will work, and what other cause and effects leading to a core issue are left to be tackled by other complementary projects (or left out altogether, and thereby potentially have an impact on the likelihood of the intervention working).
Stakeholder assessment
The first part of the LFA workshop was to undertake a stakeholder analysis, which provides a means to identify the relevant stakeholders (including target group) and assess their views and support for the proposed project. This was constrained as all the project stakeholders were not represented at the workshop. As the LGG program has quite a number of diverse stakeholders in the project design phase, it is important to articulate their motivations so that all stakeholders are aware of them, and how this needs to be incorporated into the intervention (as well as identifying aspects that may need to be left). It is also important to identify and analyse the motivations and constraints of the target group(s). This is especially important where there is diversity in the target groups, which include urban, peri-urban and rural households. Ideally, some sort of social research would be used to obtain this information, but this was lacking at the time of the workshop.
Developing a problem tree and solution tree
The second part of the LFA workshop was to identify the core problem that the project seeks to overcome. This necessitated breaking down the LGG into three distinct sub-projects, based on the workshop topics of energy, water and waste. Discussions then commenced as to the core problem within these three themes. Identifying the core question proved to be difficult as the total range of project stakeholders were not present and the people who originally put the LGG program together were not present. In the end, it was decided that the energy theme related specifically to electricity use in the home, as opposed to gas. The waste theme was focussed on reducing household waste going to landfill.
The process of identifying the causes that lead to the core problem also revealed the range of diverse causes and their effects. This is compounded by the diversity of the target group in the LGG program, which encompasses urban, peri-urban and rural households, and each has particular causes that lead to the core problem, including institutional and regulatory ones that cannot easily be addressed by the theory of change and mode of intervention (workshops).
The problem tree was turned into a solution tree by reversing negative statements into positive ones. The branches of the solution tree help identify the different chain of solutions and their effects the lead to the core problem being resolved. This part of the LFA allows stakeholders to identify which branch(es) to tackle in an intervention, and thereby overlay a theory of change over that branch. In the case of the LGG program, and as is the case of many behaviour change programs, an intervention model (workshops) was already selected prior to completing the problem and solution tree. Therefore, undertaking the LFA at this stage allows the project design stakeholders to clarify if the already selected intervention matches the identified problems and solutions, or whether the proposed model needs to be reworked before further investment is made.
The Logical Framework Matrix
The Logical Framework Matrix (LFM) provides a summary of the preferred approach from the solution tree. The LFM outlines the goal, purpose, outputs and activities that are needed to achieve the goal of the project. The LFM is typically structured as a table with the vertical headings representing the logic (cause and effect) and the horizontal headings representing how activities will be measured. This provides a framework for monitoring and evaluation and is similar to the program logic model and evaluation plan.
The workshop identified the need to develop an LFM for each of the workshop themes, and the likelihood that LFMs would also need to be developed for the different target audiences if the different household situations led to significant changes in the changes being sought (eg. households connected to town water and those that are not). The workshop also clearly demonstrated that a significant amount of time needs to be allocated to develop the LFA and subsequent LFM.
Program Logic and Evaluation Plan workshop
The LGG project staff organised a program logic and evaluation workshop, facilitated by Roberts Evaluation, on 12 March 2010. The workshop was facilitated by Roberts Evaluation, and attended by project stakeholders from Shire of Gannawarra and Regional Waste Management Group. Representatives from the City of Whitehorse and the NCS also attended.
The development of the program logic involves a process similar to that of the LFA, but focuses on collectively identifying foundation activities and activities, which feature “doing” words. This is followed by identifying the immediate, intermediate, long term and aspirational outcomes. The facilitators asked the workshop members to write down actions and outcomes on sticky notes and then post them on a wall beside the relevant headings. Following this, all the workshop participants were invited to walk up to the wall and review the collection of notes, and discuss which needed re-arranging (from one level to another) or rewording (rewording activities into verbs, or outcomes into past tense, and measurable). The facilitator encouraged all the participants to discuss what needed moving and why, so that there was collective understanding and acceptance of the final logic model.
During the process of moving activities and outcomes, the facilitator made a point of asking the workshop participants to think about the theory of change that underpinned the intervention, as this would assist in identifying what were activities, and at what level outcomes should be placed. This led to some discussion as to the theory of change for the LGG. The original project proposal states that the intervention builds on existing sustainable living program (Green Homes, Sustainable Homes Program etc) but uses existing community groups and networks to recruit participants. The discussion brought to light that the LGG seeks to engage particular community groups and networks and provide targeted rural-relevant information on sustainable living, highlighting relevant incentives and programs which they can access to assist in the adoption of more sustainable behaviours. It was also noted that the LGG did not have set or firm targets, and the amount of change was flexible.
One of the lessons from the program logic workshop, which backed up the conclusion from the LFA workshop, was that it can be difficult for project staff to have a clear understanding of the theory of change when a project presupposes a delivery model (i.e. workshops) without taking the preceding steps of clearly identifying the root causes of the core problem. There is also a lack of distinction between community engagement and behaviour change projects, and though the former can lead to changes in behaviour, the process and pathways to change occurring are less clear.
Discussion
Formative evaluation of project designs is demonstrated to be important in clarifying the logic that underpins the likelihood of success in meeting goals and outcomes. This is especially important if the logic behind a proposed project has not been mapped out prior to being awarded funds for implementation, as can often be the case. This is similar to the issue of choosing a behaviour change strategy prior to undertaking the required research to see whether the strategy fits the needs- or making sure the means justifies the end. The stakeholder analysis component of the LFA, which seeks to identify the motivations and constraints to change, has functional similarities to the barrier and benefit research element of Community Based Social Marketing (CBSM). Too often, this type of research or analysis is not undertaken, and the result is to fit a desired theory of change onto a problem, rather than analysing the problem and its many causes, and then selecting the best theory of change to reach the goal.
The LFA is also a good process to bring stakeholders together to discuss and agree on the design and implementation phase of a project. This is especially important in the case of projects that involve many organisations, and where each may have different goals and interests. In such cases, the project may be both difficult to implement and evaluate if all the parties are not on the same page.
Once a broad strategy to effect a behaviour change is selected, the program logic is an effective tool to clarify the link between activities that are undertaken and their immediate, intermediate and long term outcomes. This assists in understanding what can be evaluated in the short term, or within the project’s funded life. This is a critical aspect as all too often, the evaluation questions are targeted towards quantification of behaviour change impacts that are more likely to be long term outcomes, and therefore beyond the funded life of a project (and thus evaluation life). A program logic clarifies the realistic expectation of the outcomes of a project. A program logic can lead to the development of an evaluation plan.
Lessons
- An LFA (including stakeholder analysis and problem tree) should precede the use of program logic if there is uncertainty as to whether the selected theory of change actually fits the causes of the core problem in question.
- Allocate enough time to undertake an LFA workshop (a full day minimum).
- Undertake a stakeholder analysis prior to the LFA workshop, especially identifying the motivations and constraints of the target group. This is similar to the barriers and benefits research that is recommended as part of CBSM.
- Have as many of the projects stakeholders present as possible for both the LFA and program logic to ensure that all parties agree on the output.
- An independent facilitator is recommended in undertaking an LFA and program logic in order to reduce the chance that particular stakeholders impose their view or logic upon others.