The following case study is based on a summary of the GreenSaver Evaluation report (Gabrielle Stannus, City of Melbourne, December 2007)

Introduction

The City of Melbourne planned and delivered the GreenSaver Program between 2004 and 2006.  

The program aimed to achieve energy and water savings at a community level by conducting energy and water audits on households and providing participants with devices and advice to enable them to make further reductions following this intervention.

The initial objectives of the GreenSaver program were:
•    To recruit 400 households
•    To deliver energy and water audits to each household
•    To achieve 10 per cent reduction in greenhouse gases
•    To work with various stakeholders in achieving these goals
•    To evaluate the outcome with at least 30 per cent of participating house holds being monitored

The key objectives of the evaluation report were to:

• Determine residential energy and water savings made through GreenSaver
• Assess participants’ satisfaction with program delivery
• Provide recommendations to the City of Melbourne on future program development
• Provide feedback to project partners

Bennett’s Hierarchy was used as a framework to assess the performance of the GreenSaver Program. The components of Bennett's Hierarchy are outlined below:

• Inputs
  • Allocation of kinds and amount of resources to a program, e.g. time, costs
• Activities
  • Performance of specified activities (number and how well they are conducted)
• People
  • Level of contact between clients and advisory services
  • Number of people and groups participating in activities
  • Characteristics of participants
• Reactions
  • People’s reactions, eg. interest in educational activity, favourable responses
  • Recognition by participants of the credibility of resource people and information
• KASA (Knowledge, Attitudes, Skills and Aspirations) Change
  • A change in people’s knowledge, attitudes, skills and/or aspirations 
• Practice Change
  • The adoption of new practices, technology or changes in behaviour
• End Results
  • The end results of the practice change, e.g. reduction in energy  use

Inputs, activities and people can be considered “efficiency evaluation”, reactions, KASA and practice change can be considered “effectiveness evaluation” and end results can be considered “impact evaluation”.

Evaluation Methodology

Actual energy and water savings were measured by acquiring the resident’s gas, electricity and water billing data from the relevant distributors and/or retailers for the period of one year prior to the intervention point (i.e. audit) and one year after the intervention point. The decision to take this approach was made in order to provide at least some data to assess the GreenSaver program’s success.

Each distributor/retailer provided the City of Melbourne with a template consent form, or approved of the one drafted by council officers. The City of Melbourne then sent out these forms to each resident according to which distribution area they fell into.  Participants filled out the consent forms and signed them. On the return of these forms, the data was entered into a spreadsheet that was then forwarded on to the distributor/retailer along with a copy of the signed consent forms. Each distributor/retailer provided data where possible for a period of one year before and one year after the GreenSaver audit for each participating household as follows:

CitiPower and Powercor – Electricity consumption	Household identified by National Metering Identifier [NMI] Data collected from a minimum of four bills per year before and after audit
Envestra – Gas consumption	Household identified by its Meter Installation Reference Number [MIRN] only Data collected from a minimum of six bills per year before and after audit
City West Water - Water consumption	Household identified by account number only Data collected from a minimum of four bills per year before and after audit

 

Some households were also asked to record meter information for gas, electricity and water consumption at the time of the audit. These households were provided with a template to assist them to record this data simply. The intention was to follow up with similar meter readings after the audits to determine whether any electricity, gas and water savings were made as a result of the intervention. The collection of this meter information data was not stored in a central point, and also the exercise was not transferred across other audits, leading to a loss of the program’s historical data.

Results

In total, 67 households were audited throughout the two years, a number much lower than originally anticipated. While overall consumption was up in the areas of electricity and water, nearly half of the households had still managed to decrease their overall consumption.

 

Electricity consumption	Data received from 12 out of the 67 audited homes, of which data from only 9 homes was considered to be valid (13% of households) Increase of just over 10 per cent in greenhouse gas emissions from electricity after the audit Only 4 of 9 homes experienced a decrease in electricity consumption
Gas consumption	Data received from 17 out of the 67 audited homes, of which data from only 9 homes was considered to be valid (13% of households) Decrease of 2.6 per cent in greenhouse gas emissions from gas consumption after the audit 5 of 9 homes experienced a decrease in gas consumption
Water consumption	Data received from 17 out of the 67 audited homes, of which data from 14 homes was considered to be valid (21% of homes) Increase of just over 1 per cent in water consumption after the audit

Discussion

The first limitation is that the evaluation is unable to directly relate energy and water savings made to participation in the program as mostly quantitative data has been used in the evaluation.  Thus, certain assumptions have been necessary.

In addition, only those households whose data conformed to the requirements listed under each distributor/retailer in the “Methodology” section were included in the final evaluation of consumption data. Incomplete data was received from the distributors/retailers for several households, i.e. incorrect meter numbers, discrepancies unable to be checked, severe outliers, and incomplete billing data that only covered part of a year (e.g. when a participant moved into or out of their residence for a period of less than a year before or after their audit). Data from these households was not included in the final evaluation.

In future, seasonal climatic variations need to be monitored as part of ongoing programs in order to determine the influence on energy and water consumption.

The environmental outcomes of GreenSaver are questionable given the lack of data able to be collected and analysed. An overall increase in electricity (10 per cent) and water consumption (1.15 per cent) occurred in those GreenSaver homes monitored, while gas consumption decreased slightly (2.6 per cent). However, with such a small sample size these results cannot be generalised across all households participating in GreenSaver, or indeed the wider residential sector of City of Melbourne.

Data-gathering options available for future household sustainability programs include smart (or interval) metering as well as more traditional methods of manual readings of existing meters.