English French German Italian Portuguese Russian Spanish
List of Case Studies The Ambassadors’ Stories- a case study on using the Most Significant Change method with the Whitehorse Sustainable Ambassadors

The Ambassadors’ Stories- a case study on using the Most Significant Change method with the Whitehorse Sustainable Ambassadors

Introduction

The Most Significant Change (MSC) is a method of participatory evaluation that involves the collection of significant change stories at different levels of the intervention (for example program staff, change agents, intervention participants) and collectively deciding on the most significant change stories based on selected themes (called domains). The domains reflect broad categories, such as change in capacity to take action, or a change in participation in an activity, as well as lessons learned.


MSC has its roots in the overseas development field but is becoming more widely used in community development and behaviour change interventions in Australia.


Benefits of using the MSC method are listed as (Davies & Dart 2005: p12):

  • It is a good means of identifying unexpected changes.
  • It is a good way to clearly identify the values that prevail in an organisation and to have a practical discussion about which of those values are the most important.
  • It is a participatory form of monitoring that requires no special professional skills. Compared to other monitoring approaches, it is easy to communicate across cultures. There is no need to explain what an indicator is. Everyone can tell stories about events they think were important.
  • It encourages analysis as well as data collection because people have to explain why they believe one change is more important than another.
  • It can build staff capacity in analysing data and conceptualising impact.
  • It can deliver a rich picture of what is happening, rather than an overly simplified picture where organisational, social and economic developments are reduced to a single number.
  • It can be used to monitor and evaluate bottom-up initiatives that do not have predefined outcomes against which to evaluate.


The MSC offers a qualitative approach to monitoring that does not employ quantitative indicators, though it is possible to have a count of the stories in each domain. Davies & Dart (2005) note that MSC should be used as a complement to other evaluation methods rather than a stand-alone technique.


The Sustainable Ambassador experience

The MSC approach was trialled with the Whitehorse Sustainable Ambassador cohort of 2009. The Sustainable Ambassador program involves the delivery of training in behaviour change theory and technique to a group of community members, who then go on to undertake a project of their own (individually or as a group). The 2009 cohort undertook accredited training through the delivery of the Diploma of Sustainability unit on implementing a sustainable behaviour change project (VPB436).


The MSC approach was selected as an appropriate tool to evaluate the project as it focuses on capturing the range of participants’ experiences through a story telling approach. It is also a beneficial approach as the participants undertook a wide range of projects which they evaluated themselves using a range of methods. It was therefore difficult to develop meaningful common indicators that covered the range of projects that were undertaken.


The intention was for the project delivery staff (consisting of City of Whitehorse and the National Centre for Sustainability (NCS), Swinburne University of Technology) to collect stories from the Ambassadors and the Ambassadors to collect stories from the people they targeted as part of their own projects. In the end, only a few of the Ambassadors were able to collect stories from the people they targeted, but the project staff managed to collect stories from all of the Ambassadors. The implementation of the MSC approach is described below.

Story Collection

The collection of MSC stories was generally based on the process outlined in the MSC Quick-Start Guide (Dart & Davies 2003).


The original intention was for the MSC approach to be used by the Sustainable Ambassadors to collect stories from their projects’ target groups, as well as to collect stories from the Ambassadors themselves (see Figure bellow). To this end, the MSC approach was explained to the Ambassadors, and a story collection template was provided to them. In the end, only a few of the Ambassadors managed to collect one or more stories from their project’s target group. Based on some of the Ambassadors feedback, they felt that they did not have time to collect stories, or did not want to intrude further on their target group’s time. From the project staffs’ viewpoint, more support could have been provided to the Ambassadors to collect stories, such as through reminders and personal advice and follow up.


Original plan to collect MSC stories


Ambassador MSC story selection hierarchy

The Ambassadors’ stories were collected in a group format during an evening get-together, prior to celebrating the end of the Ambassador projects. Ambassadors were asked to sit in pairs, with one person collecting the story from the person opposite, and then vice versa. This method ensured that the collection of stories was actually undertaken. The project staff got the Ambassadors to collect stories amongst themselves, as opposed to the projects staff collecting stories, as it was felt that it was a more participatory method of collection, and it built further capacity within the Ambassadors. This collection of stories by the Ambassadors also enabled them to share amongst themselves some of the diverse impacts or lessons resulting from the project.


The story collection process consisted of the story collector asking the story teller questions based on a template that featured the following questions:

  1. Tell me why you (the storyteller) decided to become involved in the Sustainable Ambassador project:
  2. Looking back over your time since you became an Ambassador, describe a story that best describes the most significant change that has resulted from your involvement in this project:
  3. Why was this story significant for you?
  4. (optional) Have you participated in other sustainability initiatives/projects in the past, and how has this one differed from those?

The story collector recorded the answers in written form on the template. This was later typed up by project staff. None of the story tellers or collectors provided story titles, so these were later added by the project staff based on their impression of the dominant theme of the story. The collected stories can be found at the end of this case study.

Story Selection

The story selection process was undertaken by two project staff, one from the City of Whitehorse and one from the NCS. The two project staff members took turn reading stories out loud to each other. The stories were then classified as to their theme (domain), using a story selection template (see Table below). It was found that some stories had more than one theme. In this case, the dominant theme was used to classify the story. After reading the stories, they were classified in the following themes (domains).

  • Participation in sustainable practice
  • Realisation that the theme is a community of like-minded sustainability people
  • Being a credible change agent
  • Change from interest to action
  • Lessons

The project staff also noted down any points that arose from the story, such as particularly interesting details. Following a review of the notes and instructions on selecting domains in the MSC Quick Guide (Dart & Davies 2003), the two project staff decided to combine some of the themes together and reclassified the stories into four new main themes, using “change” in the title (as per the MSC Quick Guide):

  • Change in the ambassador
  • Change in the nature of people’s participation in sustainability activities
  • Change in the nature of people’s community connectedness
  • Lessons


Having reclassified the stories, the project staff reviewed the stories within each theme and discussed which was the most significant. The selection of the most significant story was a decision made by consensus. The staff ensured that the reason for selecting stories was clearly recorded (Table 1). The staff then reviewed the most significant stories in each domain and selected the overall most significant change story. Again, the reason for the selection of the story was recorded.


Categorising and selecting MSC stories for Sustainable Ambassador program

Theme / Domain

Story Teller/Title

Selector Notes

Why selected

1. Change in the ambassador

Peter

Articulates objective of the Ambassador program



Rohan

(could also have been in Theme 2, but Theme 1 was dominant)

Ambassador leading to further opportunities

A positive change in life, with the Ambassador being a springboard for undertaking the Green Loan Assessor course and increasing Rohan’s sustainability network


Lisa

The process of change


2. Change in the nature of people’s participation in sustainability activities

Mal

A first step towards sustainability



Louise

(could also have been in Theme 1, but Theme 2 was dominant)

Businesses became Ambassadors themselves

Reinforced existing sustainable behaviours and led to additional behaviours. The change is ongoing, with businesses committed to demonstrating their sustainability. Flow on effect to customers.


Kathy

Also inspiring son



Dean

Kids acceptance of change for sustainability


3. Change in the nature of people’s community connectedness

Niloo

Meeting new people



Ben

Getting to know people



Liz H

More neighbourly interactions on sustainability



Liz S

Trust between neighbours

Developing community connectedness beyond sustainability issues.

4. Lessons

Mary

How hard change can be, especially sustained change


Overall Most Significant Change Story

Louise

Louise’s story demonstrated that the Sustainable Ambassador program met its goal of building community capacity to effect change in the community. Not only did Louise gain skills and a sense of purpose, her project led to flow-on benefits with the businesses influencing their customers. The businesses also developed a greater sense of connectedness among the neighbouring businesses.



Discussion

The MSC approach proved to be a useful tool to obtain stories on how a project or intervention can affect a participant. In comparison to other evaluation methods such as quantitative surveys (for example using a Likert-type scale), the MSC approach provides a personal insight into the impact of a program. The MSC approach allows the participant to provide feedback into the type of impact (the most significant one) rather than project staff asking participants for feedback on pre-determined indicators.

The MSC approach requires the story collectors to be able to elicit the stories from the story teller. This can be done through a template which has prompt questions. The approach taken in this case study, where the participants took turns being the story teller and collector, was found to generally work well. It was also found to be expedient, as it guaranteed that the stories would be collected. However, it must be noted that the final story collection is based on the story collector taking down detailed notes, and as such, poor note taking could lead to a situation where the meaning of the story is not conveyed in an accurate form. There are other methods of collecting stories, which are outlined in Davies & Dart (2005). The plan for the Ambassadors to collect stories of their project’s target group proved to be ineffectual in this case. Some of the reasons for this include the time constraints both on the Ambassadors, as well as on their target groups.

Some of the stories were found to contain more than one significant change, or themes (domains). A reason for this was the general excitement from the story teller in being able to provide feedback on their experience. One way to avoid this occurring would be for the story collector to facilitate the storyteller to focus on the ‘most significant’ of the changes. This could be through questioning the storyteller further, such as asking “of these points, which do you feel is the most significant one”. As participants were collecting the stories from their peers, this would have required further training or explanation prior to them gathering stories. It may therefore be worth having several dedicated and well-trained story collectors (such as project staff) to collect stories.

The stories that were collected were very rich in demonstrating meaningful change and outcomes. One way to complement the words would be through asking the participants (in this case the Ambassadors) to take photos that are representative of the most significant change. This could include, for example, pictures of achievements, meetings, and the like. This would be beneficial to include in a reporting process to funding agencies. It would also assist as a means of verification of the change.

Lessons

The following lessons and recommendations are made to assist in the successful implementation of the MSC approach for evaluation of capacity building programs.

  • Explain the MSC approach to participants up front. This can be through collective reviewing the MSC Quick-Start Guide.
  • Provide advice and support to the story collectors on how to facilitate the storytellers to focus on the most significant change. This includes a story collection template.
  • Explain to the story collectors the need to take down detailed and accurate notes that convey the meaning and importance of the story. Alternatively, recording devices could be used (with permission).
  • Transcribing stories can take time. Ensure that there is resource for this, especially if there are many stories to collect.
  • Upon reading the stories the first time, jot down the dominant theme for each. Once all the stories have been read, the like-themes can be grouped, and the top 4 or 5 main themes (domains) can be chosen. The stories can then be re-grouped into these themes.



City of Whitehorse Sustainable Ambassadors Phase 2 MSC Stories

A fridge to friendship- Mal’s Story

Mal decided to become a Sustainable Ambassador after having seen the advertisement in the local newspaper. Since becoming aware of the global energy constraints and personally adopting a sustainable lifestyle, Mal became an Ambassador to participate more widely in the community.
For Mal, his most significant change resulting from the Ambassador program revolves around a case where a person was unaware of recycling and just placing items out for hard rubbish collection.  This person has a fridge to dispose of. Through the Freecycle site, someone was able to take the fridge, and the item moved half a kilometre. Not only was the product re-used, the small relocation distance meant that the move did not use much fuel (compared to carting hard rubbish to the tip), and the two new owners now used less energy in their homes. Just as important, the two people have now become friends, demonstrating that Freecycle is not just about recycling, but builds community connectedness. The person who placed the fridge on the website is now totally committed to using Freecylce for recycling other useful items.

Meeting new people- Niloo’s story

Niloo became an Ambassador after becoming aware of the program from attending the Living for our Future workshops. Niloo had become personally aware of global environmental issues and wanted to create change in the community.
For Niloo, the most significant change resulting from the Ambassador program is meeting new people in a new area, becoming friends, and becoming aware that other people were also aware of the same issues. This came about from undertaking a survey of 146 houses in 3 streets in the Blackburn Heritage streets area. Niloo also presented and organised a talk at the Blackburn Visitors Centre, and set up the ongoing communication for Blackburn Sustainability Street. Niloo was also happy to know that one person who lives outside the survey area also attended the presentation after having heard about it through word of mouth.

Support and credibility- Peter’s story

Peter became an Ambassador as he wanted to be able to make a difference after seeing that there was a need for sustainability assessments in the nursing home, where residents had high heating bills caused by draughts and the like. For Peter, the most significant change from the Ambassador program is receiving the backing of the nursing home management, as this gave him credibility in the eyes of the residents.

One thing leads to another- Rohan’s story

Rohan became involved in the Ambassador program following a family suggestion. Rohan also wanted to be involved and relate to people regarding sustainability. The Ambassador’s program allowed him to be part of the change, and become a credible expert.
The most significant change for Rohan was also becoming involved in the Green Loans program and undertaking the course. This provided Rohan with more contacts in the community. The Ambassadors program also gave Rohan an understanding of how people think on environmental issues, and an awareness of the many different (and often indifferent) positions people hold. All this was significant as it allowed Rohan to see further and help others to make changes. He is now able to take action rather than just talk about the issues.

A family affair- Kathy’s story

Kathy became an Ambassador for three reasons that converged. Firstly, Kathy thought it would be a good way to become connected with the community as she had moved in to the area 18 months ago. Kathy also had an ongoing interest in the environment. Lastly, the Ambassadors program provided an opportunity to do something meaningful with the family and was a way to encourage her son to focus his interest in the environment.
Kathy has two significant stories. The program provided the retirement village residents, where Kathy’s mum also lives, an opportunity to meet and discuss environmental issues in a very practical way. This encouraged residents to think about the bigger issues, and many of them are concerned about the world they are leaving their grand children. 100 of the 180 residents attended the “open house”, which displayed a number of sustainability features. The Ambassador program also led Kathy’s son Rohan to the Home Auditors course and he has now started his own business as a consequence. Kathy did not know if the project would work, but it had two significant impacts- on the retirement village and personally for Rohan. In addition, Kathy’s father died in June, so it gave her mother a real focus,  while also improved her unit through making it more sustainable.

The personal touch- Lisa’s story

Lisa became an Ambassador as she had a general interest in sustainability and was interested in what works in driving change in others.
For Lisa, the most significant change was finding out from a person that took part in her project that the personal touch was important in driving change. This person indicated that they appreciated being asked what they felt was working. This person felt empowered to be being approached personally. For Lisa, this is significant as it demonstrated that feedback seemed to inspire participants, and it was also reassuring that the personal approach was appropriate. Lisa, who had previously done the Train the Mentor program as part of Sustainability Street, liked the assessment and deadline provided as part of the Ambassadors. Lisa also felt personally responsible for the project’s success.

If we could all be like kids- Dean’s story

Dean decided to become an Ambassador following frustration with the lack of progress from a top-down approach in society, so he wanted to see what was happening at the grass roots level, and this also gave him the opportunity to learn what motivates people to change.
Dean’s most significant change was finding out how much kids already knew- even the grades 1 and 2. Dean found that kids were very motivated, and took things on board and actioned it. There was much less resistance to change than adults, which gave Dean a sense of hope! One example is how kids became aware of the need to turn lights off, and the parents’ feedback was that they turned into light-Nazis.  This was significant for Dean because he realised how surprisingly easy it is to effect change with kids- they exhibited less baggage and brick walls than adults. At the other end of the spectrum, just because someone is aware of what needs to be done and that there is a problem, it does not mean that they will put things into action to fix things. Dean liked how the Ambassadors identified barriers and provided an approach to overcome them.


Sharing recycling bins- Liz’s story

Liz H became an Ambassador as it was a wonderful opportunity to both learn new information about sustainability and contribute positively to the local community. The free home assessment was also attractive, and the topics looked fabulous and the time suited Liz taking part. The Ambassadors was a great way to meet like-minded people in the community, which provides additional inspiration and motivation.
For Liz, one thing that stands out is how neighbours are stopping her in the street to discuss environmental concerns. In particular, you can now see neighbours having friendly interactions as they negotiate sharing space in each others recycling bins rather than dumping recyclable items in the rubbish bin. This was completely unanticipated and provides evidence that the neighbours recognise the importance of reducing the amount of rubbish going to landfill.

 

A greater trust between neighbours- Liz’s story

Liz S became an Ambassador as it was a way to meet like minded people, form new networks, acquire new knowledge and get a home audit.
For Liz S the most significant change is the degree to which she now knows her neighbours- being more familiar and knowing their interests and homes. There have been dinner invitations, and being let into someone’s home is a sign of trust, respect and relationship. Neighbours now share letterboxes, pet care and the like. There wasn’t any of this before moving in. This is significant because it demonstrates the development of trust between previous strangers, and that this can happen if there is a reason to make contact.

Forming community- Ben’s story

Ben, having just moved in to a new area, became an Ambassador because he wanted to explore how much the community was aware of sustainability issues. Ben was keen to turn awareness into action. The program provided a way to get to know the neighbours and meet up with other sustainability groups in the area.
The most significant change for Ben was getting to meet local people and find out that they were also interested in sustainability. Other local groups, such as the local neighbourhood association, joined in the presentations. This was significant because it broke down barriers of communication and led to greater interaction and promoting of sustainability issues. Meeting neighbours made him feel like a part of the community.

Ongoing changes- Louise’s story

Louise became an Ambassador as she had an interest in sustainability and it was a way to do something and gain new skills and improve her chance of returning to the workforce in 2010 after having children.
The most significant change for Louise is seeing how the businesses have kept up or taken on more sustainable practices. Louise’s project led to an increased awareness of sustainability issues amongst the shop owners, and the businesses were really positive about the changes they have made. They still have signs up, and they keep on doing more things to improve their sustainability practices. One example of this is the post office worker, who says loudly to other customers can hear “I won’t give you a bag because we are all trying to do our bit for the environment”. The post office is now sourcing environmentally friendly products to sell and to use in the business. The changes the businesses have made are significant to Louise because it demonstrates her ability to effect behaviour change as a result of the project, and it made her feel like a productive member of society. This is especially important for Louise as she found that when you have children and stay at home, you don’t get recognition as being a productive member of society.

Change can be hard- Mary’s story

Mary became an Ambassador after seeing the program advertised in the local paper and in the Council newsletter. Mary had a strong interest in activism and sustainability.
For Mary, the most significant change from undertaking the Ambassadors program is her personal awareness that it is harder than it looks to get others to make changes. Mary found that there were many people who did not think like her or were as keen. Having always been surrounded by sustainable people, she did not realise that there were so many people that were not sustainable. However, she Mary did manage to change the community house attitudes and even get another unconnected group to change as well. Mary thanks that it may be hard to sustain the interest though, and they may now build on the composting project by building a herb garden or something similar. For Mary, her project reaffirmed her view that doing something small and local is effective. For the community house people, it was awareness raising, and it was great to see the involvement and enthusiasm of the ESL teacher who did a whole term on sustainability.

 
Donate Now

Please make a donation to upgrade the Evaluation Toolbox.

The Evaluation Toolbox is maintained by Damien Sweeney and Martin Pritchard from PREA as their in-kind contribution back to the community. The Toolbox now needs several weeks of work to review and upgrade all the contents and add new content for you. Work has begun and we are seeking your donation (big or small) to help support this major upgrade. Email us to indicate what you want to be included in the Toolbox.

case study

What is the Toolbox?

The toolbox aims to provide a one-stop-site for the evaluation of community sustainability engagement projects that aim to change household behaviours. Through the toolbox you can learn how to conduct your own evaluation of a behaviour change project using the guides and templates provided.

see more...

Why Evaluate?

Conducting an evaluation is considered good practice in managing a project.

The monitoring phase of project evaluation allows us to track progress and identify issues early during implementation, thus providing and opportunity to take corrective action or make proactive improvements as required.

see more...

 

DSE
City of Whitehorse City of Whitehorse
Accord