Storytelling provides a powerful means to obtain information on a project’s outcomes from participants’ experiences and viewpoints. Storytelling provides meaningful information that can highlight both the strong points and weaknesses of a project, as well as any unintended consequences. In a way, by asking participants to provide a story on a project, it asks them to evaluate an aspect of a project, rather than provide information for someone else to place a value on, storytelling generally brings out memorable or momentous experiences.
Storytelling has a number of benefits including:
- Understanding the project from the viewpoint of the participants
- Reflects the importance of context and its impact on outcomes
- Allows the identification of unintended consequences
- Provides a means to engage participants in evaluation.
Excerpts from stories can be used alongside graphs or in reports to provide a first person voice as to a project’s impact.
The Most Significant Change is methodology to collect, analyse and evaluate stories that is increasingly being used in evaluating projects both in Australia and overseas.
Stories can be collected through blog postings on a project’s website. For example, the Zero Carbon Moreland project have a webpage of participants’ stories.
Appreciative Inquiry is a form of research that looks at investigating a project’s positive outcomes in order to understand and replicate these rather than looking to understand why a project has not worked.
THE MOST SIGNICANT CHANGE
The Most Significant Change (MSC) is a method of participatory evaluation that involves the collection of significant change stories at different levels of the intervention (for example project staff, change agents, intervention participants) and collectively deciding on the most significant change stories based on selected themes (called domains). The domains reflect broad categories, such as change in capacity to take action, or a change in participation in an activity, as well as lessons learned.
MSC has its roots in the overseas development field but is becoming more widely used in community development and behaviour change interventions in Australia.
Davies and Dart (2005) list the following benefits from using the MSC method:
- It is a good means of identifying unexpected changes.
- It is a good way to clearly identify the values that prevail in an organisation and to have a practical discussion about which of those values are the most important.
- It is a participatory form of monitoring that requires no special professional skills. Compared to other monitoring approaches, it is easy to communicate across cultures. There is no need to explain what an indicator is. Everyone can tell stories about events they think were important.
- It encourages analysis as well as data collection because people have to explain why they believe one change is more important than another.
- It can build staff capacity in analysing data and conceptualising impact.
- It can deliver a rich picture of what is happening, rather than an overly simplified picture where organisational, social and economic developments are reduced to a single number.
- It can be used to monitor and evaluate bottom-up initiatives that do not have predefined outcomes against which to evaluate.
Pros and cons of MSC
|Participatory process of evaluation involving all stakeholders||Requires all stakeholders to accept method|
|Focus on learning and process of change||May require training in MSC|
|Can capture unintended consequence or emergence||Time consuming|
- A person to lead and coordinate the MSC process
- People to collect stories. This can involve having one or two people collecting stories, or getting participants to record stories from one another. Alternatively, you could ask people to write their own stories.
- Ways to record stories of change - digital voice recorder, or film (remember that you must seek all participant’s approval prior to recording) or notepaper/template. You can also complement this by photography or images of change, taken by the storytellers.
- A document tracking sheet to record who has provided a story, and what stories are classified as most significant.
The following provide a guide on using the MSC technique:
|MSC Quick Start Guide||The Most Significant Change Technique User Guide||Australian Government Land and Coasts Developing a Performance Story Report
The main steps to design an MSC process
|1. Starting||This is where you should introduce the MSC process to all the stakeholders and clarify its purpose. It is important to also consider who are the best people involved in the project to collect stories from.|
|2. Establishing 'domains of change'||
Domains are broad and often fuzzy categories of possible changes. For example, participants in the MSC could be asked to look for significant changes in four domains:
Changes in the quality of people's lives
Changes in the nature of people's participation in sustainability activities
Changes in the sustainability of people's lives, organisations and activities
Any other changes
|3. Collecting Significant Change (SC) stories||
The central part of MSC is an open question to participants, such as:‘Looking back over the last month, what do you think was the most significant change in the participants practice of sustainability in their home life?’
The question has six parts:
‘Looking back over the last month…’ – It refers to a specific time period.
‘…what do you think was...’ – It asks respondents to exercise their own judgment.
‘…the most significant…’ – It asks respondents to be selective, not to try to comment on everything, but to focus in and report on one thing.
‘…change…’ – It asks respondents to be more selective, to report a change rather than static aspects of the situation or something that was present in the previous reporting period.
‘…in the participants practice of sustainability …’ – It asks respondents to be even more selective, not to report just any change but a change in the participants understanding of sustainability. This tag describes a domain of change and can be modified to fit other domains of change (for example, participants understanding of sustainability in the community).
‘…in their home life?’ – Like the first part of the sentence, this establishes some boundaries. In this particular case we are not asking about people’s actions in the community or what others are doing in Australia, but focussing about people’s home life.
|4. Determining who will review the SC stories||Selecting stories usually involves a hierarchical process, where the lower levels select significant stories for the upper levels to review, and select the most significant ones from the lower levels.|
|5. Determining a process for reviewing the SC stories||A template to assign domains and select the most significant stories is available here.|
|6. Feedback||It is really important to feed back your selected SC stories and the reasons for the choice to the relevant people.|
Examples of the use of MSC
The Castlemaine 500 project used the MSC method to complement quantitative data collected through billing data. The MSC method showed a community-wide excitement around the topic of energy efficiency, and that a capacity to change had been developed in the community, even though the quantitative savings were not great. This underscores the reason why it is important to balance quantitative data with qualitative information, especially for projects that involve people changing behaviours. Behaviour changes, unlike the models of behaviour suggest, is not a simple process, and occurs through time, and amongst other supporting and conflicting influences and factors. In the case of the Castlemaine 500 project, if evaluation had solely rested on quantitative methods, the project could be construed as not having met its goals, whereas the use of qualitative data shows otherwise.
Case study on MSC
Further Links & Resources
Rick Davies MSC site
The most significant lessons about the Most Significant Change technique
MSC resources (including links to powerpoints) from the Department of Education and Children’s Services (SA)