Chapter 7
Accurately measuring results
7.1
The efficacy and success of a social economy organisation depends on
social measures to facilitate analysis and performance comparison, establish
track records and offer potential investors an assessment of the risks and
returns related to different investments.[1]
In addition, measurement can improve an organisation's performance, provide
insight into the operation of the sector as a whole and improve public policy
outcomes.[2]
7.2
The terms of reference for this inquiry do not explicitly mention
measuring social value as an imperative to attracting investment to the social
economy sector. The committee has not received much evidence on the issue of
measurement. However, the committee recognises that the ability to measure
social value is critical to social impact investment (see chapter 6).[3]
Social impact bonds (SIBs) in particular, can only be applied to policy areas
that have measureable results[4]
as they require a baseline outcome and set targets to monitor the achievement
of outcomes.[5]
The benefit of a greater emphasis on the measurement of social outcomes (rather
than simply inputs and outputs) provides incentive for government and investors
to allocate resources to achieve the greatest social impact.[6]
7.3
The Productivity Commission (PC) report, Contribution of the
Not-for-Profit Sector, provided substantive recommendations for Australian
governments to adopt a common measurement framework for the sector which takes
into account the diversity of the sector's activities and structures. The PC
outlined that a framework applied consistently across governments will minimise
compliance costs and maximise the value of data collected.[7]
The report stated that building knowledge about the sector and its impacts on
society 'is an important element in building confidence in the sector, as well
as guiding policy and program design. Evaluation of the sector’s effectiveness
is essential if the sector and government are to embrace an evidence-based
approach to social investment.'[8]
The PC's recommendations are discussed further below.
7.4
Mr Les Hems and Associate Professor Cheryl Kernot of the Centre for
Social Impact (CSI) provided a broad description of the global trends in the evaluation
of social outcomes and impact, and the growing recognition of the need to move
beyond economic indicators to more holistic measures that include social and environmental
progress. These trends are interrelated with a shift towards socially
responsible investment (SRI):[9]
Mr Hems: ...there is an international initiative at the
moment around integrated reporting. It is very much building on the blended
value and triple bottom line... In the documents that have been issued to date
there is recognition that the straightforward financial accounting is
inadequate. Indeed investors are requesting information on social impact and
environmental impact. So the argument behind the need for integrated reporting
is that investors are requesting it. Similarly, other stakeholders are wanting—
CHAIR: Are they requesting it across the board for all
investments or specifically for investments in this area?
Mr Hems: All investments. These are mainstream
institutional investors requesting effectively triple-bottom-line reporting and
our argument is that the social impact component is going to be the most
difficult to standardise and report in a meaningful way. Therefore, the not-for-profit
sector and social enterprises should actually provide the set of case studies
to allow integrated reporting to develop.
...
Prof. Kernot: In the corporate sector there has been a bit of
a hijacking of the whole notion of reporting sustainability simply along
environmental lines. Most people now recognise the challenge of incorporating
the social in that triple-bottom-line, blended value approach. There is a huge
shift going on in the corporate sector in its understanding in that particular
sphere.[10]
7.5
The committee has heard that there is a growing body of international
social measurement techniques to support these developments[11]
and an increasing willingness and recognition among Australian social economy
organisations of the importance to report on social impact.[12]
Despite these developments, measuring social value presents a number of
challenges for social economy organisations, which in turn places limitations
on the amount of investment in the sector. Some barriers to effective
measurement of the activities of social economy organisations include:
-
social and environmental measurements and metrics have not yet
been fully developed;[13]
-
metrics for social outcomes and impacts cannot always be assigned
a numerical value;[14]
-
focusing on milestones or achievements, rather than on the
quality of the service provided when measuring success may provide a skewed
result;[15]
-
a highly diverse social sector creates tensions between the need
for a standardised reporting system across organisations, and the need for organisations
to report on the most material aspects of their individual work;[16]
-
the financial costs of, and the requirement of considerable
expertise for reporting and evaluation can place undue burdens on social
economy organisations;[17]
and
-
tools used to determine social outcomes and assess participants
may be considered invasive, and can be dependent on the cooperation of the
service users.[18]
International developments
7.6
The Young Foundation recently analysed over 150 measurement tools that have
developed internationally and concluded that while these are encouraging developments,
'there are not yet any widely-agreed metrics on gauging social returns or the
impact of social ventures'. The same observation was made by the UK Social
Investment Task Force which noted that measuring social impact was a
significant challenge for the market.[19]
7.7
The departments' submission[20]
and the Centre for Social Impact (CSI)[21]
noted the leading work of a number of international organisations:
-
The Impact Investment Reporting and Investment Standards (IRIS)[22]
are an initiative of the Global Impact Investing Network (GIIN) which is
developing an independent and credible set of metrics for measuring social and
environmental performance. As part of IRIS, a beta database was recently made available
on the web working towards mechanisms for a rateable system in the US.[23]
-
The Global Impact Investing Rating System (GIIRS),[24]
which is overseen by B-Lab, is exploring a ratings approach to assessing social
and environmental impact and was launched at the Clinton Global Initiative in
September 2011, New York.[25]
-
The International Integrated Reporting Committee's (IIRC)
Integrated Reporting Framework is aiming to integrate multiple sources of value
(social, environmental and financial) in a single reporting approach.[26]
Integrated reporting framework
7.8
Integrated reporting is intended to reduce the reporting burden of
organisations while improving investors' insight and understanding. The
provision of an integrated information set, for example, will help inform those
who invest on behalf of others to discharge their fiduciary duty (see chapter
4) by taking into account the range of issues that affect organisational, and
therefore financial, success and investment returns.[27]
The IIRC has released a discussion paper with guiding principles that offers
initial proposals for an International Integrated Reporting Framework prior to
the publication of an exposure draft for integrated reporting in 2012.[28]
7.9
South Africa has become the first jurisdiction to legislate and mandate
that organisations[29]
prepare reports on an integrated basis. Organisations are required to provide
an explanation if they have not reported on an integrated basis.[30]
The Integrated Reporting Committee of South Africa has released a discussion
paper outlining a non-prescriptive, principles-based approach which offers
practical direction on the integrated report.[31]
Social return on investment
framework
7.10
A Social Return on Investment (SROI) framework has been developed by the
Measuring Social Value consortium in the UK and is promoted through the SROI
Network International.[32]
The framework was devised to understand, measure and manage the outcomes of an
organisation's activities. A SROI is similar to a cost‑benefit analysis.
It places a monetary value on outcomes so that they can be added up and
compared with the investment made. This results in a ratio of total benefits (a
sum of all the outcomes) to total investments. For example, an organisation
might have a ratio of £4 of social value created for every £1 spent on its
activities.
7.11
The Measuring Social Value consortium noted that there are two types of
SROI:
-
evaluative SROIs are conducted retrospectively and are based on
outcomes that have already taken place; and
-
forecast SROIs predict how much social value will be created if
the activities meet their intended outcomes. Forecast SROIs are useful at the
planning stage of a project, or if the right kinds of outcomes data have not
been collected to enable an evaluative SROI.[33]
In addition, the Gates Foundation has published an overview
of some emerging measurement tools.[34]
Social Traders also lists a number of resources on measuring social impact on
its website,[35]
and CSI has published a paper, Recent approaches to measuring social impact
in the Third sector: An overview.[36]
7.12
Other measurement methods are discussed in detail in the PC report and include
cost-benefit analysis, social accounting and audit, stakeholder value
management analysis, results-based accountability and logical framework.
Appendix B of the report, 'Techniques of social evaluation' provides
further detail.[37]
Australian developments
7.13
Within Australia, there is some tension between the need for a
standardised reporting system across organisations,[38]
and the need for organisations to report on the most material aspects of their
individual work.[39]
Progress has been made towards standardised reporting of inputs through the
establishment of the Standard Chart of Accounts (SCOA) (see chapter 3).[40]
However, there is currently no uniform approach to measuring the social
outcomes and impacts of social economy organisations operating in Australia.
The Finding Australia's Social Enterprise Sector survey, for example, reported
that only 65 per cent of organisations even sought to evaluate or measure
impacts in relation to their missions.[41]
7.14
CSI is undertaking work on a new reporting system being developed as
part of its Integrated Reporting Project.[42]
It aims to assist stakeholders to better understand how an organisation creates
value, including social, financial, environmental and human capital in the
short, medium and longer term.[43]
The CSI website states that its Integrated Reporting Project 'presents a unique
opportunity for innovation in impact measurement and reporting of shared value
in the Australian not-for-profit sector'.[44]
7.15
The project is a collaboration between CSI, the Australian School of
Business and private, not-for-profit (NFP) and government sector partners. The
project is intended to develop a framework to implement Integrated Reporting in
the NFP sector and will be tested across a variety of sites in collaboration
with a network of partners. The project leverages international momentum[45]
and is hoped to 'place Australia as a global thought leader in the future of
NFP reporting':
Collaboration with an array of stakeholders will be
undertaken to ensure that Integrated Reporting meets a range of expectations
including government as funders and regulators (DEEWR, FACHSIA, PM&C),
donors, and may help lay the foundations for a capital market for social
investment in Australia. The test sites will comprise a range of not-for-profit
organisations operating in a small number of policy fields.
The data collected and experiences of this process will form
an evidence base for evaluating the key success factors and recommendations for
the future of Integrated Reporting in the not-for-profit sector in Australia.[46]
7.16
CSI is working in partnership with the Salvation Army to evaluate its
Employment Plus program. This will involve capturing the impact of the program
over a five year period. It is intended that the project will contribute to the
evidence base to illustrate that 'organisations across the board have got to
take evidence and systematically capture the impact of their programs into
their DNA'.[47]
Mr Les Hems of CSI told the committee that while impact based reporting will
require capacity building (see chapters 3 and 5), the majority of organisations
are willing to adopt the new approach:
This will require a lot of capacity building, but I would say
almost every not-for-profit organisation and social enterprise that we have
spoken to has recognised the need for doing this and are willing to engage in
programs to help build up these case studies to build up a methodology to
systematically report social impact, building on the pathfinding work on social
return on investment and social accounting.[48]
7.17
Mission Australia, for example, has a sophisticated case management
system which enables it to capture data and integrate it across the entire
organisation. This system involves maintaining contact with clients post program
treatment and assessing whether they are still in a job six months, 12 months,
18 months and two years later.[49]
7.18
In addition, as part of the capital injected into the sector through the
government's Social Enterprise Development and Investment Fund (SEDIF), the
University of Sydney Business School's Innovation and Entrepreneurship Research
Group (IERG) is working with Social Enterprise Finance Australia (SEFA) to
develop key performance indicators and metrics to measure social impacts.[50]
Foresters Community Finance, also a recipient of the SEDIF funding, uses IRIS
for evaluation.[51]
Social Ventures Australia (SVA) offers consultation using SROI and tailors the
approach to the specific needs of the client. SVA can conduct the analysis, or
assist an organisation to develop the skills to carry out its own analysis.[52]
7.19
Dr Richard Seymour of the IERG noted that '[m]ultiple approaches [for
measurement] are required at this stage of research, as no one group or
approach has solved the problem of impact measurement'.[53]
...with regards impacts and indirect value creation, we are
seeking to prioritise the subjective: We have questioned whether it can be
appropriate or meaningful to objectify the impacts/outcomes such as ‘enhanced
sense of self’ or ‘changed attitudes or values.’ Our framework considers that
these would be better celebrated ‘subjectively’ and from the appropriate
perspective (such as a beneficiary or participant). Furthermore, we think this
is more aligned with the way employees, participants, beneficiaries, donors and
other stakeholders engage and understand their purposeful activity and impact.
...
Measurement frameworks often ignore the perspectives of
clients, beneficiaries or participants, and end up presenting condescending or
objectifying data. Measurement frameworks often promise supporters or funders
transparency and rich information but end up being lists of comparative lists
and rankings. We will endeavour to keep the measurements aligned with the
interests (quantitative and objective as appropriate, qualitative and
subjective as appropriate). The tensions between such data should be celebrated
rather than ignored.[54]
Financial costs of evaluation
7.20
The PC and Christian Super noted that the financial costs of reporting
and evaluation may place undue burdens on social economy organisations and
require considerable expertise.[55]
Associate Professor Kernot of the CSI emphasised that the government, and other
partners, may need to assist in developing reporting resources for the sector:
The problem is that many non-profits do not have the
resources to invest in that even though it is a fundamental requirement of
being capacity ready for greater social investment. There is an enabling role
for government and other partners there.[56]
7.21
The departments' submission noted that international developments and
collaborations will reduce the transaction costs of building market
infrastructure:
Leaders in the social impact investment market are
collaborating to reduce high transaction costs by building market
infrastructure such as common social return metrics and framework...These new
measurement tools are guiding financiers, including governments and
philanthropists seeking to determine the most effective way of acting and bring
a focus and discipline to the task of choosing interventions that are shown to
be the most effective relative to the resources invested.[57]
7.22
The PC report proposed a commonly agreed measurement framework with the
intention that any type of organisation in the sector should be able to adopt
it, including those that lack the resources to undertake comprehensive
evaluation.[58]
Adopting a measurement framework
7.23
An evaluation framework is a compulsory requirement for a capital market
for social economy organisations.[59]
The PC explored the importance of a measurement framework for the social
economy, drawing on international research and current techniques such as SROI
and results based accounting. The PC report recommended that all nine
Australian governments adopt a framework based on four distinct levels:
-
inputs—measures of the resources used;
-
outputs—indicators of the level of activity undertaken;
-
outcomes—direct costs and benefits to the activity of
participants; and
-
impacts—longer term net benefits to the participants, and other
costs and benefits to the broader community.[60]
7.24
Inputs and outputs provide an insight into the processes and activities
of organisations and, when aggregated, the scale of the sector. The Australian Bureau
of Statistics' (ABS) Non-profit Institutions Satellite Account provides input
and output data on economically significant NFPs in the sector. Outcomes
measures provide information on the direct benefit of an organisation's
activities. At the far end of the spectrum, impacts aim to reflect the net
benefit for the broader community, and its longer term effects. In an analogy
of medical services, for example, the impacts would include the general
productivity and social benefits of a healthy population.[61]
The report emphasised that the measurement of outcomes and impacts will inform
the governments' resource allocation for the sector, and without these
measurements, the understanding of the contribution of the sector will be
understated.[62]
The framework recommended by the PC is shown in Diagram 7.1.
7.25
Mr Robert Fitzgerald, the commissioner that led the PC inquiry into the
contribution of the sector, elaborated on the four levels of the proposed
measurement framework in evidence to the committee:
Mr Fitzgerald: ...As you move down, going from inputs,
outputs, outcomes and down to impacts, the measurement techniques become less
and less robust. As a consequence of that, when you are looking at impacts you
need to use a range of measures. You need to use both quantitative and
qualitative approaches, which draws off good evaluation, good case studies and
other evidence, but absolutely measuring impacts, both at an organisational and
sectoral level is where we want to get to. There is work being done on that
internationally and I think we have made a significant contribution to that.
CHAIR: In some areas the ability to measure impacts would be
more mature than in other areas, depending on the social benefit you are trying
to deliver.
Mr Fitzgerald: Absolutely. Since this report has come out—and
I am still involved in the sector in a number of ways—one of the good things is
that there has been a real desire by the sector, including the philanthropic
part of the sector, to move to looking at impacts and trying to design ways by
which that can be measured in the broad sense. I think that is an area where
there has been great progress. The point I make is that we should not fall into
the trap of trying to turn everything into a number. The point that we have made
in the report is that where it is measurable you can measure it, but where it
is not measurable you can still in fact try to ascertain its value through
other processes and other techniques. I think that in the social policy area
that is an appropriate way to go.[63]
Diagram 7.1: Measurement framework and types of indicators
![Diagram 7.1: Measurement framework and types of indicators](c07_1.jpg)
Productivity Commission, Contribution
of the Not-for-Profit Sector, January 2010, p. 40.
7.26
Mr Fitzgerald also highlighted the need to establish an evaluation
framework during the design of a program, and not simply at a point of failure
as the evaluation component 'will inform the design features' of the program.
The Logical Framework approach, for example, is used to establish outputs,
outcomes and impacts of the activity early in the design of a program.[64]
Mr Fitzgerald also emphasised the need to use evaluation as part of a quality
improvement framework, rather than simply a mechanism to determine success or
failure. He elaborated on this point, using indigenous social policy as an
example:
One of the issues with social policy is that often you have
to adapt as you learn. Not only do you need early intervention by the
government in terms of bringing the right parties together, you need
good-quality evaluation frameworks designed at the beginning. You need them to
be used to inform and then you need an ability to adapt as you go forward.
If we look at the Indigenous social policy area—and you are aware
that the commission convenes the working group that produced the Overcoming
Indigenous disadvantage report—we see over and over again a lack of
good-quality evaluation, and evaluations used once the program has already
started to fail, and no capacity to use them for quality improvement. That is a
micro example of a much greater issue. Again, our commentary in the report
about evaluation is very important. It is not just about program evaluation; it
can be applied to a much broader concept than just very small programs. It can
be looking across a range of programs or initiatives.[65]
7.27
A March 2010 paper by Social Finance underlined the distinction between
a program's outputs and outcomes, noting that a focus on milestones (outputs) rather
than the quality of the service provided (outcomes) may provide a skewed
result. The report explained:
A youth training scheme, for example, might be evaluated on
the basis of how many young people attend the training course (an input) or
receive a Health and Safety certificate (an output). The hope (the desired
outcome) is that better qualified young people will be more likely to find
jobs. Too often such outcomes are not used to measure the success of such
schemes or to judge whether or not to renew service provider contracts.
This trend is driven by the fact that contracting for inputs
and outputs is simpler than contracting for outcomes. It is easier for a
commissioner to measure and attribute outputs to a service provider – they can
count the number of Health and Safety certificates awarded. It is more
difficult to attribute an outcome, such as employment, to the work of one
agency as inputs from multiple organisations may be required to deliver a
positive outcome.[66]
7.28
The Cape York Institute also noted that SIBs could act as a catalyst for
the sector to move from outputs based measurement to outcomes, and that this
would have particular benefits for indigenous programs.[67]
7.29
The PC measurement framework proposes a full spectrum of techniques to
accommodate different measurement needs and purposes for organisations. The PC
report concluded that there is no standard measurement 'which will be best
suited to all NFPs under every circumstance' and acknowledges that certain
activities pose specific measurement challenges. It cites the spiritual value
of retaining connection to land for Indigenous Australians as an example.[68]
Government response
7.30
The Office for the Not-for-Profit Sector in the Department of the Prime
Minister and Cabinet is acutely aware of the need for an evaluation framework
for social economy organisations. The Office found that SROI was one of three
critical areas that need to be examined to create a robust capital market for
the social economy and to establish 'transparency and accountability of social
initiatives for building credibility with investors, funders, donors and the
like':[69]
Certainly at the Office of the Not-for-Profit Sector we are
acutely aware of how critical it is that we have the right sort of data and
that we have evaluation frameworks. Whether the best outcome is that we have
one evaluation framework that we are all able to use I am not at all sure. It
may be that we have a range of evaluation frameworks that are appropriate for
different policy settings, but the underlying need to improve our data to have
a much greater emphasis—and this is from a government perspective as well as
from a not-for-profit sector perspective and a for-profit sector perspective—on
evaluation and a strong measurement of outcomes is undoubtedly a critical issue
in all of this.[70]
7.31
The Office told the committee it is in the initial stages of responding
to the PC report's recommendations on evaluation, and has begun work with the
Australian Bureau of Statistics around framing the PC's recommendations.[71]
7.32
In addition, CSI discussed the need for government to move towards
outcomes-based assessment systems and to develop this approach through a series
of demonstration projects. CSI proposed that these projects could focus on policy
areas where results based funding is already utilised (such as pathways to
employment) or 'other policy areas where robust outcome measures can be
formulated and where there is a robust evidence base'.[72]
Conclusion
7.33
An effective measurement framework is fundamental to the development of
a robust capital market for social economy organisations in Australia. The PC
has provided the groundwork for a measurement framework to be applied to the
sector by government, social economy organisations and investors.[73]
CSI has also made significant contributions towards a measurement framework through
its Integrated Reporting Project.
7.34
The Council of Australian Governments' adoption of the SCOA has provided
initial steps towards standardisation of input measurements. The committee
acknowledges, however, that standardisation of outcomes and impacts has added
challenges, such as measuring unquantifiable contributions and producing
comparable results across a sector undertaking a diverse set of activities. As
a result, many social organisations will not have the expertise or funding to
adopt emerging measurement techniques.
7.35
In the committee's view, the government has an enabling role to develop
reporting resources to guide the sector and promote measurement techniques that
have proven to be the most effective relative to the resources available to
social economy organisations. This will improve efficiencies and consistency
and facilitate better analysis and performance comparison across the sector. It
will also reduce the transaction costs for social economy organisations while
the market infrastructure is being established.
7.36
The committee believes the PC's measurement framework captures a broad
range of social economy activity, and is flexible enough to allow users to
choose measurement techniques most suited to their circumstances. The committee
notes that the Office for the Not-for-Profit sector has begun work with the Australian
Bureau of Statistics to respond to the PC's recommendations on evaluation. The
committee recommends that in addition to this work, the Office for the
Not-for-Profit Sector develop a guide on the PC's proposed evaluation framework
for social economy organisations and government departments and agencies. The
guide should take into consideration emerging international measurement tools.
7.37
The committee notes the CSI's proposition that government should move towards
outcomes-based assessment systems and that this should be developed through a
series of demonstration projects. The committee recommends that the government
identify existing policy areas where results based funding is already utilised
and document an evidence base of these programs. The evidence base should contribute
towards the development of a robust measurement framework and the proposed
guidance material on measurement for the sector and departments and agencies.
Recommendation 7.1
7.38
The committee recommends that the Department of the Prime Minister and
Cabinet identify policy areas where results based funding is already utilised
and use any relevant programs as an evidence base towards the development of a
robust measurement framework for social economy organisations in Australia.
Recommendation 7.2
7.39
The committee recommends that the Office for the Not-for-Profit Sector
prepare a guide for social economy organisations to assist in the evaluation of
their performance. The guide should be based on the evaluation framework
recommended by the Productivity Commission using inputs, outputs, outcomes and
impacts and include Australian case studies and emerging international
measurement tools.
7.40
The guide should provide social economy organisations with a number of
measurement techniques as options to measure their outcomes and impacts. The
committee recommends that the guide be adopted by the Council of Australian
Governments and distributed to all government departments and agencies.
Navigation: Previous Page | Contents | Next Page