"Statistically Significant" Gold dust for funders ears...and your project
If you are not taking impact capture seriously for the projects you are running then I believe you to be looking down the barrel of a funding gun. It amazes me how many organisations don't collect the simplest of data like attendance or any form of 'story' from their participants. Money is scarce and getting scarcer. Commissioner's and grant givers require evidence that what we do is effective.
Isn't it a pain the backside though?
We've just started analysing a year of data, following a development to collecting 'well-being' data as our measurement tool. Following the old adage of not reinventing the wheel we adopted a well-being scale that has had extensive testing over 7 years of peer-reviewed research, is used by the NHS and trusted by commissioners. It's called the Warrick and Edinburgh well-being scale (google is your friend). Users fill it out pre and post project giving us 'journey travelled' data.
Below are just the initial commissioner friendly highlights of our first years data.
We saw:
- Doubled levels of self-reported High Well-being
- Halved levels of self reported Low well-being
- Consistent meaningful increases of self-reported Well-being across all age ranges and genders (using a nationally validated and trusted tool)
- Most importantly, using tools others had developed, we can prove that we are statistically significant - that means against a population who've created a 'norm' measured using this tool, our results (in comparison) can be seen to be replicable anywhere in the UK. Our results can be measured against a standardised score and shown to be highly effective.
We've also just had an independent report, funded by the cabinet office, done that verifies the results - this is also 'music to those commissioners ears". A few of things you can do.
- If you're not collecting impact data then just start
- Use a tool that the people who pay you (your customers, not your end users) trust
- Consider getting an independent authority to conduct research/get involved with a University - validate your work
Presenting this kind of data to heads of services, commissioners, press etc is a lot easier than saying "Well I think it works.....?"
[1] WEMWBS is a 14 item scale with 5 response categories, summed to provide a single score ranging from 14-70. WEMWBS was developed by researchers at the Universities of Warwick and Edinburgh, with funding provided by NHS Health Scotland, to enable the measurement of mental well-being, published and validated in 2006: http://www2.warwick.ac.uk/fac/med/research/platform/wemwbs/