Evaluation guidance - writing your evaluation
Step 4: Writing your evaluation
Taking an outcomes approach: from planning to evaluation
- When writing up your evaluation, you should structure your report in a way that corresponds with the outcomes and indicators that you have set. Presenting your outcomes under separate headings as per the online form will not only help the readers of your report to digest all the information, but will also help to keep you focused on what you are writing. There is likely to be some overlap between outcomes, but you can explain this in your document (for example you might say “see more on this topic under outcome 4”.)
- Tell us how many responses you collected and when you took them. We know it won’t always be possible to collect responses from every young person at every point of the way, so if you tell us this and let us know how many of the young people you are including in your analysis it will be much clearer (and more accurate). If you are collecting data at various points of your project, you may not get a full set of responses from one or two young people – maybe if they start the project late, or don’t finish it. In this case, it would be wise to consider how these responses might affect the assumptions you make about your data – that isn’t to say you should exclude them altogether, but you might think about whether it’s accurate to include them in statistical analysis, for example.
- Make sure all your data has been anonymised. If data includes real names or contact information of anyone involved in the project, it could violate the data protection responsibilities you have for your participants, their parents, and the workforce (for more information, see the Ethics and Data Protection paragraphs in Step 2: monitoring your outcomes).
- Work chronologically: if you have collected data at several points of the project, tell us the situation at the beginning of the project (baseline data) and use the narrative to compare it to later responses collected, so that it is clear to the reader what change has occurred. We appreciate there are some circumstances where you won’t able to collect extensive data at every point of the project, but the purpose of your evaluation report is to determine whether a change or improvement has been made. As a minimum, if you are able to collect data at the beginning and end points of your project, you should be able to carry out basic analysis to decide whether or not this change or improvement has occurred.
- Describe the emergent trends and themes in your data. With qualitative data, if, for instance, you’ve carried out interviews with participants, parents and teachers, and they’re all talking about the same thing, let us know. In the same way, if you notice in your quantitative data that the average scores from Likert scales are increasing or decreasing, tell us why it’s interesting and what you think it means.
- Use selected quotes in your evaluation to demonstrate overall progress toward your intended outcome. You may wish to use quotes showing overall trends or identify particular case studies. You may also wish to include quotes from the outliers and examine why they differ from the rest.
- If you submit photos or videos as evidence, make sure to describe what is happening and explain how they demonstrate progress toward your intended outcome.
- Tell us about any surprises in the data. We understand that trends may not always show improvement – if you spot something like this, explain why you think it happened and how you intend to respond to it. Similarly, if something has improved at a greater scale or faster pace than you anticipated, consider why that might be. This can sometimes be the most valuable information to share with Youth Music, and with other projects, so people can learn from mistakes or approaches that haven’t had the desired effect.
- Tell us about the limitations of the data. There will sometimes be issues with collecting full data sets so it’s good practice to consider that when you write the full report. Write about the external factors that might have contributed to the outcomes of your project – for example, if any of your participants are also having external music lessons, these might have an impact on their improving skills alongside the work you’re doing with them.
What not to do:
- Don’t just tell us what data you’ve collected. Summarise the findings of the data (providing examples where relevant) and explain how you interpret it. In order to convince your reader that the outcome has been fully or partially achieved, a detailed analysis is crucial.
- Don’t simply submit your raw data without commenting on it. The data you have collected is most meaningful when it is embedded into the narrative of your report and linked back to the project and its outcomes.
This example demonstrates each of the ‘what to do’ steps and how to use them when you write your evaluation report. It is made up of extracts from high quality evaluation reports submitted to Youth Music (these have been adapted and changed in order to preserve the anonymity of the organisations and ensure continuity with the narrative of the example project).
This example shows a write-up for just one outcome – you would need to do a similar write-up of each of your outcomes.
Outcome 1
- To improve the employability skills of young ex-offenders
Indicators for Outcome 1
- Young ex-offenders’ assessment of their own employability skills
- Music leaders’ assessment of young ex-offenders’ employability skills
- Number and level of accreditations achieved
Data collection tools for Outcome 1
- Evaluation scales
- Music leader reflective diaries
- Arts Award and NOCN certificates
a) Indicator and data collection tool 1 (150 words)
Of 70 participants, 59 returned evaluations at the beginning (84%).We collected the next set of data halfway through the project, and at the end
At the beginning:
- 22% rated their motivation to find employment as strong/fairly strong and 14% felt they had a good idea of the job they would like to apply for.
- 24% felt they had the necessary skillset for a job in a customer service environment
- 29% agreed/strongly agreed with the statement ‘I feel good about myself’.
By the end:
- 83% rated their motivation to find employment as strong/fairly strong and 69% felt they had a good idea of the job they would like to apply for.
- 44% felt they had the necessary skillset for a job in customer service. This is not as high as we anticipated (8), but we have considered this further in the reflection below.
- 88% agreed/strongly agreed with the statement ‘I feel good about myself’.
b) Indicator and data collection tool 2 (150 words)
Music leaders completed reflective diaries at the end of every session, providing comments on the young people’s personal development, especially in terms of employment. Observations of one young person, David*, illustrate the distance travelled:
"David’s editing has progressed this week but there is still work to be done. He took quite a strong leadership approach in the group session demonstrating a confident streak in him that we haven’t seen before. However, he got frustrated with one of the other boys in the group and stormed off to work on his own half way through the session." – Music leader observation, week 3
"On his way out today David told me he was interested in training to be a care worker. He wasn’t convinced he should go for it but the fact he has acknowledged it shows progress, both in his confidence and in his consideration of others." - Music leader observation, week 10
c) Indicator and data collection tool 3 (150 words)
At the beginning of the project, some young people seemed reluctant to start an Arts Award, with one or two commenting that they didn’t want to put in the extra work towards gaining qualifications, they just wanted to have a bit of fun. However, this reluctance started to lift and 62 of the young people were awarded their Bronze Arts Award earlier this month. A few have expressed an interest in progressing on to Silver while others are looking forward to using their portfolios and certificates on their CVs.
The NOCN qualifications were a little harder to come by as they involved extra preparation beyond the remit of what the young people needed to do for their Arts Award portfolios. However 30 young people were entered for their Level 1 Certificate in Employability and we are awaiting the results. We anticipate that at least two thirds will achieve this goal.
Reflecting on the above, summarise what the indicators collectively demonstrate about your progress toward this outcome. In your discussion, you may wish to consider the strengths and limitations of the data you collected and any anomalies in your findings. Do all your indicators suggest movement in the same direction? If not, can you suggest why? (300 words)
We have seen a great deal of progress towards this outcome, which is mostly evidenced by the young people’s perception of their increased employability skills, but supported by their accreditations and our Music Leaders’ observations. The most pertinent result is that our participants feel better about themselves and have found motivation to enter employment in the coming weeks and months, following the end of the project.
There were those who were less positive in their final evaluation scales, and we consulted them on the reasons why this might be. Reasons include feeling that they needed some more direct one-on-one sessions with a careers adviser, or wanting to follow a more specific route like healthcare or cooking rather than a general customer service role. For those who were still unsure, we plan to refer them to our partners who can offer advice that is more specific to their needs.(8) This nevertheless demonstrates an awareness of what is required of them to enter employment professionally and confidently, and their willingness to improve is demonstrated in the successful completion of Arts Award and NOCN accreditation.
We recognise that there could be external factors affecting these improvements as well as the intervention of the project. For example, two of our participants whose parents have been very active in their efforts to help find the young people employment have been in contact with a separate careers specialist who has worked one-on-one with them for an hour a week for the final five weeks of the project. (9)
Finally our music leader observations have given us an insight into some of the specific skills the young people have honed, such as patience and team working, which will ultimately contribute to their successful entry into professional working environments.
- When writing your report, remember to make the most out of the data you have collected by telling us what you think it means in terms of the progress you have made towards your outcomes. Telling us what data you collected is one thing, but it means so much more if you tell us what implications this data has for your project, its outcomes, and your future work.
- Tell us about any surprises in the data that you perhaps weren’t expecting – for instance, if the data is telling you that your progress towards your intended outcomes isn’t as far as you anticipated, that’s OK, but tell us why and what you think about it. Equally, tell us about any limitations of your data, for example, if you’ve had issues with collecting from a particular source, make sure you highlight that in your report.
- It is important that you don’t submit your raw data without commenting on it. This doesn’t do your hard work justice, and it could also have implications for your data protection responsibilities.
