Evaluation guidance - monitoring your outcomes

When choosing your indicators, you will naturally start to think about how you will collect and record this data. The data that you capture in relation to your indicators are called data collection tools. They are records of your indicators that allow you to measure or assess progress toward your intended outcomes. For example, if your indicator is young people’s self-assessment of their musical skills, your data collection tools for this indicator are the physical (or electronic) records of their self-assessments, such as evaluation scales, reflective diaries, or video interviews.

An indicator can have many different data collection tools, and it is up to you to decide which data collection tool is the most effective way of collecting and recording data relating to that indicator. Different data collection tools provide different types of data about your indicators, which have different levels of usefulness in different contexts. In particular, you need to decide whether your data collection tools will provide quantitative or qualitative data.

Quantitative data tools produce a number, quantity, amount or range. It includes data collection tools like evaluation scales. These data collection tools allow you to easily collect and analyse data from a large number of respondents but can provide a superficial view of what is happening. Although quantitative data collection tools may appear to be more objective, it is important to remember that they can also carry biases. For example, evaluation scales may be quantitative, but they are still rooted in the subjective opinions of the people completing the scales.

A primary benefit of quantitative data is that they can enable you to compare your work with other projects happening across the sector, particularly if you use shared measurement tools. Shared measurement comes from multiple projects or organisations using the same tools to monitor and evaluate progress toward similar outcomes. These tools are typically validated, meaning that there has been research carried out to demonstrate their effectiveness at measuring these outcomes. Using shared measurement tools also helps funding bodies like Youth Music and other infrastructure organisations to collate data from a wide range of organisations into a national body of evidence of the impact of music-making.

Qualitative data is descriptive, and less easily measurable. It includes data collection tools like reflective diaries and video interviews. These data collection tools provide a more nuanced view of progress but are more labour-intensive to collect and analyse. With large target groups, it may not always be possible to collect and analyse qualitative data for every participant, and you may need to focus on a smaller sample.

Different data collection tools for qualitative data can give vastly different results. For example, if you ask for young people’s self-assessment of their musical skills in one-to-one interviews and in group contexts, they may provide different levels of information or even give entirely different assessments. It is important to make sure your data collection tools are well matched to the respondent and the information you are seeking.

Data collection tool: Accreditations achieved

Strengths

• An objective measurement

Weaknesses

• If started at the beginning of the project, an accreditation is concrete evidence of a change or improvement in skill that can be cited as a result of the project
• Not always appropriate for target participants
• Sometimes costly

 

Data collection tool: Feedback forms

Strengths

• Gives a viewpoint about the project other than your own
• May identify things you had not thought about

Weaknesses

• May not be fully reflective of participant’s opinions: they may be saying what they believe you want to hear

 

Data collection tool: Informal feedback through conversations and observations

Strengths

• Feedback gathered can often be more spontaneous or genuine ‘off the cuff’ remarks
• Captures the person’s thoughts at that exact moment

Weaknesses

• Difficult to record spontaneously or recall exactly what was said later on

 

Data collection tool: Notes, minutes, and emails

Strengths

• Good supporting evidence which is easy to dig out and provide alongside an evaluation report
• Often provides good anecdotal evidence about a particular child or session that might be forgotten later on or in a more formal interview/questionnaire setting

Weaknesses

• Not always necessary/appropriate to share externally
• Rarely relevant as standalone evidence

 

Data collection tool: Evaluation scales

Strengths

• Objective – can be used with a number of participants to measure the same factor.
• Can often be interpreted quantitatively to generate percentages, statistics etc.
• Can be used at beginning/middle/end of a project to track progress.

Weaknesses

• Can be time consuming if you are using scales to measure a large number of participants.
• Often rigid – some scales may not be appropriate for target participants, for example young people with literacy problems may need support to complete it themselves.

 

Data collection tool: Reflective diaries

• Useful for gathering observations from the participant’s perspective, including things that they might not wish to say in an interview
• Can collect a high volume of data
• Data collection does not require a lot of organisational input as it is up to the participant

Weaknesses

• If too structured, the participant may not have the freedom to write what they really like.
• If too free, the participant may not know what to write.
• Time consuming for participant
• Analysis can be time consuming for organisation
   

Data collection tool: Photos, videos, and recordings

Strengths

• Easy and quick to share if time is limited
• A good way of showing off participants’ work resulting from the project delivered

Weaknesses

• Can be difficult to obtain the correct permission from parents/carers
• For evaluation purposes, they need to show us something – we always love to see pictures and videos of your projects but in this case they need to demonstrate a change rather than just document what you did.

 

Data collection tool: Observations

Strengths

• If recorded over a prolonged period of time, observations can demonstrate real change or development in groups and individuals
• Often more ethical than other more intrusive methods

Weaknesses

• Often requires an extra staff member to make and record observations, which isn’t always possible
• Can be difficult to keep consistent across a prolonged period of time
• People may behave differently if they know they are being observed

 

Data collection tool: Focus groups (e.g. youth boards and councils)

Strengths

• Good for gathering a lot of different perspectives in one go
• Participants often steer the conversation to interesting topics that interviewer may not have thought about, but are important.

Weaknesses

• Some participants may dominate the conversation – or shy away from saying anything. Interviewer may have to mediate or regain control if the conversation veers off topic.

 

Data collection tool: Structured interviews

Strengths

• Allows you to design your own list of interview questions to find out exactly what you want to know
• Can be used with a number of participants to measure the same factor

Weaknesses

• Strict
• Time consuming if conducting a lot of interviews
• Often contain a lot of closed end questions, which tend to generate less rich data

 

Data collection tool: Semi-structured interviews

Strengths

• Involves more open-ended questions which generate richer data, whilst still having a list of questions to guide the interviewer
• Allows you to design your own list of interview questions to find out exactly what you want to know

Weaknesses

• Time consuming if conducting a lot of interviews
• Although more flexible than structured interviews, this can make it harder to ensure the conversation stays relevant
• Answers will differ more, making them harder to compare to those of another interviewee

 

Data collection tool: Creative methods

Strengths

• Adaptable to exactly what you want to measure
• Provides a bespoke method of data collection for working with people who might have more complex needs

Weaknesses

• Can require a lot of thought and planning to be effective
• May be daunting to participants – needs to remain fairly simple

 

Data collection tool: Case studies

Strengths

• Detailed, personal, in-depth study of an individual participant and their story

Weaknesses

• Gives information from the perspective of just one individual
• Very personal, meaning it is unlikely that every participant will be willing

 

As you can see, all types of data collection tools have their strengths and their weaknesses. Therefore, as with indicators, it is important to use a variety of data collection tools so that these strengths and weaknesses balance each other out. By triangulating your indicators and data collection tools, or verifying your findings by combining multiple perspectives and methods, you can overcome the weaknesses of any one perspective or method, giving your evaluation greater credibility and validity.

In choosing your indicators and data collection tools, you should ensure that they are appropriate to your project, taking into account the ages and circumstances of your participants and that they will draw out the data you need in a positive way that will engage your participants. For example, if your project is working with children who aren’t confident with reading and writing, you will have more success collecting self-assessment data via video interviews than via written questionnaires. If your participants experience social anxiety, they will be likely to feel more comfortable providing feedback in one-on-one situations than in a group.

Additionally, you can embrace musical creativity in your approach to monitoring and evaluation! For example, you could use recordings of participants’ music-making as a prompt for them to reflect on their musical skills, perhaps comparing recordings from the beginning of the project and the end of the project. Or you could ask participants to draw a picture or write a short story that reflects on their development through your project.

Whatever data collection tools you decide on, it is important to consider at the outset how long it will take you to collect and analyse your data collection tools before you finalise your evaluation plan. For example, it may seem prudent to record all music-making sessions to use as evidence, but will you have capacity to review hours of video footage? Who will analyse the footage, and how long would it take them?

Likewise, it is important to consider the number of people you survey (i.e. your sample size). It is likely you would not have time to interview every participant taking part in your programme of work, but by interviewing a representative sample from across the programme you can present the range of experiences. Equally, if you are working with thousands of young people, you would not necessarily expect to be able to survey every participant to collect quantitative data, so you should think how to best represent the participants across your programme, and who it is most important to survey. A questionnaire completed by a participant who has attended a years’ worth of activity is likely to be much more revealing that one completed by a participant taking part in a single taster session.

 

Example: Music-Making Project

Aim: To improve the access to employment for ex-offenders aged 18-25 in Camden

Intended outcome

• To improve the employability skills of young ex-offenders

Indicators

1. Young ex-offenders’ self- assessment of their employability skills
2. Music leaders’ assessment of young ex-offenders’ employability skills
3. Number and level of accreditations achieved

Data collection tools

1. Evaluation scales
2. Music leader reflective diaries
3. Arts Award and NOCN certificates

 

Intended outcome

• To increase young ex-offenders’ confidence in applying for jobs

Indicators

1. Young ex-offenders’ self-assessment of their confidence in applying for jobs
2. Music leaders’ assessment of young ex-offenders’ job application materials
3. Feedback from employers about participants’ job applications and interviews

Data collection tools

1. Reflective diaries
2. Annotated job application materials
3. Email/telephone interviews

 

Intended outcome

• To improve employers’ perceptions of candidates with a previous conviction

Indicators

1. Number of different employers hiring graduates of our programme
2. Number of programme graduates receiving job offers from employers participating in the scheme
3. Feedback from employers about the work skills of young people hired from our programme

Data collection tools

1. Records kept by project staff
2. Records kept by project staff
3. Follow-up email/telephone interviews

 

Example: Workforce/Organisational Development Project

Overall aim: To improve the sustainability of music-making projects in early-years settings across the local area

Intended outcome

• To improve early-years practitioners’ skills in delivering music-making activities

Indicators

1. Early-years practitioners’ self-assessment of their skills in delivering music-making activities
2. Mentor/trainer’s assessment of early-years’ practitioners skills in delivering music-making activities
3. Project manager’s assessment of early-years practitioners’ skills in delivering music-making activities

Data collection tools

1. Reflective notes using Youth Music Professional Practice Scale and Youth Music Quality Framework
2. Session observations notes using Youth Music Professional Practice Scale and Youth Music Quality Framework
3. Session observations notes using Youth Music Professional Practice Scale and Youth Music Quality Framework
    

Intended outcome

• To increase support for music-making activities among senior leadership in early-years settings

Indicators

1. Self-reported level of support for music-making activities by senior leadership in early-years settings
2. Feedback from early-years practitioners about the level of support for music-making activities among senior leadership in their setting
3. Project manager’s assessment of the level of support for music-making activities among senior leadership in early-years settings

Data collection tools

1. One-to-one interviews with senior leadership in early-years settings
2. One-to-one interviews with early-years practitioners
3. Minutes from meetings with senior leadership in early-years settings
    

Intended outcome

• To strengthen the relationship between the local Music Education Hub and early-years settings

Indicators

1. Depth of partnerships between hub partners and early-years settings
2. Number of new partnerships between hub partners and early-years settings
3. Duration of new partnerships between hub partners and early-years settings

Data collection tools

1. Follow-up online survey to hub partners and early-years settings
2. One-to-one interviews with senior leadership in early years settings
3. One-to-one interviews with senior leadership in early years settings