Sheldon had alerted me to the recent stabbing of a young person, and we talked about being prepared for anything. He was right. An incident had kicked off and a young woman arrested near a large group of over-excited young people and bystanders. The youth workers began to talk to the young people, quietly moving through the group, calming things down. Eventually everyone dispersed…

The inevitability of dealing with complex situations has become part of ‘everyday’ youth work, where youth workers expect to face a myriad of people and events during any session – this is everyday practice, but also remarkable. Remarkable that a team of youth workers can walk into a storm of young people, general public and police and help to lift the fog. Remarkable that youth workers can form a bridge, translating differing needs across a community- speaking to the public, to young people and others. Remarkable that this practice, so undervalued, misrepresented and misunderstood continues to retain a sense of identity and purpose. (Fieldnote, LD, Melham Youth Service.)

We begin our article with this scenario highlighting one of many instances of everyday yet remarkable practice we have observed during our study, Rethinking Impact, Evaluation and Accountability in Youth Work. Our research investigates how youth impact and evaluation mechanisms are enacted in youth work settings, and how the evaluation and monitoring of youth work are experienced and perceived by young people and youth workers.

While reflecting on what we heard and experienced during project visits and research interviews, we were struck by the remarkable examples of the impact of youth work, as well as by the rich and meaningful nature of everyday youth work practice and its contribution to young people’s lives and wider communities. Yet we are concerned that the way practice is recognised and valued by those most deeply involved is disconnected from the way it is required to be measured, monitored and evaluated.

At a time when there is a growing focus on proportionate and equitable evaluation within and beyond youth work, we draw on our initial findings to argue that evaluation must be more clearly rooted in the needs and realities of practice by exploring the following questions:

1. Does evaluation suit the setting?
2. Does evaluation reinforce or challenge unequal power relations?
3. Does evaluation capture and value both the everyday and the remarkable elements of practice?

While our discussion focuses on youth work, we hope these questions might be useful to evaluators, practitioners, educators and researchers from a range of related settings. We offer this article in the spirit of collaborative endeavour and debate, as a contribution to the defence and celebration of youth work, as well as to the continued development of suitable and flexible systems of evaluation and accountability more widely.

The article begins by summarising the methods of our study and providing our definition of ‘open youth work’. We then discuss the tensions in measuring and evaluating youth work in the current political, economic and social context, before explaining why the three questions above should be considered when designing evaluation. We conclude by advocating for a grassroots democratic approach that centres on creating the conditions for youth work to flourish.

Our study

Our three-year study (2018-21) investigates how a policy focus on impact measurement is enacted in youth work settings, and how it is perceived and experienced by those involved. The study will involve around 150 participants in qualitative research over four phases:

1. Policy network analysis
2. Visits, focus groups and interviews
3. In-depth case studies
4. Film-making, dialogue and mutual learning

This article is the first to be published from our research. It draws on Phase Two of our study, in which we visited eight youth work settings around England between two and four times each in the first half of 2019. We participated in and observed 26 youth work sessions, debriefs and meetings, and undertook 33 interviews and focus groups with 15 managers and administrators, 27 youth workers, and 43 young people. Interviews were recorded and professionally transcribed with the informed consent of participants; all individuals and organisations have been anonymised. Transcripts and fieldnotes were analysed through a collaborative process of open and thematic coding, discussion, and memo-writing. We will report in more detail in future articles on the methods and findings from this and other phases of our research.

While social impact is a current focus of policy and practice throughout the public and social sectors, our research focuses specifically and deliberately on ‘open youth work’ in England. The term ‘open youth work’ is widely used in Europe to denote open access settings such as youth clubs and detached (street-based) work (POYWE, 2016); we also include work aimed at specific groups focused on shared identities and experiences, rather than on adult-defined needs, deficits or ‘risk factors’. For us, these practices (clubs, detached and shared identity groups) have an ‘open’ orientation in two senses:

Firstly, they are open to young people to join on a voluntary basis without compulsion or coercion; they can join by choice, rather than being referred or targeted. Groups are open to everyone in their remit but may have some boundaries – for example, there is often an age limit, and some groups are for those with specific shared identities and experiences (e.g. work with young women or Black and minority ethnic youth groups).

Secondly, they are open in terms of timescale, content, and intended outcomes. This contrasts with pre-planned, time-limited interventions and those that aim primarily to serve adult-defined objectives (e.g. to lower teenage pregnancy rates, reduce knife crime, improve employability, or raise church attendance). Open-ended does not mean ‘anything goes’; projects may emerge, there might be a routine to sessions, and there could be a focus on specific activities. Yet in open youth work, these structured elements are neither compulsory nor permanent; they are fluid and responsive to need.

While there are tensions and challenges involved in evaluating all forms of practice, these characteristics of open youth work can make it a particularly challenging context, as we will now discuss. 

Measuring youth work: tensions and challenges

Previous research has set out the disjuncture between youth work practice and how it is evaluated, emphasising that open youth work is an essentially informal and process-oriented approach with unpredictable outcomes that emerge over time (Duffy, 2017; Ord, 2014; de St Croix, 2018). Open youth work is acknowledged in policy as a particularly challenging context for impact measurement (NPC and Centre for Youth Impact, 2019). Our starting point is that this is not merely a practical challenge, but also a political one. Measurement and evaluation are shaped by their economic and social context; they also have the potential to reproduce and/or challenge existing power relations. Therefore, it is essential to pay attention to the context – characterised by inequalities, poverty, colonial histories and environmental devastation – in which youth work and its evaluation take place.

Youth work is open to all, yet is often chosen by groups of young people who are marginalised and oppressed in wider society, particularly working class young people, many of whom live in poverty or on low incomes and have limited spaces to spend their time outside of school. The youth club can be more than simply a leisure space; for many it is a ‘home from home’ a space of safety, and a space of sustenance, sometimes quite literally:

“We do [the food] really low cost…being aware these young people are coming from very deprived families… there’s a school holidays poverty… at least we know they’re getting a meal with us.  And what we try and do is open it out to the community as well, so mum and dad might come… it’s a really big issue round here, round food poverty, especially…when they’re not in school.”  (Zara, programme manager, The Vaults.)

Such contexts shape the setting, as well as the acceptability or unacceptability (in the eyes of youth workers and young people) of impact measurement practices. Working class young people of diverse gender and racialised identities are already disproportionately subjected to practices of surveillance, othering and measurement that are rooted in histories of eugenics, colonialism and social control (see e.g. Khan, 2013; Merry 2011; Baldridge, 2019). Thus, what might be seen as unhelpful reluctance on behalf of young people and youth workers to comply with evaluation and tracking may be an understandable resistance to being subjected to measurement and surveillance as a means of social control (Skott-Myhre, 2006).

While evaluation can be an opportunity for mutual learning and practice development, it is also a practice of neoliberal governance in which organisations must compete to survive. Accountability mechanisms are performative; in other words, they do not merely represent practice but shape practice, often rendering it standardised and comparable (Ball, 2003). Asking young people to complete complex and intrusive paperwork can undermine the informality of settings and obstruct relationships between youth workers and young people (de St Croix, 2018). Ill-fitting or superficial methods of monitoring can create an uncomfortable dilemma for practitioners, enforcing the quantitative measurement of short-term ‘outcomes’ on a qualitative, long-term practice.

Whilst visiting youth work settings, it has been notable that impact measurement, monitoring and evaluation infuse daily life, appearing taken-for-granted and necessary, whilst simultaneously perceived as problematic. Managers made principled attempts to prevent practice being shaped inappropriately by measurement; in some settings, colleagues and young people were consulted and meaningfully involved in evaluation processes; in others, youth workers were shielded to some extent from the bureaucratic burden. Nonetheless, when preservation of jobs and the very survival of projects and organisations are at stake, performative impact mechanisms were difficult, if not impossible, to avoid.

Strategies for democratising evaluation

In discussing tensions, challenges and constraints, we are not arguing against evaluation and accountability; indeed, in the settings we visited there was an acknowledgement that certain types of evaluative processes supported a reflective approach to practice. As experienced youth workers, we recognise that meaningful evaluation contributes to professional development and the centring of young people’s views and experiences; we also recognise that funders (governmental and non-governmental) want to know that organisations are accountable for the resources they use. Nevertheless, impact measurement and accountability mechanisms can and must be more clearly rooted in the needs and realities of practice. To support a more democratic approach, we raise three questions for consideration when designing and carrying out evaluation.

Does evaluation suit the setting?

Youth work is different from school, social services, youth offending, and other statutory services that intervene in young people’s lives. The primary differences are its informality, its youth-centred nature, and the voluntary principle (young people choose whether and how to get involved). In focus groups with young people and youth workers, it was clear that some forms of monitoring and evaluation were experienced as repetitive, boring or meaningless:

That’s what school is for. The thing here [in the youth club] is outside [of school], you come to play pool or whatever, whatever. You don’t come here to fill forms, you know? It’s like – I’m not going there if they’re gonna make me fill forms. (Jasmine, young person, Fairlight Youth Centre)

The norms and cultures of individual youth work settings vary, and ‘one size fits all’ approaches are insufficient. For example, while many young people disliked form-filling, others accepted questionnaires that felt relevant and non-intrusive. Similarly, some young people enjoyed reflective conversational and creative approaches to evaluation, while others preferred a quicker method, or to be left to enjoy the space with their peers on their own terms. Notably, it seemed that any approach was likely become dull, lose meaning and feel imposed if used repetitively and/or arbitrarily over time. It is important to recognise the flexibility of youth work as a practice – youth workers adapt, almost chameleon-like, in their interactions with young people and many expressed a desire for evaluative methods that could be moulded to fit their needs in any given setting or interaction.

Rather than proposing a single approach, then, we suggest that policy makers, funders and managers engage with those closest to practice – young people and youth workers – enabling them to use a range of tools and approaches that can be selected and used flexibly in different settings and by different young people over time. In some cases, a quick and easy survey may be least intrusive; in others, a participative and reflective process may be more meaningful and appropriate. While such an approach would may limit standardisation or comparability, we argue that these are not the primary aims of evaluation at the project level. Instead, there is a need for open, flexible and dynamic approaches that are non-prescriptive and sensitive to the realities of youth work practice.

Does evaluation reinforce or challenge unequal power relations?

At its best, youth work creates an anti-oppressive space for young people to gather and spend time in their peer groups, celebrating and developing their own cultures. Yet youth work practices – and their evaluation and monitoring – also have the potential to reinforce structures of domination and inequality. The young people engaging most regularly and intensely with youth work are often from minoritised groups experiencing oppression; in addition, many have been labelled elsewhere as ‘at risk’ or ‘risky’ in their behaviours or in terms of their neighbourhood or family backgrounds (Baldridge, 2019). As such, many experience heightened levels of surveillance, and are suspicious and critical of evaluation and monitoring that seems to echo or reinforce unequal structures:

It’s invasive… I mean I get that question a lot, do you have a disability? … and maybe you do but you don’t wanna like disclose that, you know?… Questions can be asked that are a bit much. But then they should give you an option to be like you don’t have to answer this if you don’t want to. Although people say that… Someone asked [youth worker] if this was gonna give [the project] more money… and he said probably, and they were like right, give me it… Because they were like, this youth group means so much to me, I have to like do it, you know. (Sabian, young person, Journeys.)

Many youth workers have witnessed the disproportionate policing of Black young people, the state surveillance of Muslim young people, the medicalisation of disabled young people and those experiencing mental health issues, or the intrusive questioning of LGBTQ+ young people. As such, many expressed discomfort about monitoring and evaluation and experienced it as disrespectful and disruptive of trusting relationships. Yet, as can be seen above, if a funder required certain information, staff and young people felt they had little choice but to comply.

We suggest therefore that funders and policy makers need to lower the stakes, while avoiding recording and evaluation mechanisms that are intrusive, surveillant, or otherwise reinforce inequalities and oppression. Where evaluation claims to be about ‘learning’, there is a need to reflect on whether this is how it is experienced on the ground. Ethics need to be considered, and perhaps academic research practice can be instructive here; in research, participants must have the right to informed consent over whether they take part, should not be disadvantaged if they choose not to do so, and questions should (where possible) avoid causing stress (BERA, 2018). These principles could well be extended to monitoring and evaluation.

Does evaluation capture and value both the everyday and the remarkable?

Amanda is about to do her homework at a desk in the corner, but wants to tell me her views on the youth club: “For the majority, this is a place to come, play games, see friends, do nothing for a while. For a few, they come for a certain reason. To get support, to get something off their chest. The workers are always there when we need them. (Fieldnote, TD, Dove Street Youth Project.)

Our emphasis on the importance of the ‘everyday and remarkable’ aspects of youth work comes from spending time in youth work settings, witnessing a reflexive, responsive and responsible process that is created ‘in the moment’ and developed over time, often in adverse circumstances. Evaluation cannot possibly capture everything and is likely to be reductive of the complexity and subtlety of what actually happens. In our view, rather than look for more evidence or one tool that meets all needs, it is important to match a fluid process with a dynamic range of approaches.

In terms of evaluating the ‘everyday’ aspects of youth work, it is important to emphasise the intrinsic value for young people of having a space outside of school and home where they can be together. This needs only light touch evaluation that enables young people and youth workers to reflect, develop, and own the space. This may be best achieved through building awareness amongst funders and politicians about the value of youth work and what it contributes to young people’s everyday lives.

Alongside and arising from the everyday, youth work is rich in remarkable events, processes and conversations: from what one youth worker called the ‘sombre moments’ when a young man opens up about his emotions for the first time, to the avoidance of a potentially tragic incident. These aspects of youth work might perhaps be best represented by qualitative research – whether interviews, story-telling, focus groups or creative research methods – although sensitive ‘tick box’ questionnaires (with the option of qualitative comments) might be practical in some situations. Above all, evaluation systems need to be dynamic, flexible and adaptable.

Conclusion

In looking critically at youth impact, we are not denying the need for accountability or evaluation; however, we assert that there are significant limitations to much of the evaluation currently required by funders and called for in policy. When inappropriate mechanisms are introduced in youth work settings, they distort the distinctive relationship between youth workers and young people, and fail to capture either the everyday value and influence of youth work or its remarkable contribution to the enrichment of young people’s lives.

Rather than seeking to ‘measure’ practice, a grassroots democratic approach to accountability would attempt to create the conditions in which high quality practice can be nurtured and developed. These conditions must include less hierarchical relationships within the youth sector, funding bodies and policy-making more generally, to allow for the development of evaluation processes that enhance practice, are anti-oppressive, build trust, and not reductive of the complexity and subtlety of what happens in youth work settings.

More fundamentally, perhaps, there is a strong argument that the conditions for high quality youth work do not centre on evaluation exclusively, but also on long-term investment, support for professional training and education, the valuing of staff through decent contracts, and halting the sale of buildings and the closure of popular grassroots facilities. Appropriate youth-centred evaluation is needed but what is most important is the rebuilding of an adequate and proportional youth work sector that is able to have an everyday value and a remarkable impact on young people’s lives.