Developing a community of student engagement practitioners: Evidence of value from a postgraduate professional development course at a UK university
Alan Donnelly1, Liz Austen1, Tom Lowe2 and Dr Maria Moxey3
1 Sheffield Hallam University, Sheffield, UK 2 University of Westminster, London, UK 3 University of Bristol, Bristol, UK
Corresponding Author:
Alan Donnelly, Student Engagement, Evaluation and Research (STEER), Sheffield Hallam University, Howard Street, Sheffield, S1 1WB, UK
Email: A.Donnelly@shu.ac.uk
Abstract
Student engagement practitioners, who work within and beyond higher education (HE) providers, have vital roles in contributing to students’ learning and experiences and they collaborate with various staff and student groups across organisational boundaries to initiate change. It is essential that organisations and the HE sector support these staff to achieve their potential, for example, by addressing the lack of relevant professional development opportunities available. This paper presents evidence of an effective pathway of development that enables a community of student engagement practitioners, conceptualised as ‘integrated practitioners’ (McIntosh & Nutt, 2022), to have impact in diverse HE settings. The findings of an impact and process evaluation of the UK's first accredited professional development course for student engagement practitioners, the Postgraduate Certificate (PGCert) and Master’s (MA) in Student Engagement in Higher Education at the University of Winchester, are presented. Co-constructed with an advisory group of practitioners who were studying on the course and graduates, the evaluation was planned and structured using Theory of Change. A total of 22 practitioners provided evidence of the value of the course by taking part in peer-led interviews, online reflective activities and meetings, while seven staff members from the programme team participated. The paper outlines how the evaluation generated understanding about the purposes and design of the course; the effectiveness of its blended delivery model and processes; and how the course contributed to the formation of a community and equipped practitioners to enact changes in their practice and within their organisations.
Keywords
student engagement, practitioner, integrated practice, higher education, course evaluation, theory of change
Introduction
Higher education (HE) providers in several Western countries, such as the United Kingdom and Australia, are increasingly facing the challenges of responding to the marketisation of HE sectors (Lowe, 2023). Lowe and El Hakim (2020) observed that providers are under “growing pressure from numerous governments globally to produce measurable student outcomes, enhance learning and evidence an investment worthy of the taxpayer and tuition fees from students” (p. 3). Beyond marketised HE national systems, wider stakeholders at all levels within and beyond HE providers are also focusing on student engagement to enhance student support, learning and success as best practice and to respond to data driven measures. Scholars and practitioners across the world have emphasised the importance of adopting approaches to student engagement where there is a commitment to working with students as partners in educational developments and providing student centred services and pedagogies (Lowe, 2023; Lygo-Baker et al., 2019; Tight, 2020). Amid both best practice and this neoliberal demand for creating and accessing sources of knowledge about student engagement, there is a need to support both academic and professional staff, who find themselves accountable, to reflect from within their practice and avoid uncritical adoption of student engagement practices (Moxey et al., 2022).
The term ‘integrated practitioner’ refers to staff who are working in roles and spaces in HE that span academic and professional areas of activity (McIntosh & Nutt, 2022). These practitioners have a key role in contributing to the learning and experiences of students and staff and in helping organisations meet their goals. McIntosh and Nutt (2022) wrote that integrated “practitioners must be supported, encouraged and nurtured if they are to bring about the change […] required to support universities in the future” (p. 224). It has been suggested that staff in these roles can experience a lack of recognition and be affected by barriers to career progression and a lack of access to relevant professional development due to the diversity of their roles and the rigidity of existing organisational structures (Akerman, 2020; Webster, 2022). Locating existing opportunities for development, or creating new ones, is often dependent on practitioners’ own agency (Nutt, 2022). Organisations and the HE sector must be more proactive in recognising and supporting their progression and potential to initiate change. This paper presents evidence of an effective pathway of development that enables a community of student engagement practitioners to have impact in diverse HE settings.
Context
In 2017, the University of Winchester concluded the Realising Engagement through Active Culture Transformation (REACT) project, which spent two years researching and developing student engagement practices in English HE. This project identified the need for further professional development opportunities for academics and professional staff members of all levels who are tasked with engaging students in educational developments. To the knowledge of the programme team, there were no or very few existing relevant qualifications in the UK and beyond aside from discipline or teaching-focused programmes (Moxey et al., 2022). Student engagement practitioners “are central to building partnership between staff and students and between institutions and students’ associations” and they “regularly connect operational support for student representation and feedback, with strategies for learning, teaching and wider institutional functions” (sparqs, 2022, p. 2). As the University of Winchester closed the REACT project, the senior management team established a new research Centre for Student Engagement which would explore an income generating staff development opportunity to address the needs identified previously. In 2017-18, the UK’s first PGCert Student Engagement in Higher Education was validated at the University of Winchester, which was followed by the validation of a two-year MA programme in 2020. Since September 2018, 66 practitioners have studied on the course, of which 14 worked at the University of Winchester and could study for free, and 52 were located elsewhere across the UK and Irish sectors. These practitioners worked in diverse spaces within sector organisations, students’ unions and HE providers, and across professional and academic roles relating to student support and experience, quality assurance and enhancement, and teaching and learning.
Lowe and Bols (2020) suggest that a focus on student engagement in HE is “necessary to create meaningful conversations, research and projects between staff and students, in order to develop education in our communities and beyond” (p. 282). Their commentary of the development of student engagement as an emerging field of study provides a rationale for the creation of a professional development course and the need for a bespoke evaluation of its outcomes. The concept of integrated practice is an appropriate lens for exploring the experiences and identities of staff working in partnership with students (McIntosh & Nutt, 2022), who are united in their values and interests rather than by their specific roles. This context has influenced the exploration of how the PGCert/MA contributes to the development of a community of integrated practitioners and the methodological approach outlined in this paper.
The course’s part-time mode of study aims to support practitioners to study while they also occupy professional roles in HE. The PGCert contains two modules over a single academic year. Students can choose to progress on to the MA which contains five modules over two academic years. Using a blended delivery approach, practitioners are expected to attend a campus-based retreat at the beginning of each module and subsequent learning occurs in the course’s online environment through readings and lectures. Each retreat lasts two days and is designed to build practitioners’ understanding of different perspectives, practices and policies in student engagement by grounding the learning in the experiences of their peers and staff on the course. The course is taught by staff based at the University of Winchester and visiting fellows, who are staff positioned at other institutions. As student engagement practitioners are often “the only people doing precisely that job in their institution” (Varwell, 2022, para. 5), the delivery of the course “opens up the possibility of fostering developmental communities of interest comprised of otherwise isolated individuals” (Parkinson et al., 2020, p. 199). The effectiveness of this delivery model has potential implications for the cultivation of HE communities across multiple professional contexts or locations.
As previous evaluations focused on capturing practitioners’ outcomes up until the point of course completion, the programme team identified a need to learn about the long-term impacts, to understand how the course was contributing to these impacts, and to identify potential areas of enhancement. An external evaluation was commissioned and a team of independent evaluators from Sheffield Hallam University worked on the project.
This paper explores the following questions:
1. What are the expectations and experiences of student engagement practitioners prior to enrolling on the course?
2. How effective are the processes underpinning the course at creating and sustaining a community of student engagement practitioners?
3. What value does the course have on student engagement practitioners, their organisations of employment and other groups?
This paper will provide a brief overview of existing literature, which relates to the broader area of professional development programmes in academic and educational development, and it will articulate how these sources have influenced the evaluation design. The methodology section will outline the study design, the evaluation approach and provide a rationale for the data collection, the data sources and the analytical framework that were used. The results section will present the findings of the evaluation under the different stages of the learning journey of practitioners on the course, which comprises Pre-course expectations and experiences; Programme design and delivery; Learning; and Legacy. The discussion section will then address the three research questions in detail. The 4M Framework (Friberg, 2016) will be applied as an analytical framework to address the third research question, which explores the value of the course at different levels within and beyond the context of an organisation: micro; meso; macro; and mega. The 4M Framework has been used previously to evaluate the impact of interventions and educational development programmes in the context of staff-student partnerships (e.g., Marquis et al., 2020) and in other areas of teaching and learning, such as scholarship of teaching and learning (e.g., Simmons, 2020). Further information about this framework is presented in the methodology. Finally, the conclusion will provide a brief overview of the study, explain its significance in relation to the areas of student engagement and professional development in HE spaces, and highlight the study’s limitations.
Areas and indicators of value
A desk-based review of contextual literature, which included databases, search engines and individual websites, was undertaken to identify potential areas of value of the course. However, there was no existing literature specifically relevant to student engagement professional development programmes. The literature review subsequently focused on the broader areas of academic and educational development and how the value of these programmes has been articulated and evaluated. Within previous literature, potential indicators of value of these programmes have been proposed at different stages of the learner journey (Table 1). However, the quality of evidence to support these links is mixed and evaluation approaches have been criticised for focusing on participants’ satisfaction rather than the long-term impact (Inamorato dos Santos et al., 2019).
Table 1. Potential indicators of value of academic and educational development programmes (e.g., Inamorato dos Santos et al., 2019; Kneale et al., 2016; Rutten, 2021)
|
Area of value |
Indicators |
|
Participants’ knowledge and skills |
· Pedagogical knowledge and research skills · Collaboration skills · Understanding of students and their needs · Increased value placed on reflection · Enhanced confidence and self-efficacy · Willingness to adopt new practices |
|
Participants’ behaviours and practices |
· Improved teaching practice · Application of skills and knowledge · Collaboration with others · Engagement in scholarship and research |
|
Students |
· Satisfaction · Learning · Achievement |
|
Organisational culture |
· Expectations around teaching within local areas · Sharing knowledge and practices · Development of teaching policies and strategies · Reward and recognition opportunities |
|
Career progression |
· New career responsibilities, job or promotion |
|
Other |
· Development of professional relationships, networks and communities |
Bamber and Stefani (2016, p. 242) called for practitioners to challenge the “positivist, new managerialist” discourse used by many HE providers, which privileges quantitative metrics about students’ learning and assumes that linear relationships exist. These authors and others advocated for the use of contextually grounded approaches that account for the complexities of academic development (Miller-Young & Poth, 2021) and which evidence the value of programmes, such as changes to participants’ thinking, practices and confidence. Theory-driven evaluation, such as Theory of Change, has been championed as a way of demonstrating how and why a programme is expected to bring about change and for establishing coherence between course design, outcomes, and evidence collection (Amundsen & D’Amico, 2019; Bamber & Stefani, 2016). This approach was adopted in the course evaluation.
Methodology
Study design and approach
A post-test design was applied to evaluate the processes of delivery and to understand the value of the course. This study comprised an impact and a process evaluation, which are commonly used within the same framework to provide a breadth of evidence (Parsons, 2017). An impact evaluation is focused on “measuring […] the consequential change that results from […] activities” and understanding the attribution of impacts, while a process evaluation “focuses on evaluating the mechanisms through which an intervention takes place” (Parsons, 2017, pp. 16-17). The evaluation was structured using Theory of Change (ToC), which articulates how, why, and in which context an intervention operates (Lam, 2020). A ToC model can account for multiple pathways of change, starting points, and outcomes (Miller-Young & Poth, 2021) and the learning theory underpinning the course design (Amundsen & D’Amico, 2019).
Practitioners currently studying on the course and graduates were recruited to form an advisory group, who were responsible for working collaboratively with the evaluators to co-construct the ToC model and outcomes for the course, design the data gathering methods and reflect on findings. Student engagement practices within HE have been widely discussed in relation to models of staff-student partnership (Lowe & Bols, 2020). Adopting a participatory approach enabled responsibility for the evaluation to be shared with practitioners on the course, key questions and outcomes to be identified and the evaluation design and methodology to be strengthened (Saunders et al., 2011). A total of 16 practitioners expressed an interest in joining the advisory group by completing an online sign-up form, where they provided information about their professional background which was used to select 10 advisory group members. Consistent with the integrative methodology (Jones-Devitt et al., 2017), discussions from the three advisory group meetings were captured as data and formed part of the evidence base. Advisory group members could also participate in the other primary evidence collection (interview and Padlet). Further information about this participatory approach has been outlined in Austen and Donnelly (2023).
The evaluation obtained ethical approval from the University of Winchester and Sheffield Hallam University. Participants were required to provide written informed consent to take part and they were informed of their rights to withdraw, their confidentiality and anonymity in the processing of the data and the dissemination of the findings. Advisory group members were paid £10 for each hour that they contributed, while practitioner participants in the other primary evidence collection were also paid.
Data collection
A mixed-methods, predominantly qualitative, approach was used to collate evidence from a range of sources and perspectives between May and July 2022. Given the importance of evidence that is situated within individual programmes (Bamber & Stefani, 2016), primary data were collected to capture the narratives of the student engagement practitioners on the course and the staff responsible for its design and delivery. Storytelling is commonly utilised in evaluations to “understand motivation, values, emotions, interests, and factors that influence behavior” (Krueger, 2010, p. 405).
The advisory group suggested that the evaluators gather the primary evidence from practitioners using peer-led interviews. Each interview, which was held on Zoom online video conferencing platform, involved two current learners or graduates who took it in turns to ask each other a set of questions and provide a response. The peer-led interviews, which were influenced by the ‘Listening Rooms’ method (Heron, 2020), enabled discussions to occur without interruption by the evaluators, who were only present to welcome participants, introduce the structure of the interview, share the questions on-screen, and debrief participants. The questions opened with an adaptation of the ‘Most Significant Change’ concept, which is a participants-led story generation technique (Davies & Dart, 2005). Outcome-aligned questions were then asked which explored practitioners’ motivations and expectations (e.g., ‘What were your reasons for studying on the PGCert/MA?’), course value (e.g., ‘What has the PGCert/MA helped you to achieve?’) and processes (e.g., ‘Are there any factors about the design and delivery of the course that have: contributed to the impact of the PGCert/MA; or created barriers to it having impact?’). If scheduling issues arose, the peer-led interview could be pivoted to a one-to-one interview with the evaluator, which happened six times.
Padlet sites, which are virtual bulletin boards, were used to capture the perspectives of the advisory group and the programme team, while practitioners on the course had the option of taking part in the interviews, Padlet, or both. Each stakeholder group was given access to separate Padlet sites, which were password-protected and anonymous to encourage honest contributions within a safe environment. Existing evaluation evidence about the course, such as student experience questionnaires and external examiner comments, were analysed as secondary data sources.
Data sources
A stratified purposeful sampling approach was used “to capture major variations rather than to identify a common core, although the latter may also emerge in the analysis” and to investigate individual groups that may “constitute a fairly homogenous sample” (Patton, 2002, p. 240). Two stakeholder groups were identified who were “information-rich cases” (Palinkas et al., 2015, p. 2). The first group was practitioners studying on the course, who were split into three sub-groups: those currently studying; graduates; and a small number of students who started the course but withdrew. The second group was the programme team, which included visiting fellows. All members of these samples received information about the study via email from the course leader and they were asked to register their interest by completing an online form. Table 2 shows that 22 practitioners, most of which worked elsewhere than the University of Winchester, and seven programme team members contributed to the primary evidence collection. No practitioners who withdrew from the course took part.
Table 2. Number of participants in the primary evidence collection
|
|
Practitioners |
Programme team |
||
|
Currently on the course |
Graduates |
Withdrew from the course |
||
|
Both interviews and Padlet |
7 |
6 |
|
|
|
Interviews only |
0 |
1 |
0 |
|
|
Padlet only |
2 |
0 |
|
7 |
|
Advisory group meetings only |
5[1] |
1[2] |
|
|
|
Total participants |
14 |
8 |
0 |
7 |
Data analysis
The primary evidence was entirely qualitative and was analysed using thematic analysis (Braun & Clarke, 2006), which is a process of identifying patterns and themes. The analytical framework was predominantly deductive, which involved coding the data against outcomes from the ToC model, describing the themes that were found and critically exploring their links to existing evidence. These outcomes were developed from input from the advisory group, the programme team, evaluation data and research. An inductive component was also used so that additional themes that were identified as being relevant to the evaluation questions were incorporated. Evidence was triangulated from multiple methods, perspectives, and sources to strengthen the evaluation methodology and to ensure that there was confidence in the conclusions (Parsons, 2017).
The 4M Framework (Friberg, 2016) was also incorporated into the analytical framework to address the third research question and to explore the value of the course at different levels within and beyond the context of an organisation. The 4M Framework comprises four levels (Marquis et al., 2020):
· micro refers to individual staff and students;
· meso is a cluster of courses, departments, or teams;
· macro is entire organisations or providers; and
· mega refers to the wider community of student engagement.
For the purposes of this study, meso and macro were combined to form a single level.
Results
The findings are presented under the different stages of the learning journey of practitioners on the course and the wider impact of the course: 1) Pre-course expectations and experiences; 2) Programme design and delivery examines the effectiveness of the course’s processes; 3) Learning focuses on the knowledge and skills which practitioners achieved while studying on and completing the course; and 4) Legacy examines the medium and long-term changes to practitioners, their organisations of employment and other groups.
The sub-sections of the results section align with the course’s ToC model (Figure 1) in the following ways: Programme design and delivery (inputs, activities and processes); Learning (short-term outcomes); and Legacy (medium-term and long-term outcomes).
Figure
1. Theory of Change model for the PGCert/MA Student Engagement (adapted
from Donnelly & Austen, 2022)
Pre-course expectations and experiences
Student engagement practitioners on the course were united in their desire to build on their existing experiences and acquire further knowledge and skills, but there were also various individual motivations for studying on the course; for example, career progression, personal goals and acquiring a postgraduate qualification. It was common for practitioners to have years of experience working within areas of student engagement but some “did not have much knowledge of literature” (graduate) and many reported having limited or “no access to student engagement communities” (graduate). Several practitioners were keen to improve their self-confidence, address a perceived lack of recognition in their roles, such as when working with academic colleagues, and enhance their career prospects. Practitioners felt that the area of student engagement was sometimes viewed as being ambiguous and subsequently misunderstood by the wider HE sector.
The course’s unique focus was reflected in the responses of practitioners, who stated that there were either no or very few other appropriate types of professional development available for their own contexts. For instance, one participant stated: “I didn’t want to do something for the sake it, I wanted it to stretch [me] personally and professionally and bring back to my role” (current learner). References were regularly made to the course’s specialist and sustained nature. Practitioners alluded to having transformative and supportive conversations with the programme team, who provided clarity and assurances and were proactive in contacting potential students “to find out about their motivations” (current learner).
Programme design and delivery
There was evidence that the learning environment provided practitioners with effective access to course materials and learning. The blended approach optimised accessibility as “students from a wide range of locations, backgrounds and roles” were able “to do the programme alongside their work” (staff). A graduate stated the course was “perfect in the way it was structured, delivered, timing of classes, assessment deadlines […] it suited us and the way we lived our lives outside the course”. A staff member was able to “deliver a session from America which brought an international perspective to teaching that we wouldn’t have otherwise had” (staff). The quality of the course’s online and on-campus delivery and the communication tools and support available meant that practitioners “did not feel that they were learning at a distance” (graduate). The impact of COVID-19 was minimised due to the effectiveness of the learning environment and online delivery. The expectation to attend on-campus for the retreats was a potential barrier for those who lived far away from Winchester due to the monetary and time costs of travelling, which meant that the option of engaging online, if necessary, was welcomed.
The course provided effective spaces for practitioners to learn and interact. The retreats at the beginning of each module, especially when held on-campus, allowed practitioners to build relationships with their peers and staff, establish common interests, and address shared concerns. Meeting their peers aided practitioners’ transition onto the course, as some individuals were anxious having not studied in HE for several years, when they realised that there were “other people in the room in the same position as me” (graduate). This sense of camaraderie continued throughout the course: “each week I’ll speak to people across the country with similar challenges and expectations” (current learner). Social components embedded into the design of the in-person retreats, such as the opportunity to socialise over dinner, and teaching sessions, facilitated engaging and honest discussions in the online environment.
The work-related design of the curriculum, learning and assessments made it easier for practitioners to balance their studies with their professional roles, for example, the course and practitioners’ work was described as “interwoven” (graduate). The links between modules, and the possibility of building learning from one assessment to the next, enhanced the confidence of practitioners as it enabled “progression of content and thinking” (graduate), which was consistent with comments from external examiners. A few practitioners perceived that the course curriculum placed too much emphasis on the context of HE in England and in universities and these individuals felt that co-design could be further embedded to allow them to more comprehensively tailor topics to their own national and employment contexts.
The model of using visiting fellows, who are based at other institutions, to teach on the course was beneficial as it allowed practitioners to experience and access a variety of teaching and learning approaches and “perspectives from multiple institutions and contexts” (staff member). Practitioners commented that the teachers broadened their understanding of student engagement, while others identified that the opportunity to learn from reputable experts “validated the importance of the qualification” (current learner). The teaching staff were “approachable” and “supportive” (current learners) which enabled the individual needs of practitioners to be met. Positive perceptions about the teaching and support were also reflected in course-level satisfaction surveys.
Despite no representation from practitioners who withdrew from the course, existing course review documentation suggested that workload challenges and commitment issues were among the reasons. These were predominantly staff based at the University of Winchester who enrolled on the course for free.
Learning
There was evidence that practitioners routinely achieved their learning outcomes on the course, which indicated they had significantly broadened their understanding of student engagement, such as about staff-student partnerships, co-design, and students’ unions. Development of critical thinking skills and use of evidence to inform practices were commonly reported. Engaging with peers, staff and materials from diverse perspectives supported practitioners to “see the bigger picture of student engagement”, reflect on their own situation and “challenge my assumptions” (current learners). By coupling theory and knowledge with practitioners’ personal experiences, participants were able to acknowledge the “breadth of experience that I had working […] in higher education” (graduate). Greater “awareness of the needs of different groups of students” (graduate) and understanding of the influence of political and societal events on student engagement and HE were additional examples reported.
Research and evaluation skills were commonly developed, for instance, one practitioner reflected “I am so much more methodical, scholarly and convincing in the research that I do” (graduate). The course provided new insight into research project design, qualitative approaches and evaluation methodologies. Key areas of development reported by current learners and graduates were academic writing and enhanced self-confidence about succeeding in academic spaces, such as being able to “take part in some of the more theoretical conversations around what we are doing and why” (current learner). Practitioners also expressed greater confidence in collaborating with different staff and student groups: “l'm always thinking who do we need to bring in, who will have a concern, trying to think about those things early" (graduate).
Practitioners had immediate access to other individuals involved in student engagement. The cohort was described as a group that “all see value in student engagement” but with “different values and ways of approaching it” (current learner). Additional benefits of the group were the “community, friendship” and a “support network” (current learner), with one staff member recalling that “students have described becoming each other’s critical friend where they discuss work outside of the allocated class activities”. Communication and social media channels were regularly used by practitioners to ask their peers about topics, assessments, and to raise awareness of opportunities across the sector.
Legacy
Learning and knowledge acquired from the course were applied by practitioners to their practices and projects, such as in reviewing student partnership processes, creating new student voice mechanisms, and embedding co-creation with students within projects. Graduates also indicated that they continued to learn about, model and support the use of student engagement practices beyond the duration of the course. Throughout the course and after completion, practitioners reported they felt they had an “elevated status”, gained recognition for their knowledge of student engagement, and were better placed to “influence and challenge decisions” (graduates) within their profession. For instance, one participant described how they had become the “go to person for student engagement”, which was “a confidence boost […] and significant that people in my organisation ask about stuff that I’m involved in” (current learner).
Practitioners believed that the course helped them to address feelings of self-doubt when speaking with academic staff as they “feel more like an equal” (current learner). Another participant expanded on this point by stating that they have gained “more respect from colleagues towards our work because we can talk about it [topics] in academic terms” (current learner). Contact with peers on the course was commonly maintained after studying ended, which created further instances to support each other. The course was described as a “stepping stone to building relationships” (graduate) as practitioners were introduced to the networks of their peers.
Graduates provided examples of how they have disseminated the outcomes of work that they completed during and after the course, such as by publishing journal articles, presenting at conferences and writing internal reports. Those currently on the course aspired to publish in the future. The encouragement of the course on publishing and the support and feedback provided by the teaching staff were welcomed by those in professional services roles, where the opportunity to write and conduct research was “not your job” (graduate). Another key legacy was career progression, such as promotion or a change in job role or responsibilities, which was identified by many graduates and those currently on the course. These practitioners praised the course for instilling them with the “language to speak about all the work I’ve done” (graduate) and “the confidence […] to pursue that next [career] step” (graduate). The course helped to reaffirm some practitioners’ long-term career plans and it enabled others to achieve personal success and life goals, for example, achieving a postgraduate qualification.
Within the context of practitioners’ organisations of employment, perceived changes in attitudes, values and practices around student engagement were reported, although the challenge of attributing this directly to the course was acknowledged. Learning and completed assignments were often shared from the course with colleagues to help their development, and practitioners supported staff and student officers “with the theory and implications of different approaches to student engagement” (graduate). The positive impact of projects that practitioners led on were deemed to convince others about the value of student engagement, which subsequently helped to generate new resources and posts: “managers have now realised that they need someone to be doing it” (graduate). There were examples of staff and students working in partnership. For example, one practitioner recalled that the impact of a co-creation project involving students, which was designed and implemented within the context of the course, led to a “mindset change” within their organisation: “anytime we’re thinking of development somebody is [thinking] how do we get students to complement […] it demonstrated to anyone with doubts about what students could do” (graduate). However, a lack of funding and time within organisations impeded the implementation of the ideas of some practitioners.
The students located at practitioners’ organisations, or at different organisations across the sector, were identified as potential beneficiaries of the course. Among the examples provided, there were claims of “enhanced collaboration between staff and students” (graduate), “increases in the number of extra-curricular opportunities for students” (graduate), and the involvement of students in project design to avoid practices that may inadvertently disadvantage students. A perception among practitioners was that the influence of the course on students was indirect, with one graduate commenting that they “had a positive impact on the staff and student officers who do have that direct impact”, but others stated that the link to students was “too hard to measure” (graduate).
Discussion
This paper aimed to frame student engagement practitioners as an important community within and beyond HE providers and present evidence of an effective way of supporting this community to have impact in diverse HE settings. The discussion addresses the three research questions in detail.
What are the expectations and experiences of student engagement practitioners prior to enrolling on the course?
Practitioners on the course were connected through their mutual value and appreciation of student engagement as a way of supporting and enhancing the experiences of students in HE. The challenges facing some of this group prior to the course resonates with those identified in previous literature; for example, a perceived lack of recognition or misunderstandings about their roles or about student engagement (Akerman, 2020; Varwell, 2022). Practitioners evidently took responsibility for their own development (Nutt, 2022) but accessing relevant and sustained opportunities to further their knowledge and networks were very limited. Given the reputation of the course and the availability of the programme team, practitioners were convinced that the course offered the most appropriate route to enhance their practices and to achieve their own goals, which varied significantly across the cohort.
How effective are the processes underpinning the course at creating and sustaining a community of student engagement practitioners?
Reflecting on the barriers facing integrated practitioners, Akerman (2022) advocated for the creation of “bridges” to overcome these issues which comprise of “partnerships, […] mutual support from colleagues and from external networks, prospects to develop credibility through support for professional development, and leading development change projects” (p. 128). The course design subsequently focused on embedding learning within practitioners’ working lives and interests; creating opportunities to foster social interaction, trust and criticality; and embedding flexibility so that individuals were able to access and engage with the course. These are also consistent with the main characteristics of a community of practice and the mechanisms that make them effective (Beaton & Hope, 2022; Wenger-Traynor & Wenger-Traynor, 2015).
The course was developed on the idea that professional learning is most effective when it is located within the working lives and relationships of practitioners, which is consistent with Lave and Wenger’s (1991) situated learning theory. By drawing on the contexts of practitioners’ experiences and organisations of employment, the professional development was relevant, applicable, and effective at accounting for individual differences across the cohort (Inamorato dos Santos et al., 2019). This established an “identity defined by a shared domain of interest” for its members, in this case student engagement, which is a key characteristic of a community of practice (Wenger-Traynor & Wenger-Traynor, 2015, para. 6).
Acknowledging the role of others in knowledge construction and application (Lave & Wenger, 1991), mechanisms were embedded into the course to create an environment that allowed interactions to take place within and outside of the curriculum and across physical and virtual spaces. These exchanges, particularly those in the early stages of study and the on-campus retreats, were integral for building practitioners’ familiarity with their peers and the programme team, which helped to develop positive relationships and a sense of mattering among the cohort (Austen et al., 2021). Furthermore, practitioners’ interactions enabled them to become aware of different perspectives and reflect on and challenge their own assumptions and those of others (Pleschová et al., 2021), which corresponded with literature highlighting the importance of understanding context and disrupting thinking to enhance practices (Jones-Devitt, 2022). These points are also consistent with other features of a community of practice which involve creating “a social space”, relationships and a “blending of individual and collective learning in the development of a shared practice” (Wenger, Trayner & De Laat, 2011, p. 10). The accounts of practitioners engaging in collective learning and peer support demonstrate that the course successfully created an environment to foster social interaction, trust, and criticality.
Principles of flexible learning were implemented within the design and delivery of the course to facilitate practitioners’ engagement (Loon, 2022). The blended delivery approach meant that a diverse cohort of practitioners, in addition to teaching staff, from across the UK and beyond were able to access the course. Mixing professional services and academic roles within the course was perceived to be effective for learning (McIntosh & Nutt, 2022), while interacting with peers from different backgrounds has been identified as an important condition for instigating change in academic development (Pleschová et al., 2021). Practitioners accommodated their study alongside their work and life commitments, while the on-campus experiences were vital for developing a community. The course was also well positioned to minimise the disruption to teaching and learning caused by COVID-19.
What value does the course have on individual student engagement practitioners, their organisations and other groups?
Applying the 4M Framework (Friberg, 2016) to explore the value of the course within and beyond the context of an organisation, multiple types of value were reported for individuals at the micro-level. The knowledge and skills cited by practitioners, which corresponded with indicators outlined in sparq’s (2022) professional standards framework for student engagement (PSFSE), provided evidence that they were all equipped to reflect and think critically about student engagement upon completion of the course. The outcomes of practitioners resembled the characteristics of “positive disrupters”, who are “risk aware”, “find solutions” and challenge “positively institutional and sectoral policy and practice” (Akerman, 2022, p. 127). Reflecting its profound influence, the value of the course on other personal, professional, and social aspects of practitioners’ lives was recognised; for example, changes in confidence in academic spaces, personal validation, collaboration skills, and professional success (Inamorato dos Santos et al., 2019).
There was evidence of the course’s value at the meso and macro-levels, albeit to a lesser extent than the micro-level. Practitioners frequently perceived that their applications of learning from the course created avenues to influence the culture of student engagement within their local areas and organisations (Kneale et al., 2016), such as in values, attitudes and practices. These endeavours were deemed to have demonstrated the significance and increased the visibility of the work to organisations of employment (Webster, 2022) and were viewed as being indicative of a new or sustained focus on student engagement. There was recognition of the difficulty of attributing these outcomes directly to the course, but practitioners asserted that without the course the magnitude of changes achieved would have been minimal or non-existent. Several practitioners were not able, or not motivated, to instigate change at these levels though; for example, due to limited resources within their organisations or practitioners’ desire to only achieve individual goals. There were positive perceptions about the influence of the course on students who practitioners supported, although some stated this was indirect or too difficult to measure (Inamorato dos Santos et al., 2019).
A key example of how the course contributed to the mega-level was by enabling individuals to form a community of practice with others operating in student engagement spaces. Practitioners engaged in collective learning, peer support and collaboration on a shared domain of interest during and after studying to continue to learn from each other and to overcome challenges (Wenger-Traynor & Wenger-Traynor, 2015). Another legacy of the course was in practitioners’ dissemination of learning and publication of outputs, which contributes to efforts to influence practices of the wider community and HE sector (Marquis et al., 2020). The value of the course corresponded with the benefits reported for professional development programmes launched in other areas of integrated practice in HE. For example, providing the profession with a “clearer pathway of development” and a “united identity” that has “the potential to improve the standing of the profession externally and enhance the impact it is capable of having within institutions” (Webster, 2022, pp. 184-185) and organisations.
Implications for organisations and practitioners in higher education
This study offers transferable findings and practice to other postgraduate professional focused programmes, such as HE Learning and Teaching Postgraduate Certificates and other employment focused part-time courses. Although this study focused on one HE provider, there is learning for the broad and growing field of student engagement and arguments debating the concept of student engagement in neoliberal higher education contexts. Given the multiple stakeholder groups involved in student engagement, the findings of this course evaluation have highlighted the importance of supporting practitioners to critically reflect on practices and to navigate and work effectively within these potentially complex spaces. Equipping practitioners to design and implement well-designed, evidence-informed interventions can increase the likelihood of students’ engagement, learning and their experiences being enhanced and supported.
The findings of this evaluation of the PGCert/MA Student Engagement in Higher Education supports previous recommendations that relevant professional development programmes should be available for all HE professions due to the potential benefits for individuals, their organisations and the wider community (McIntosh & Nutt, 2022). Organisations and the HE sector should create and support access to these opportunities rather than rely on practitioners’ individual agency.
Conclusion
Evaluation in complex educational settings can be challenging. Whilst this evaluation has explored an innovative and robust approach to course evaluation which moves beyond the conventional (Bamber & Stefani, 2016), there are some notable limitations in the sampling and data gathering. It was only possible to collect primary data at one point in time using a ‘post-test’. There was no representation in the primary evidence collection from practitioners who started but withdrew from the PGCert/MA. A counterfactual perspective, which “identifies what would have happened if an intervention […] had not been implemented and compares this to the measured outcomes after the intervention” (Parsons, 2017, p. xiii), was subsequently missing. There was also the possibility that the participants who decided to take part in this evaluation were more likely to be those who have benefitted the most from the course.
The collection of self-report data, which has well documented limitations (Centre for Social Mobility, 2019), presents some potential issues relating to reliability and validity. The inclusion of other stakeholders within the evaluation, such as work colleagues of practitioners, was discussed but deemed to be beyond the scope of this study, which means that claims of impact are primarily based on the reports of practitioners on the course. However, triangulating the self-report data with observations from the programme team and other sources of evidence provides more confidence in the findings and conclusions. Finally, practitioners’ responses might have been influenced by the decision of the University of Winchester to close the course to new entrants, which was made halfway through the data collection period.
As evidenced in this methodology, the design and evaluation of professional development programmes should prioritise the perspectives of their students to ensure that there is correspondence between the programme and students’ experiences, expectations and resources (Parkinson et al., 2020). Furthermore, participatory approaches to evaluation, such as co-constructed ToC, can establish coherence between course design, outcomes and evidence collection (Amundsen & D’Amico, 2019) and generate greater understanding about programmes (Bovill & Woolmer, 2020). This evaluation also demonstrated the importance of capturing commonalities and variations in outcomes and accounting for multiple pathways of change and practitioners’ individual differences (Miller-Young & Poth, 2021).
This paper has demonstrated that the design and delivery of the course has supported the cultivation of a growing community whose members are connected by mutual values and relationships around student engagement, despite being dispersed across professional roles and geographical locations.
Acknowledgements
The authors of this paper gratefully thank the study’s participants and the advisory group.
Declaration of interest statement
The evaluators at Sheffield Hallam University were independent to the PGCert/MA course but were known to the programme team, who funded the evaluation, prior to this study. The methodology describes the efforts made to ensure that this evaluation presents a neutral, impartial perspective.
References
Akerman, K. (2020). Invisible imposter: Identity in institutions. Perspectives: Policy and Practice in Higher Education, 24(4), 126-130. https://doi.org/10.1080/13603108.2020.1734683
Akerman, K. (2022). Close encounters of the third kind. In E. McIntosh & D. Nutt (Eds.), The Impact of the Integrated Practitioner in Higher Education Studies in Third Space Professionalism (pp. 126-132). Routledge. https://doi.org/10.4324/9781003037569-15
Amundsen, C., & D’Amico, L. (2019). Using theory of change to evaluate socially-situated, inquiry-based academic professional development. Studies in Educational Evaluation, 61, 196-208. https://doi.org/10.1016/j.stueduc.2019.04.002
Austen, L., & Donnelly, A. (2023). “Trusted in being the experts of being a student”: Participatory evaluation in higher education. Journal of Participatory Research Methods, 4(2). https://doi.org/10.35844/001c.75240
Austen, L., Hodgson, R., Heaton, C., Pickering, N., & Dickinson, J. (2021). Access, retention, attainment and progression: An integrative review of demonstrable impact on student outcomes. Advance HE. https://www.advance-he.ac.uk/knowledge-hub/access-retention-attainment-and-progression-review-literature-2016-2021
Bamber, V., & Stefani, L. (2016). Taking up the challenge of evidencing value in educational development: From theory to practice. International Journal for Academic Development, 21(3), 242-254. https://doi.org/10.1080/1360144x.2015.1100112
Beaton, F., & Hope, J. (2022). Exploring the identities of blended professionals in higher education. In E. McIntosh & D. Nutt (Eds.), The impact of the integrated practitioner in higher education studies in third space professionalism (pp. 84-92). Routledge. https://doi.org/10.4324/9781003037569-11
Bovill, C., & Woolmer, C. (2020). Student engagement in evaluation: Expanding perspectives and ownership. In T. Lowe & Y. El Hakim (Eds.), A handbook for student engagement in higher education (pp. 83–96). https://doi.org/10.4324/9780429023033-7
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Centre for Social Mobility. (2019). Using standards of evidence to evaluate impact of outreach. Office for Students. https://www.officeforstudents.org.uk/media/f2424bc6-38d5-446c-881e-f4f54b73c2bc/using-standards-of-evidence-to-evaluate-impact-of-outreach.pdf
Davies R., & Dart, J. (2005). The ‘Most Significant Change’ (MSC) technique – A guide to its use. https://www.mande.co.uk/wp-content/uploads/2005/MSCGuide.pdf
Donnelly, A., & Austen, L. (2022). A process and impact evaluation of the PGCert and MA Student Engagement in Higher Education programme. Sheffield Hallam University Evaluation Repository. https://shura.shu.ac.uk/id/eprint/30905
Friberg, J. (2016, July 11). Might the 4M framework support SoTL advocacy? The SOTL Advocate. https://illinoisstateuniversitysotl.wordpress.com/2016/07/11/might-the-4m-framework-support-sotl-advocacy/
Heron, E. (2020). Friendship as method: Reflections on a new approach to understanding student experiences in higher education. Journal of Further and Higher Education, 44(3), 393-407. https://doi.org/10.1080/0309877x.2018.1541977
Inamorato dos Santos, A., Gausas, S., Mackeviciute, R., Jotautyte, A., & Martinaitis, Z. (2019). Innovating professional development in higher education: Case studies. Publications Office of the European Union. https://publications.jrc.ec.europa.eu/repository/handle/JRC115595
Jones-Devitt, S. (2022). The viral landscapes of third space working: Debating and challenging the power dynamic of the new “blended” approach. In E. McIntosh & D. Nutt (Eds.), The impact of the integrated practitioner in higher education studies in third space professionalism (pp. 63-75). Routledge. https://doi.org/10.4324/9781003037569-8
Jones-Devitt, S., Austen, L., & Parkin, H. (2017). Integrative reviewing for exploring complex phenomena. Social Research Update, (66).
Kneale, P., Winter, J., Turner, R., Spowart, L., & Muneer, R. (2016). Evaluating teaching development activities in higher education - A toolkit. Higher Education Academy. https://s3.eu-west-2.amazonaws.com/assets.creode.advancehe-document-manager/documents/hea/private/evaluating_teaching_development_in_he_-_toolkit1_1568037347.pdf
Krueger, R. A. (2010). Using stories in evaluation. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (3rd ed., pp. 404–423). Wiley. https://doi.org/10.1002/9781119171386.ch21
Lam, S. (2020). Toward learning from change pathways: reviewing theory of change and its discontents. Canadian Journal of Program Evaluation, 35(2). https://doi.org/10.3138/cjpe.69535
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press. https://doi.org/10.1017/CBO9780511815355
Loon, M. (2022). Flexible learning: A literature review 2016 - 2021. Advance HE. https://www.advance-he.ac.uk/knowledge-hub/flexible-learning-literature-review-2016-2021
Lowe, T. (2023). Advancing student engagement in higher education: Reflection, critique and challenge. Routledge. https://doi.org/10.4324/9781003271789
Lowe, T. & Bols, A. (2020). Higher education institutions and policy makers: The future of student engagement. In T. Lowe & Y. El Hakim (Eds.), A handbook for student engagement in higher education (pp. 267-284). Routledge. https://doi.org/10.4324/9780429023033-26
Lowe, T. & El Hakim, Y. (2020). An introduction to student engagement in higher education. In T. Lowe & Y. El Hakim (Eds.), A handbook of student engagement in higher education (pp. 3-26). Routledge. https://doi.org/10.4324/9780429023033-2
Lygo-Baker, S., Kinchin, I. M., & Winstone, N. E. (eds.). (2019). Engaging student voices in higher education: Diverse perspectives and expectations in partnership. Springer. https://doi.org/10.1007/978-3-030-20824-0
Marquis, E., Black. C., Guitman, R., Healey, M., & Woolmer, C. (2020). From the ‘micro’ to the ‘mega’: Toward a multi-level approach to supporting and assessing student-staff partnership. In T. Lowe & Y. El Hakim (Eds.), A handbook for student engagement in higher education (pp. 110-121). Routledge. https://doi.org/10.4324/9780429023033-10
McIntosh, E., & Nutt. D. (Eds.). (2022). The impact of the integrated practitioner in higher education: studies in third space professionalism. Routledge. https://doi.org/10.4324/9781003037569
Miller-Young, J., & Poth, C. N. (2021). ‘Complexifying’ our approach to evaluating educational development outcomes: Bridging theoretical innovations with frontline practice. International Journal for Academic Development, 27(4), 386-399. https://doi.org/10.1080/1360144x.2021.1887876
Moxey, M., Lowe, T., Bovill, C., Bryson, C., Neary, M., LeBihan, J., Islam, M., Green, P. & Marie, J. (2022). The Postgraduate Certificate & Master's in Student Engagement in Higher Education: A professional development opportunity to critically examine and influence research, policy and practice. All Ireland Journal of Higher Education, 14(1), 1-13.
Nutt, D. (2022). Rewarding blended working and integrated practice. In E. McIntosh & D. Nutt (Eds.), The impact of the integrated practitioner in higher education studies in third space professionalism (pp. 252-264). Routledge. https://doi.org/10.4324/9781003037569-31
Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 42, 533-544. https://doi.org/10.1007/s10488-013-0528-y
Parkinson, T., McDonald, K., & Quinlan, K. M. (2020). Reconceptualising academic development as community development: lessons from working with Syrian academics in exile. Higher Education, 79, 183-201. https://doi.org/10.1007/s10734-019-00404-5
Parsons, D. (2017). Demystifying evaluation: Practical approaches for researchers and users. Policy Press. https://doi.org/10.46692/9781447333913
Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks
Pleschová, G., Roxå, T., Thomson, K. E., & Felten, P. (2021). Conversations that make meaningful change in teaching, teachers, and academic development. International Journal for Academic Development, 26(3), 201-209. https://doi.org/10.1080/1360144x.2021.1958446
Rutten, L. (2021). Toward a theory of action for practitioner inquiry as professional development in preservice teacher education. Teaching and Teacher Education, 97, 103194. https://doi.org/10.1016/j.tate.2020.103194
Saunders, M., Trowler, P., & Bamber, V. (2011). The practice turn: Reconceptualising evaluation in higher education. In M. Saunders, P. Trowler & V. Bamber (Eds.), Reconceptualising evaluation in higher education (pp. 203-226). Open University Press.
Simmons, N. (2020). The 4M framework as analytic lens for SoTL’s impact: A study of seven scholars. Teaching and Learning Inquiry, 8(1), 76-90. https://doi.org/10.20343/teachlearninqu.8.1.6
sparqs. (2022). Professional standards framework for student engagement. https://www.sparqs.ac.uk/upfiles/Professional%20standards%20framework%20for%20student%20engagement.pdf
Tight, M. (2020). Student retention and engagement in higher education. Journal of Further and Higher Education, 44(5), 689-704. https://doi.org/10.1080/0309877x.2019.1576860
Varwell, S. (2022, July 1). A professional standards framework for student engagement. WonkHE. https://wonkhe.com/blogs/a-professional-standards-framework-for-student-engagement/
Webster, H. (2022). Supporting the development, recognition, and impact of third space professionals. In E. McIntosh & D. Nutt (Eds.), The impact of the integrated practitioner in higher education studies in third space professionalism (pp. 178-187). Routledge. https://doi.org/10.4324/9781003037569-22
Wenger, E., Trayner, B., & De Laat, M. (2011). Promoting and assessing value creation in communities and networks: A conceptual framework. Ruud de Moor Centrum. https://www.researchgate.net/profile/Maarten-Laat/publication/220040553_Promoting_and_Assessing_Value_Creation_in_Communities_and_Networks_A_Conceptual_Framework/links/0046353536fa177004000000/Promoting-and-Assessing-Value-Creation-in-Communities-and-Networks-A-Conceptual-Framework.pdf
Wenger-Trayner, E., & Wenger-Trayner, B. (2015, June). Introduction to communities of practice. Wenger-Trayner. https://www.wenger-trayner.com/introduction-to-communities-of-practice/