Using Scottish school inspection documentation as a primary source for research dissertations
Stephen C. Scholes1
1Queen Margaret University, Edinburgh, UK
Documentary research in education can often be overlooked as an approach to exploring policy and practice issues, particularly for dissertation research projects. However, a range of documents can offer valuable data. In particular, school inspection documentation can be an illuminating and accessible source. Building on my recent research and teaching experiences, this paper provides a guide to utilising school inspection documentation from the Scottish context as primary sources for research dissertations in education and other disciplines. The intention is to support students and researchers undertaking research projects to navigate these sources confidently and to enable them to develop insightful analyses and discussions.
documentary research, school inspection, Scottish education,
dissertations
As a historian of education and qualitative researcher interested in school inspection, I have been increasingly fascinated by the insights into educational policy and practice issues that can be gleaned from school inspection documentation. For example, for a recent special issue of the British Journal of Religious Education, I analysed the inspection documentation of 85 schools to analyse the interplay between the law, inspection, and religious education in Scottish non-denominational secondary schools (Scholes, 2022). Moreover, since 2019, several of my students have worked on research topics with Scottish school inspection documentation as a key primary source. Students have successfully studied topics as varied as tracking and monitoring, religious education in Catholic primary schools and the spending of the Pupil Equity Fund. Having recognised the potential for school inspection documentation through research and teaching, I am keen to promote the use of such material further and help researchers to navigate it.
School inspection documentation, as used here, is a catchall term to describe the various outputs produced and made public by education inspectorates following a site visit to a school. The inspection report is the most common example across UK inspectorates. These documents summarise inspectors’ findings, detail their evaluations of a school’s work and, often, note the next steps for improvement (Altrichter & Kemethofer, 2015). However, additional material can be published alongside such reports, and where this is the case, I extend the term to include such items. In Scotland, this consists of an extended report of inspection findings and questionnaire responses. Such documentation is increasingly published online; in the case of Scotland, this is via a web-based, freely accessible, searchable database.
In what follows, I provide a guide to utilising school inspection documentation as a primary source for research projects. The paper will focus on material from Scotland, but many of the points raised can be considered with regard to other education systems. For example, the inspectorates of England (OFSTED, n.d.), Wales (Estyn, n.d) and Northern Ireland (Education and Training Inspectorate, n.d.) all maintain their own online databases of inspection documentation. The discussion focuses on school inspection documentation published since August 2016, as the approach to inspection currently in use in Scotland started then. In addition, the website that hosts the available documentation typically only holds documentation published in the most recent five-year period.
The first section of the paper will explain how school inspection documentation can be considered with respect to documentary research in education. The paper’s second section outlines how to find and navigate the school inspection documentation from Scotland. After that, the paper considers key analytical angles for researchers to consider when working with inspection documentation and notes important pitfalls they should be aware of. To conclude, the final section will consider further suggestions on the possibilities of documentary research with school inspection documentation.
Documentary research is a research approach within the social sciences that uses documents and data from documents to answer researchers’ questions. (McCulloch, 2004; Prior, 2003; Tight, 2019). Documentary research can be the only approach deployed in a research project or could be used as one of several other methods to enable the researcher to consider the issue from different evidence bases and triangulate their findings (Bowen, 2009). Assuming that there is sufficient data available from the inspection documentation, however, there is nothing to stop a project that only deploys documentary research from being a productive undertaking.
Definitions of a document can be restrictive or inclusive, with most commentators opting to broaden their criteria to include a full range of written or audio-visual items from digital and offline contexts. As Merriam and Tisdell (2016, p. 162) put it, the term document “is often used as an umbrella term to refer to a wide range of written, visual, digital, and physical material relevant to the study”. Flick (2014) rightly notes that documents might be generated as part of a research project or be items that already exist independently of the project. In the case of school inspection documentation, the documents already exist, were created for purposes other than the researcher’s project and may or may not be or contain data relevant to the chosen research questions (McCulloch, 2011). Following Bowen (2009, p. 29), it should be recognised here that where inspection documentation is the only data source for a project it will sit most comfortably “for studies designed within an interpretive paradigm”. Given that the existence of the documentation is a result of multiple processes and interactions that researchers must take account of and consider in relation to any relevant data, it is important too to emphasise that inspection documentation demands careful interpretation.
Documents can be further described in various ways to categorise them for research (Flick, 2014; Scott, 1990). Concerning inspection documentation, two key points of distinction should be considered here. First, it is important to recognise inspection documentation as a primary source. Such terminology might seem more at home in historical studies, but a loose distinction between primary and secondary sources is helpful in documentary research (Scott, 1990). Considering inspection documentation as a primary source encourages us to see the inspection documentation as distinct from other written material that will be engaged with for research projects, namely published research, and other relevant documents, such as policy texts. As Bowen (2009) notes, in documentary research, it is important to be clear about what we consider as our sources of data that will be subjected to analysis. In short, it is important to be specific about what we will use to answer our research questions. Therefore, considering the inspection documentation as a primary source enables us to distinguish it from the secondary material that provides the background and scholarly commentary that our research projects seek to engage with. However, it should be noted here that, where appropriate to the research questions, there is nothing to stop us from using policy texts or scholarship as primary source material in documentary studies. The important thing would be to make this clear. For example, in my recent study that used school inspection documentation, I frequently referenced reports by the inspectorate that offered national overviews of Religious Education and curriculum documents to provide background. These, however, were not systematically analysed to answer the core questions I had in the article (Scholes, 2022). For my ongoing PhD research, however, these same national reports are a core primary source and have been subjected to rigorous analysis. These specific choices need to be explained and justified appropriately in the write-up stage of the projects.
Scholars who engage in documentary research have articulated a range of questions we should ask about the documents we use in our projects and see these as a vital step in the research approach (Bowen, 2009; McCulloch, 2011 & 2004; Prior, 2003). Most of these questions centre on ensuring that the documents under scrutiny are not fake, contain relevant and reliable material and are considered carefully with respect to the realities of the insights they offer. Morgan (2021, p. 70) tackles these points by considering the “factors” of authenticity, credibility, representativeness and meaning. To explain,
Authenticity involves the extent to which a document is genuine […] Credibility relates to the extent to which the source is free from errors and distortion […] Representativeness has to do with how typical a document is […] Meaning involves the significance of a document’s content and pertains to whether the evidence is clear and understandable […] (Morgan, 2021, pp. 71-72).
When thinking about these factors, it is helpful to consider whether documents are public or private and official or unofficial (McCulloch, 2004). These are best thought of as loose categorisations; in the case of inspection documentation, we deal with official documents that have been made public. They are official in that they are produced by civil servants, namely inspectors, working for a government agency. They are public in that they are made available by the inspectorate via Education Scotland’s (n.d.) online database. The officialness of these documents helps to ascertain that such documentation is authentic and credible. Their public nature enables comparisons across the documentation to see that individual documents are representative. As the inspectorate generates the documentation through the work of inspectors concerned with school provision, what they are and what they contain is likely to be relevant and meaningful to relevant research projects concerned with education-related topics. Here, of course, it is essential to conduct preliminary surveys, as discussed in more detail below, to ascertain if there will be sufficient data for the demands of the research project.
Finally, it is important to recognise that documentary research can consider documents at two basic levels. First, documents themselves can be seen as evidence of some process, event, happening or something else. Second, the documents’ text offers data relevant to answering research questions (Rapley, 2007). For example, school inspection documentation is evidence of the processes that occur and lead to the production of such documentation and who is involved in such endeavours. Equally, the text within the documentation could provide insights into specific aspects of school provision. For example, if a report discussed a particularly effective interdisciplinary learning project, we could use this as evidence of the nature of interdisciplinary learning occurring within schools. In short, inspection documentation is evidence of and contains insights into other educational processes, practices, and issues. However, it is important to note that these will be limited by their context-specific, snap-shot nature.
School inspection in Scotland has been the responsibility of His (previously Her) Majesty’s Inspectorate of Education (HMIE) since 1840 (Bone, 1968). Under current governance arrangements, since July 2011, HMIE has operated within and through Education Scotland, the Scottish Government’s “national improvement agency for education in Scotland” Education Scotland, 2022a: 7; Hutchison, 2018). The inspectorate’s role in the education system is complex, but the focus here is on their work inspecting schools in Scotland and the resulting publications.
Researchers looking to work with inspection documents must be clear about what aspect of schooling they are focusing on, as the schooling provision covered by the inspectorate’s remit is varied. School inspections cover nursery (ages 3-5), primary (ages 5-11), secondary (ages 12-18) and special school (all ages) provision in both the state-funded and private sectors in Scotland. Within Scotland, state-funded school-type is also further differentiated by religious affiliation. Denominational schools are usually Roman Catholic in Scotland, but there are three Episcopalian primary schools and one Jewish. Non-denominational schools do not have an explicit religious affiliation (Scholes, 2020). Researchers will also note that the term 'inter-denominational' is used for a small number of special schools in Glasgow (Scottish Government 2022b). The inspectorate inspects all of these types of schools and usually records the school type in the documentation they produce.
To support researchers in navigating the school types in Scotland, they should use the Scottish Government’s (2022a & 2022b) national overview lists of school contacts[1]. Moreover, they should consider where such lists can aid initial searching and categorisation. For example, the lists noted above record the local authority responsible for the school. This is significant, as in Scotland, all state-funded schools (other than one that the government directly funds) are run by one of thirty-two local education authorities (McGinley, 2018). Such information could help select geographical areas for study or comparisons between areas later in the research project. Where researchers are engaged in comparative studies across countries, attention should be paid to the set-up of the respective schooling systems. Due consideration should be given to school-type and governance arrangements.
Approaches to inspection in Scotland have changed over time and are an aspect of educational history much in need of further study (Bone, 1968; Hutchison, 2018; Ozga et al., 2015). Since 2016, and as relevant to the material that researchers can easily access, the workings of inspection need to be understood in light of two key documents. First, How Good Is Our School? (4th edition) (HGIOS4), published in September 2015 by Education Scotland (2015). Second, a letter issued to all local authority Directors of Education and Headteachers in Scotland in June 2016 from Education Scotland’s then-Chief Executive, Bill Maxwell (Education Scotland, 2016).
HGIOS4 is the latest edition of a series of frameworks, beginning in 1996, that contain a series of ‘quality indicators’ (QIs) against which schools are expected to self-evaluate and that the inspectorate uses in their scrutiny of schools’ work. It is essential to recognise that HGIOS4 is the guiding framework for how the inspectorate inspects, what they focus on and what information makes it into inspection documentation (Baxter et al., 2015). HGIOS4 is made up of fifteen QIs that comprise two or three sub-themes. The QIs are divided across the three ‘categories’ of ‘leadership and management’, ‘learning provision’ and ‘successes and achievements’. It is important to pay attention to the QIs as these are the aspects of HGIOS4 that the inspection documentation relates most directly to. This focus can also help the researcher minimise the chances that they ‘spend many hours, days or weeks studying them with little result’, as they are better informed as to what the inspection documentation may offer them and can search it more strategically (McCulloch, 2004, p. 26). The table below details the fifteen QIs, and researchers are encouraged to review the descriptions of each of these in HGIOS4 as a guide to what topics may be viable for study.
Table 1. HGIOS 4 quality indicators. Adapted from HGIOS 4 (Education Scotland, 2015, p. 15)
|
Categories |
Quality Indicators |
|
Leadership and Management |
1.1 Self-evaluation for self-improvement 1.2 Leadership of learning 1.3 Leadership of change 1.4 Leadership and management of staff 1.5 Management of resource to promote equity |
|
Learning Provision |
2.1 Safeguarding and child protection 2.2 Curriculum 2.3 Learning, teaching and assessment 2.4 Personalised support 2.5 Family learning 2.6 Transitions 2.7 Partnerships |
|
Successes and Achievements |
3.1 Ensuring wellbeing, equality and inclusion 3.2 Raising attainment and achievement 3.3 Increasing creativity and employability |
The significance of HGIOS4 to the content of the documentation can be demonstrated through the findings of another project I worked on, this time focusing on Religious Education in non-denominational primary schools (Scholes, 2021). Whilst a range of topics might be explored in Religious Education, it is evident from the comments on offer in recent inspection reports that a descriptor of ‘highly effective practice’ from HGIOS4 was shaping what inspectors were focusing on and reporting on. The descriptor for an aspect of the quality indicator ‘3.1 Ensuring Wellbeing, Equality and Inclusion’ notes that effective practice in this area would see that “the curriculum provides children and young people with well-planned and progressive opportunities to explore diversity and multi-faith issues, and to challenge racism and religious intolerance” (Education Scotland, 2015, p. 49). And, in connection to this, inspectors’ comments in reports regarding Religious Education focused on these areas about the subject area. An example from a report on a primary school illustrates this well:
The school has a religious education programme which includes discussions on equality, race and religious understanding. The rolling programme within religious and moral education (RME) is planned well to include opportunities to explore diversity and faith. The school understands it now needs to be more focussed in ensuring these areas are covered in a more methodical way throughout the school session (Education Scotland, 2017, p 10).
In the above example, the areas commented on relate to elements of what the inspectorate articulates as good practice, as illustrated in HGIOS4. This dynamic relationship between inspection documentation and HGIOS4 needs to be continually considered in any research project using Scottish school inspection documentation.
The second key document that provides a point of orientation for research on school inspection documentation is a letter from June 2016 that outlined the inspection approach that would be deployed in schools from August 2016 onwards. The Education Scotland (2016) letter explained that there would be three areas of significant change (Hutchison, 2018). First, schools would either be inspected via a full-model week-long inspection or a short-model inspection that would last a couple of days. Second, the range of QIs that would be inspected was clarified. All full-model inspections would cover the following QIs: ‘1.3 Leadership of change’, ‘2.3 Learning, teaching and assessment’, ‘3.2 Raising attainment and achievement’ and ‘3.1 Ensuring wellbeing, equality and inclusion’. Each of these would be formally evaluated by the inspectorate as either Excellent, Very Good, Good, Satisfactory, Weak or Unsatisfactory. Thereafter, other relevant themes from various QIs would also be incorporated into inspections to “capture the evidence’ required for ‘improvement and […] to inform national policy development” (Education Scotland, 2016, p. 2). In addition, schools would select an additional quality indicator for further feedback, but it would not be formally evaluated. Short-model inspections would only focus on two quality indicators; since August 2018, these have been quality indicators 2.3 and 3.2 (Education Scotland, 2018). Third, the format of the documentation, as described in more detail below, would change to “better communicate inspection findings to parents” and these reports, along with “the inspection evidence” would be published online (Education Scotland, 2016, p. 2).
The significance of these three points, particularly the first two, is that they should support researchers in navigating the construction of the inspection documents, which are structured by the above-noted quality indicators. In addition, depending on the research question being pursued, focusing on the quality indicators can be a useful starting point for comparing where relevant comments occur in the documentation. For example, the researcher could ask, are comments relating to a specific area of the curriculum only made with reference to a particular QI, or is the distribution more varied and is such distribution potentially significant? It should be noted too that the selection of a QI by schools may also influence what appears in the reports and that for the self-selected QIs local priorities will shape the content of the documentation.
The school inspection documentation currently accessible online for a particular school will normally have been published in the previous five years from the month in which the database is searched. Typically, three main documents will be issued for each school that has been inspected. For each school, there will be an Inspection Report (IR). The IR is a letter-style communication addressed to parents and carers directly, and it provides a summary of the main strengths and areas for development from the inspection, the evaluations of the quality indicators in a table, and a note to clarify if the inspectorate will or will not be engaged with any follow-up activity to monitor improvement. The second document will be the Summarised Inspection Findings (SIF) report, organised by quality indicators. This is the record of evidence from the inspection and is the most detailed of all the inspection documentation. SIFs are, therefore, the most lucrative with respect to the data they contain. The final document is the Evidence Report (ER). These include a quantitative summary of the responses to the questionnaires issued to parents, staff, and children as part of the inspection process.
In addition to these standard documents, some schools may also have a Continuing Engagement Report (CER) or a Follow-Up Report (FUR). These items are reports on a school’s progress against the next steps detailed in an original IR. Where schools have a CER or a FUR, the initial IR may be available and may date from beyond the most recent five-year period.
The documents are all usually published online at the same time. Researchers should be mindful that whilst the materials have a publication date, the inspection will have taken place several weeks previously. This time difference should also alert researchers to the fact that a lot may change in a particular school between the inspection, the publication of the documentation, and the researcher exploring the data. This is undoubtedly a limitation of the inspection documentation as a primary source but needs only to be acknowledged explicitly, recognising that the inspection reports, much like an oral account or an observation, offer only snapshots of practice and life in schools.
Inspections in Scotland ceased in mid-2020 because of the pandemic, with the final publication of documentation from inspections occurring in August 2020. Over the academic year since early Spring 2022, the inspectorate has started to carry out ‘recovery visits’ focused on support and not on evaluations against quality indicators (Education Scotland, 2021). These ‘recovery visits’ have resulted in an additional document known as a Visit Letter (VL), which is presented as a letter addressed to parents and careers. The contents are focused on the school’s work during the pandemic, progress since the school’s last inspection and a statement on the next steps for the school concerning future inspectorate involvement. These VLs offer a fruitful source of data on schools’ work during the pandemic and deserve further attention from researchers. Standard inspections resumed as normal in August 2022. Ahead of this, schools were informed that the inspectorate “recognises that the pandemic has had an impact on all” and it “will continue to be proportionate and responsive” in its approach to gathering evidence during inspections (Education Scotland, 2022b, p. 1). For research projects, new school inspection documentation has, therefore, become available from late-2022 onwards.
As indicated above, Scottish school inspection documentation is freely available and accessible via Education Scotland’s (n.d.) online searchable database. This can be found via the link in the reference list. Inspection documentation is organised in the database by school. Usually, as noted, only schools that have been inspected within the last five years will have material available, except where follow-up activity leads to older material remaining on the system. Making use of such documentation has the advantage of being easily accessible. If the researcher saves and backs-up offline copies of the relevant documentation, there is minimal risk in terms of access to data (Merriam & Tisdell, 2016).
The interface for the database on the ‘School reports’ tab offers several options to navigate the material. Documentation can be searched for by ‘school name’, all schools within a ‘local authority area’ can be called up, it is possible too to look specifically at secondary, primary, special, nursey or/and Gaelic schools by ‘school type’, and to select schools by their ‘funding type’. The ‘local authority area’, ‘school type’ and ‘funding type’ filters can be selected in various combinations from their respective drop-down boxes. A date range can be searched on the ‘Reports by date’ tab, and an individual ‘local authority area’ can be selected too. Awareness of the filtering functionality should help researchers to narrow down their selection. However, it should be noted that the filters are limited, for example, in their ability to combine date ranges with school type. Therefore, the filters applied will normally be ineffective, and much of the searching will be done manually. Indeed, regardless of the filters applied, researchers will have to select each result individually to check whether individual schools within the filtered results have documentation available or not. On this point, it is worth noting that researchers are advised to right-click on the name of the school they want to select and opt to open the result in a new tab or window. This will enable them to keep the original list of filtered schools as they search through. Selecting the school and then the ‘back’ option will return the researcher to the site’s homepage, and the researcher would have to start from the beginning each time.
As Tight (2019, p. 23) notes, “documentary research is necessarily constrained […] by the documents that are available and accessible”. Therefore, at the searching stage, it is equally important to consider how the documentation will be collated and an overview of the findings generated. This will ensure that the researcher has a clear picture of what is available to them. As mentioned above, lists of all possible schools are valuable tools for keeping track of available documentation. For example, if the research is focused on all Roman Catholic secondary schools in Scotland, having a complete list of all such schools would support a systematic search of the inspection documentation online database. As illustrated in Table 1, constructed from made-up data, such an approach can provide an overview of the total number of schools, the number of schools inspected within the particular period selected for study, and the number of schools with inspection documentation containing something of interest to the research project.
Table 2. Example search table. The table uses made-up examples of schools and local authorities.
|
School name |
Local authority |
Denominational (RC) |
Document-ation available |
Document-ation published between 01.08.18 and 31.08.20 |
Document-ation contains relevant data |
|
St Andrews High School |
West Ayrshire |
YES |
YES |
YES |
YES |
|
St Benedict’s High School |
Lowlands |
YES |
YES |
NO |
NO |
|
St Catherine’s High School |
Lowlands |
YES |
NO |
NO |
NO |
|
St Dominic’s Academy |
North Seaside |
YES |
YES |
YES |
NO |
|
St Mary’s High School |
Hill City |
YES |
YES |
YES |
YES |
From the made-up data in Table 1, it is already possible to highlight that of the five schools in the sample, only four have inspection documentation available, with three schools having material that falls within the selected date range and, of these, only two contain comments relevant to the research questions. Such an overview can help provide insights into how widespread the inspectorate’s activities are concerning particular school types and topics. Moreover, the focus on local authorities enables some discussion about the dynamics of the issue under investigation at a national level or, at least, in comparison to other chosen authorities or those with available documentation. Such an overview can also be used to collate relevant data from the documentation and conduct further specific analyses.
The inspection documentation available for research can, in the first instance, be viewed as data itself. The existence of IRs, SIFs, ERs, and the other items available as inspection documentation helps us to explore and understand the inspectorate’s role in the education system and the inspection process. Researchers who seek to understand these aspects can use the documentation in several ways to answer a range of questions.
By adopting the approach detailed in Table 1, researchers can investigate and offer insights into the inspectorate’s activity in relation to school inspection across the education system. For example, they can detail the number of schools inspected by sector and consider that number as a percentage of the total number of schools. In such an investigation, it would be important to think carefully about the chosen date range, schools that may have been inspected before it and any schools that had a continuing engagement with the inspectorate. Such considerations move us beyond simple claims about the percentage of schools inspected in a particular period and recognise the complexity of the inspectorate’s involvement with schools in Scotland. The issue of date range also highlights that the available documentation does limit the ability of researchers to comment on longer term trends connected to particular topics to within a five-year timeframe.
The inspection documentation also offers some insights into how the inspectorate engages with school communities during inspections and the nature of the communities’ engagement with the inspection process. The ERs, for example, provide insights into what aspect of schooling the inspectorate considers significant for various groups within the school communities, namely pupils, support staff, teachers, and parents/carers. Equally, the response rates could suggest the degree of engagement with the inspectorate where response numbers are provided, and original population numbers, such as school roll or staffing complement, can be established. It should be noted here that the inspection documentation is limited as a data source with respect to the range of stakeholders’ views that are captured. The ERs provide direct reports from key groups, such as parents, but the other documentation only communicates stakeholders’ views through the inspectorate’s words and judgements.
Inspectorates across Europe report in different ways, and not all issue public reports, like those published in Scotland. Ehren et al. (2013, p. 10) contend that public reports are “expected to promote a ‘market mechanism’ where improvement of schools is motivated through informed school choice and the voice of parents”. The inspection documentation means that researchers can consider whether the inspectorate’s priorities are in sync with the idea of a ‘market mechanism’ or explore the other priorities that might be at play. For example, the ‘voice of parents’ is evident in the IRs, but the ERs focus on a more diverse audience, namely school staff and the pupils themselves, in the Scottish context. In short, the inspection documents are artefacts of processes and the enacting of priorities that the researcher can explore.
The contents of inspection documentation, chiefly the text contained within them, can be considered valuable data on a range of in-school practices and the inspectorate’s ways of working. The inspection documents, particularly the SIFs, contain the accounts of what inspectors experienced during site visits and offer researchers accounts of what is happening in schools. They are also replete with comments that provide insights into how inspectors judge and engage with school communities.
Three important points need to be kept in mind when thinking about such accounts. First, the accounts are restricted by being accounts of happenings at a particular time, and things can and very often do change in schools over time (Perryman, 2009, 2010; Scholes, 2022). Second, the accounts are those of inspectors, and researchers must do what they can to disaggregate inspectors’ judgements and the practice being recounted. Indeed, as explored above, such judgements are made against the backdrop of HGIOS4. Third, such accounts result from editing activities and a “quality assurance process” that will have shaped the final outputs. For example, the IR is produced in a draft form first, and the headteacher, relevant local authority personnel, and chair of the parent council of the school are then “asked to provide any comments or suggested amendments” (Education Scotland, 2018, p. 8). Therefore, whilst such accounts are valuable, they must be analysed carefully and understood as constructed.
Looking at the text contained within any single IR or SIF, with the latter of these offering the researcher richer pickings, researchers can gain insights into what is happening in schools. There can be a range of topics covered, including developments in leadership and management processes, discussions of curriculum offerings, examples of approaches to learning and teaching, notes of extracurricular offerings, and examples of how schools engage with parents, families and the wider school community, such as local business, charities, colleges, and other partners. Researchers should interrogate the quality indicators and HGIOS 4 for potential themes and topics. An individual instance of some element of school practice appearing in a single IR or SIF is likely to be of limited use to the researcher (Tight, 2019). In cases where inspection documentation is part of a broader case study on one particular setting, it could be valuable data for triangulating findings from other evidence (Bowen, 2009; Simons, 2020). However, where inspection documentation is the sole data source, the researcher would need to gather further examples from the inspection documentation from other schools before they could offer any well-informed comment on their particular area of study. Ascertaining the availability and extent of data available for analysis on any one particular topic is not immediately knowable. Rather, researchers will have to engage in sampling batches of documentation.
The decision about what would make for sufficient data from inspection documentation depends on the focus of the researcher’s project. With the Scottish material, researchers could focus on one local authority, one school type, or perhaps look at different regions. Again, as noted above, well-organised and systematic surveying of what documentation is available and careful and methodical recording (as illustrated in Table 1) is needed here. An important point at the beginning of a study that utilises the Scottish inspection documentation is to be clear on the selection of schools to be reviewed in terms of school type and geographical spread. Researchers should be mindful that selecting only one local authority may only provide a limited amount of relevant data. Therefore, if the study is region-specific, it would be important to consider other types of documentation, such as school handbooks and improvement plans. Alternatively, it may be the case that the researcher needs to broaden the geographical parameters of the study if the intention is to focus specifically on inspection documentation.
The researcher should also consider the chronological dimension of the data extracted from the inspection documentation. As noted, the reports and other documentation speak to a particular window of time, and it is important to be mindful of change over time. For example, it is possible to notice time-specific differences across the data across a selection of comments relating to one specific theme. As I noted in my recent article, the language used by the inspectorate to describe the legislative requirements surrounding Religious Education noticeably changed between 2016 and 2020 (Scholes, 2022). In this instance, the research focus was less on practice found in schools and more concerned with the inspection process.
Inspection is a dynamic and complex process, and the data within the inspection documentation offers insights into how inspections operate and what inspectors do in connection with a site visit. The range of evidence that inspectors engage with during visits, such as school attainment data and pupil work, are captured in inspection documentation. There are notes of the conversations had with groups within the school. The documentation also provides insights into the inspectorate’s stances and judgements on key issues of policy and curriculum enactment, leadership, and school processes. With a careful reading of the texts, researchers can explore such topics via the documentation. Here, an example from a primary school is instructive. The report notes that:
In listening and talking most children achieve early level by the end of P1. Almost all children are achieving first level by the end of P4 but less than half achieve second level in listening and talking by the end of P7. These figures from 2015-2016 are not consistent with evidence observed by inspectors. They are also inconsistent with the data from CEM tests. The headteacher and staff will revisit their assessment, tracking, monitoring and moderation procedures immediately to ensure they are robust in future (Education Scotland, 2017, p. 11, emphasises added).
In the italicised section, there is evidence of what the inspectors did during this inspection, namely that they reviewed school attainment data and compared this to other data sources, with CEM being a standardised assessment in this context. Equally, the text in italics also contains evidence of the ‘directive’ feedback inspectors provided to the school. As Ehren & Visscher (2008, pp. 211-212) explain ‘directive’ feedback in the context of inspections is feedback whereby “an inspector clearly points out the strengths and weaknesses of the school, the probable causes of its level of functioning, and potential improvements”. Exploring such ‘directive’ feedback, as contained in the inspection documentation, can offer the researcher opportunities to examine the workings and effectiveness of inspection.
Tight (2019, p. 27) notes that “most of the issues encountered in carrying out empirical research also apply to documentary research”, and this is no less true when it comes to the ethical considerations that researchers need to address when working with inspection documentation. The public nature of the documentation allows the researcher to be confident that the inspection documentation can be explored without additional consent being required. However, researchers should consider the issue of anonymity.
Unsurprisingly, inspection documentation for individual schools records the name of the school concerned. Researchers, utilising the documentation, can therefore refer to such schools directly. However, as Grek et al. (2015) explored, inspection is an emotive topic for teachers and school communities. The researcher should be mindful of the impact of referring directly to specific schools in situations where the discussion is centred on potentially sensitive topics. This is particularly important given that the documentation under scrutiny from the online sources is likely to be recent and connected with schools that are still active. Defining what could be considered a sensitive topic is not easy. Still, McCulloch and Richardson (2000, p. 104) insist on “maintaining anonymity in cases where identification may cause embarrassment or offence for either personal or professional reasons”. The researcher does not need to contribute to any potential further distress by making it easy to identify a particular school and should use pseudonyms instead.
Interest in and studies focused on school inspection documentation is an exciting area of documentary research within education. The above discussion focused on projects completed by individual researchers on topics that the researcher tackles systematically on a small scale. Research using inspection documentation, however, has been undertaken that tackles far larger samples. Bokhove and Sims (2021), for example, have conducted a computer-aided analysis of some seventeen-thousand reports from OFSTED. Such approaches are unexplored in the Scottish context and offer exciting possibilities.
Hundreds, if not thousands, of Scottish reports dating back to 1840 and housed in the National Records of Scotland await further analysis. These historical documents hold valuable insights into schooling in previous decades and centuries. Moreover, they can inform discussions about the changes and continuities in educational practice and policy over time. This, in turn, would help to mitigate the five-year timeframe imposed by the online database. Such archival work demands careful planning and recording processes, but would no doubt provide fresh and informative perspectives (McCulloch, 2004).
The school inspection documentation accessed via the Education Scotland website is publicly and readily available. With careful analysis and an appreciation of how the documentation came to be and the mediated nature of its contents, it offers insights into school practices and the inspectorate’s ways of working. The online database can be searched but is not as user-friendly as it could be. Any surveying and searching should be done systematically and within clear parameters concerning school type, date range and local authority coverage. Researchers can focus solely on inspection documentation if they maximise their sample by carefully considering their study's geographical and temporal parameters. School inspection documentation also allows the researcher to work ethically by considering the issue of anonymity in their reports and write-ups. School inspection documentation can, therefore, be a valuable and insightful primary source for research dissertations in education.
Altrichter, H., & Kemethofer, D. (2015). Does accountability pressure through school inspections promote school improvement? School Effectiveness and School Improvement, 26(1), 32-56. https://doi.org/10.1080/09243453.2014.927369
Baxter, J., Grek, S., & Segerholm, C. (2015). Regulatory frameworks: Shifting frameworks, shifting criteria. In S. Grek & J. Lindgren (Eds.) Governing by inspection (pp. 74-95). Routledge. https://doi.org/10.4324/9781315758091-6
Bokhove, C., & Sims, S. (2021). Demonstrating the potential of text mining for analysing school inspection reports: A sentiment analysis of 17,000 Ofsted documents. International Journal of Research & Method in Education, 44(4), 433-445. https://doi.org/10.1080/1743727X.2020.1819228
Bone, T. R. (1968). School inspection in Scotland, 1840-1966. London University Press.
Bowen, G. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40. https://doi.org/10.3316/QRJ0902027
Clarke, J., & Baxter, J. (2014). Satisfactory progress? Keywords in English school inspection. Education Inquiry, 5(4), 481-496. https://doi.org/10.3402/edui.v5.23485
Education and Training Inspectorate. (n.d.). Inspection Reports. https://www.etini.gov.uk/publications/type/inspectionreports
Education Scotland. (2015). How good is our school? (4th edition). Education Scotland. https://education.gov.scot/nih/Documents/Frameworks_SelfEvaluation/FRWK2_NIHeditHGIOS/FRWK2_HGIOS4.pdf
Education Scotland. (2016). Changes to education inspection in August 2016. https://education.gov.scot/media/txhpkahx/lettertodirectorsofeducation-ins.pdf
Education Scotland. (2017). Summarised inspection findings: Minnigaff primary school, Dumfries and Galloway. https://education.gov.scot/media/1a2drft5/minnigaffpssif290817.pdf
Education Scotland. (2018). Briefing note for headteachers of secondary schools being inspected using a short inspection model. https://education.gov.scot/Documents/Shortmodel-BriefingNoteSecondary.pdf
Education Scotland. (2021). Updated plans for school inspections. https://education.gov.scot/education-scotland/news-and-events/news/updated-plans-for-school-inspections/
Education Scotland. (2022a). Annual report and accounts. https://education.gov.scot/media/htqhwita/es-annual-report-2021-22.pdf
Education Scotland. (2022b). Inspection advice note (June 2022). https://education.gov.scot/media/pwql4hyd/inspection-advice-note-june-2022.pdf
Education Scotland. (n.d). Inspection reports. https://education.gov.scot/education-scotland/inspection-reports/?orderBy=dateDescending
Ehren, M. C. M., & Visscher, A. J. (2008). The relationship between school inspections, school characteristics and school improvement. British Journal of Educational Studies, 56(2), 205-227. https://doi.org/10.1111/j.1467-8527.2008.00400.x
Ehren, M. C. M., Altrichter, H., McNamara, J., & O’Hara, J. (2013). Impact of school inspections on improvement of schools – Describing assumptions on causal mechanisms in six European countries. Educational Assessment, Evaluation and Accountability, 25, 3-43. https://doi.org/10.1007/s11092-012-9156-4
Estyn (n.d). Inspection Reports. https://www.estyn.gov.wales/inspection-reports
Grek, S., Lindgren, J., & Clarke, J. (2015). Inspection and emotion: The role of affective governing. In S. Grek & J. Lindgren (Eds.) Governing by inspection (pp. 116-136). Routledge. https://doi.org/10.4324/9781315758091-8
Hutchison, D. (2018). School inspection and school improvement. In T. G. K. Bryce, W. M. Humes, D. Gillies & A. Kennedy (Eds.) Scottish education (5th edition, pp. 206-215). Edinburgh University Press.
McCulloch, G. (2004). Documentary research in education, history and the social sciences. RoutledgeFalmer.
McCulloch, G. (2011). Historical and documentary research in education. In L. Cohen, L. Manion & K Morrison (Eds.) Research methods in education (7th edition, pp. 248-255). Routledge.
McCulloch, G., & Richardson, W. (2000). Historical research in educational settings. Open University Press.
McGinley, B. 2018. Local governance in Scottish education. In T. G. K. Bryce, W. M. Humes, D. Gillies and A. Kennedy (Eds.) Scottish education (5th edition, pp. 184-193). Edinburgh University Press.
Merriam, S. B., & Tisdell, E. J. (2016). Qualitative research: A guide to design and implementation (4th edition). Jossey-Bass.
Morgan, H. 2021. Conducting a qualitative document analysis. The Qualitative Report, 27(1), 64-77. https://doi.org/10.46743/2160-3715/2022.5044
OFSTED. (n.d). Find an Ofsted inspection report. https://www.gov.uk/find-ofsted-inspection-report
Ozga, J., Segerholm, C., & Lawn, M. (2015). The history and development of the inspectorates in England, Sweden and Scotland. In S. Grek & J. Lindgren (Eds.) Governing by inspection (pp. 58-73). Routledge. https://doi.org/10.4324/9781315758091-5
Perryman, J. (2009). Inspection and the fabrication of professional and performative processes. Journal of Educational Policy, 24(5), 611-631. https://doi.org/10.1080/02680930903125129
Perryman, J. (2010). Improvement after inspection. Improving Schools, 13(2), 182-196. https://doi.org/10.1177/1365480210369878
Prior, L. (2003). Using documents in social research. Sage Publications Ltd. https://doi.org/10.4135/9780857020222
Rapley, T. (2007). Doing conversation, discourse and document analysis. Sage Publications Ltd. https://doi.org/10.4135/9781849208901
Scott, J. (1990). A matter of record: Documentary sources in social research. Polity Press.
Scholes, S. C. (2020). Challenges and opportunities in religious education: Re-Considering practitioners’ approaches in Scottish secondary schools. Religious Education, 115(2), 184-200. https://doi.org/10.1080/00344087.2019.1677996
Scholes, S. C. (2021). Re-assessing religious education: Understanding the challenges and exploring the possibilities in Scottish non-denominational primary schools. 16th British Education Studies Association International Conference, 24-25 June 2021.
Scholes, S. C. (2022). Precarious provision and mixed messages: Religious education, school inspection, and the law in Scottish non-denominational secondary schools. British Journal of Religious Education, 44(4), 512-527. https://doi.org/10.1080/01416200.2022.2036101
Scottish Government. (2022a). Independent schools in Scotland: Register. https://www.gov.scot/publications/independent-schools-in-scotland-register/
Scottish Government. (2022b). School contact details. https://www.gov.scot/publications/school-contact-details/
Simons, H. (2020). Case study research: In-depth understanding in context. In P. Leavy (Ed.) The Oxford handbook of qualitative research (2nd edition, pp. 676-703). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780190847388.013.29
Tight, M. (2019). Documentary research in the social sciences. Sage Publications Ltd. https://doi.org/10.4135/9781529716559
[1] Links to the national lists of schools will likely change over time but these were accurate at the time of writing. Researchers may have to search the Scottish Government website directly for such lists.