Why evaluate student success interventions?
- Evaluation outcomes provide an assessment tool for academic schools to know the extent to which their interventions are effective and are having a positive impact on students’ achievement. It is also a tool for schools to explore avenues for student support, or for staff engagement in Equality Diversity and Inclusion activities. The impact of these interventions will subsequently result in a reduction of our institutional gaps.
- The OfS Regulatory Framework requires institutions to provide evidence of impact on the delivery of the Access and Participation Plan (APP) through evaluation, and in line with the standards of evidence guide provided by the OfS.
- To meet the institutional principles of accountability, good practice and value for money, this evaluation framework provides a tool to identify interventions which are most effective in reducing attainment differentials, thus leading to a more agile mechanism to meet institutional OfS targets, as well as a faster reduction in gaps.
How do we evaluate?
The Student Success evaluation framework is informed by Theory of Change (ToC), as an approach to programme planning and evaluation.
In terms of programme planning, the Student Success ToC provides a tool and road map for academic schools and divisions to:
- define the purpose and rationale of their interventions and activities
- establish how the data on awarding and continuation gaps will inform decisions around targets and expected outcomes
- identify the domains of change and pre-conditions necessary to effectively develop the programme
- outline the strategic priorities in terms of type of interventions and target groups of students within socio-demographic characteristics, stages, and evaluation outcomes.
In terms of programme evaluation, the Student Success ToC includes two types of evaluation that are interlinked, process evaluation and impact evaluation.
Process Evaluation allows us to understand how interventions have been implemented and delivered, and identify the extent to which such a process has been effective in achieving the expected outcome of the interventions. Through the data monitoring of student and staff engagement in these interventions, case studies, and feedback analysis, we can establish how effective they have been in targeting students, and how the intervention has contributed to changing behaviours and attitudes for institutional change. In terms of the OfS standards of evidence this type of evaluation is defined as Type 1 Narrative.
Impact evaluation provides a methodology to establish what difference the intervention made and to what extent the intervention outcomes can contribute to students' improvement in attainment or continuation. This type of evaluation is considered by the OfS as the Type 2 Empirical Evidence of impact evaluation.
We begin the impact evaluation by establishing intervention groupings, this gives us a much better chance of having a population that we are comfortable drawing conclusions from. There are 15 broad groupings and 60 granular groupings in the Student Success Evaluation Framework. By calculating the students’ change in overall average between two data points we can then examine where attendance at an intervention correlated with a higher increase in average. Outliers (data points that do not follow the behaviour of the rest of the group) are removed prior to this analysis taking place and we account for the fact that, for example, different subject areas are likely to have a different distribution of the changes in averages.
The methodology has developed over time as we refine and make the statistical analysis more robust. We justify the implementation of different statistical techniques by examining the distributions and clarify our reason for removing data that we identify as an outlier.
Once we have a clean dataset of non-outlier data, we move to categorise intervention codings which align to positive impact on the students’ attainment or attendance, and those that align to negative impact. We also clarify where there is either no data or insufficient data to conduct an evaluation. Further categorisation follows to identify strong, medium, and weak impact.
Given that some intervention codes benefit only certain subject areas, we also clarify where this is the case.