Equity or unfair advantage? Impact of access arrangements on students’ performance

Equity or unfair advantage? Impact of access arrangements on students’ performance

20 Dec 2021 (14:17)

Equity or unfair advantage? Impact of access arrangements on students’ performance. Carmen Vidal Rodeiro and Sylwia Macinska, BERA conference 13-16 September 2021.

Video transcript

[00:00:03:60] Hello my name is Carmen and I am a senior research officer at Cambridge assessment today i'm going to present some work that my colleague Sylvia and I carried out last year looking at the impact of access arrangements on a student's performance.

[00:00:20:56] Access arrangements are reasonable adjustments agreed before an assessment to allow candidates with a specific needs, for example candidates with special educational needs, candidates with disabilities, candidates with temporary injuries to access the assessment and remove unnecessary barriers without changing its demands for example a candidate who is unable to write due to a motor impairment may be awarded a scribe who will write down the answers for them.

[00:00:46:88] Other access arrangements are for example extra time readers word processors color overlays or supervised rest breaks the main name of an access arrangement is to meet the particular needs of an individual candidate without affecting the integrity or the demand of the assessment.

[00:01:04:80] Access arrangements need to be agreed in advance of the assessment and they are based on evidence of need for example cognitive test scores can be used to determine the need for extended time requests access arrangements should as much as possible be the same or similar during lessons classroom tests and highest take assessments the principle of the access arrangement to align with the student's normal way of working aims to ensure that the students are not introduced to an unknown procedure or technology during the assessment.

[00:01:35:84] Familiarity is an important consideration if the arrangement is to be beneficial for the candidate, however the practice of providing access arrangements is not without controversy.

[00:01:49:04] It is not clear to what extended workers intended and whether or not they lead to more valid test scores for some groups of students. The goal of the assessment is to measure the student's achievement on the constructs of interest rather than their ability to access the test.

[00:02:05:28] So access arrangements should be provided if a student's disability or learning difficulty affects their performance in a way unrelated to the construct of interest.

[00:02:15:28] However some argue that access arrangements may potentially lead to an unfair advantage for some students rather than simply leveling the playing field.

[00:02:24:32] If that were the case the test scores of the students with arrangements will be inflated which will also have a detrimental effect on the validity of the assessment.

[00:02:36:24] Providing access arrangements to students with special educational needs or disabilities has become the focus of increased attention in recent years in the United Kingdom awarding bodies aimed to make general and vocational qualifications accessible to candidates with particular requirements in a manner that does not undermine the standards or compromise the assessment criteria of the qualifications.

[00:02:58:56] The joint council for qualifications provides information and guidance for centers on all possible access arrangements that are available for candidates with disabilities and learning difficulties by providing reasonable adjustments to the assessment procedures according to bodies promote inclusivity in education and comply with the equality act content from 2010.

[00:03:24:64] There has been some research on the effectiveness of access arrangements particularly in the United States but the findings have been contradictory for example a recent review of research on extra time revealed that out of 28 identified studies 16 suggested that the extra time allowance did not confer any advantage to students with learning difficulties

[00:03:46:00] whereas 12 studies suggested that the extra time over-inflated the scores of the students the authors argue that the methodological and design differences between the studies such as task type or amount of extra time granted are likely to explain some of the differences in the findings

[00:04:05:60] to date the majority of the research studies and access arrangements come from classroom based experimental designs where the students were randomly assigned to different groups with or without access arrangements without considering their actual need this also means that the learners may not be familiar with the arrangement or the technology they are supposed to use in the assessments this can affect the effectiveness of the arrangement.

[00:04:30:56] Furthermore most of the studies come from the us where their procedures tend to be different for example the amount of extra time is more generous in the us than in the uk while these studies provide a wealth of evidence on whether or not arrangements provide intended assistance to students with disabilities or learning difficulties it is difficult to extrapolate the rule the results to large high-stake assessments especially outside of the united states.

[00:04:58:72] So this work aim to investigate whether access arrangements are working as intended for this they create equity and they level the playing field for candidates with disabilities and learning difficulties without conferring them an unfair advantage. The strength of the current study is that we use actual assessment data from high-stakes examinations.

[00:05:17:92] This ensured that access arrangements were awarded based on the evidence of need are not abundant and that they were the normal way of working with the students.

[00:05:28:80] So before talking you through the results of our study let me tell you a bit about our data and the methodology we used the data for the research was provided by cameras international, an international awarding body that offers qualifications in more than 150 countries in the world so in particular they provided individual level data on the students who requested access arrangements in international gcses in the uk in the academic year 2016-17.

[00:05:57:92] The data cover all center delegated arrangements this can be approved directly by the center for example lecture time of 25 and non-delegated arrangements and this need a problem by the awarding body so for example extra time over 25 a candidate might need a particular arrangement in

[00:06:15:36] one type of assessment but not in others or they may need the same degree of assistance in every subject. The data provided by chems international did not identify the subjects in which the arrangements were requested just the qualification and as a result in this work we assume that the same arrangements apply to candidates in of the subjects they took.

[00:06:36:80] So this graph shows the most common access arrangements for righteous cse examinations in the uk in the 2016-17 academic year. The focus of the research was on the most frequently used arrangements which are extra time up to 25 percent word processor with their spell check disabled supervised resurrects breathing assistance and writing assistance.

[00:06:57:20] So note that a reading assistant includes the reading pen the reader and the computer leader and writing the systems includes scribes and also the use of voice activated software.

[00:07:10:40] So to investigate the impact of access arrangements on performance we compare the performance of candidates with and with our access arrangements we look at each of the five arrangements I have just mentioned in turn and we focus on GCSE English language which was the most popular I just used in the UK in 2016-17 then we use propensity escort machine

[00:07:33:28] to examine the impact of the arrangements on performance taking into account the background characteristics of the candidates so this strategy provides a framework to identify under specific assumptions causal effects rather than simple measures of association so want to match the performance of comparable candidates with them with how arrangements must compare candidates that were much on the following background characteristics

[00:07:56:96] at the individual level they were matched on gender school year and concurrent attainment

[00:08:01:28] so the grade in a coursework component was used as a measure of attainment and it can be argued that coursework rate is a good measure of attainment for candidates with taxes arrangements as it might not be influenced by the candidates having the arrangements in place for the other exams at the school level their candidates were much on the type of the school whether it was comprehensive

[00:08:19:52] independent etc the gender composition of the school the socio-economic deprivation of the school and the average school performance for each of these arrangements the students will also match on whether or not they have requested the other types of arrangements considered in the research

[00:08:35:84] so let's move to the results

[00:08:40:24] over 99 000 students who took igcse english language in june 2017 were considered in the research. This table shows the numbers and percentages of the students who requested each of the five arrangements students can be awarded more than one arrangement for example students could have extra time and a reader in the same exam

[00:09:00,72] in the first step we carried out descriptive analysis to look at the performance in english language amongst the students with and without the five types of access arrangements

[00:09:10,08] So you can see in this graph that the performance was lower for the students who had access arrangements in place with the difference being statistically significant in all arrangements

[00:09:19:28] The biggest difference was for writing sorry for reading assistance where candidates with the arrangement achieve over one grade lower than candidates with our systems similar differences were found for writing assistance and extra time difference were much smaller from the use of word processors or supervised rest breaks however these descriptive statistics do not account for possible differences in the two groups of the students it's the group with and without access arrangements.

110

[00:09:48:08] So next we look at some background characteristics of the students who had requested the selected arrangements and compare them to those students without the arrangements to check if indeed the two groups of the students were different so this table here shows the comparisons for extra time up to 25 percent so in terms of concurrent attainment the students with extra time had lower attainment

[00:10:07:92] than their students without extra time around half a grade lower the average deprivation was higher for students without extra time however this difference between both groups of those students was very small the gender ratio for students with and with our arrangements showed that male students were more likely to have access arrangements in place than female students

118

[00:10:29:36] and the majority of the students taking the english language qualification were in jail 11 so they were 16 year rules regardless of whether or not they had requested access arrangements but for example the proportion of the students in year 12 that's the 17 year olds was slightly higher

[00:10:44:80] for students with access arrangements a large proportion of the students with arrangements done of the students without them were in selective or independent schools and these patterns were very similar for all the arrangements so we have seen here that there are differences in the background characteristics of the students with and with our access arrangements and therefore it is important to consider such characteristics when looking at an exam performance

[00:11:13:60] so next we carried out further analysis taking into account the background characteristics of the different groups of the students and as I mentioned earlier we used propensity scores to control for the imbalances in the characteristics

[00:11:24:48] the grade in igcse english language was then compared for equivalent groups of students so when only comparable students are considered in the analysis the differences in GCSE english language performance between candidates with and with how each access arrangements are much smaller so this was the graph from before the propensity is for machine so the difference after the propensity is chromatin although quite small are still significant for extra time and readiness systems

[00:11:54:48] but the large sample size available in this research makes statistical significant differences likely to emerge even for small effects so a closer look at the average grade of two groups reveals that difference are no more than a third of a grade so the biggest was for reading assistance which is much smaller than before so it's 0.32 compared to 1.37 for grade so candidates with access agreements performed similarly to candidates without them when we took into account their background characteristics.

[00:12:24:16] So to finish just a couple of slides with some conclusions and limitations of this work

[00:12:33:84] comparable candidates with access arrangements perform similarly to candidates without arrangements the research shows that access arrangements seem to support the students in demonstrating their achievement and to some extent they create equity between the groups the results were consistent across the different access arrangements and we didn't find any evidence that access arrangements can fair advantage to students so even with the access arrangements students with arrangements perform slightly worse than comparable students without them

[00:13:03:12] an important issue to consider is that the effects of disability of performance cannot be quantified that is the access arrangements need to be awarded on a case-by-case basis where there is clear evidence of need this approach ensures that disadvantaged students receive the support needed

[00:13:19,20] and is tailored to their requirements so the results of this study demonstrate that when these processes follow access arrangements fulfill their role increase the level playing field for candidates with various needs and should continue to be used in assessment where the need is justified while this study helped us to better understand the impact of arrangements on the students performance it had some limitations first of all we only investigated the most popular five access arrangements secondly many students are awarding more than one type of support

[00:13:49:44] which is likely to have a compound effect in the performance so future wars should investigate the impact of multiple arrangements on performance one of the strengths of the work was that we match the students on a number of background characteristics but many factors can affect students performance and examinations and we did not have data on all of them

[00:14:08:48] okay that's everything from us today thank you very much for listening

Return to top