[00:00:03:60]
Hello my name is Carmen and I am a senior
research officer at Cambridge assessment
today I'm going to present some work that my
colleague Sylvia and I carried out last year
looking at the impact of access
arrangements on a student's performance.
[00:00:20:56]
Access arrangements are reasonable
adjustments agreed before an assessment
to allow candidates with a specific needs, for
example candidates with special educational
needs, candidates with disabilities, candidates
with temporary injuries to access the assessment
and remove unnecessary barriers without changing
its demands for example a candidate who is unable
to write due to a motor impairment may be awarded
a scribe who will write down the answers for them.
[00:00:46:88]
Other access arrangements are for example extra
time, readers, word processors, colour overlays or
supervised rest breaks the main aim of an access
arrangement is to meet the particular needs of an
individual candidate without affecting the
integrity or the demand of the assessment.
[00:01:04:80]
Access arrangements need to be agreed in
advance of the assessment and they are based
on evidence of need for example cognitive test
scores can be used to determine the need for
extended time requests access arrangements
should as much as possible be the same
or similar during lessons classroom tests and
highest take assessments the principle of the
access arrangement to align with the student's
normal way of working aims to ensure that the
students are not introduced to an unknown
procedure or technology during the assessment.
[00:01:35:84]
Familiarity is an important consideration if the
arrangement is to be beneficial for the candidate,
however the practice of providing access
arrangements is not without controversy.
[00:01:49:04]
It is not clear to what extended workers intended
and whether or not they lead to more valid test
scores for some groups of students. The goal
of the assessment is to measure the student's
achievement on the constructs of interest
rather than their ability to access the test.
[00:02:05:28]
So access arrangements should be provided if
a student's disability or learning difficulty
affects their performance in a way
unrelated to the construct of interest.
[00:02:15:28]
However some argue that access
arrangements may potentially
lead to an unfair advantage for some students
rather than simply levelling the playing field.
[00:02:24:32]
If that were the case the test scores of the
students with arrangements will be inflated
which will also have a detrimental
effect on the validity of the assessment.
[00:02:36:24]
Providing access arrangements to
students with special educational
needs or disabilities has become the focus
of increased attention in recent years
in the United Kingdom awarding bodies aimed
to make general and vocational qualifications
accessible to candidates with particular
requirements in a manner that does not
undermine the standards or compromise the
assessment criteria of the qualifications.
[00:02:58:56]
The joint council for qualifications provides
information and guidance for centres on all
possible access arrangements that are available
for candidates with disabilities and learning
difficulties by providing reasonable adjustments
to the assessment procedures according to bodies
promote inclusivity in education and comply
with the equality act content from 2010.
[00:03:24:64]
There has been some research on the effectiveness
of access arrangements particularly in the United
States but the findings have been contradictory
for example a recent review of research on extra
time revealed that out of 28 identified
studies 16 suggested that the extra time
allowance did not confer any advantage
to students with learning difficulties
[00:03:46:00]
whereas 12 studies suggested that the extra time
over-inflated the scores of the students the
authors argue that the methodological and design
differences between the studies such as task type
or amount of extra time granted are likely to
explain some of the differences in the findings
[00:04:05:60]
to date the majority of the research studies
and access arrangements come from classroom
based experimental designs where the students
were randomly assigned to different groups
with or without access arrangements without
considering their actual need this also means
that the learners may not be familiar with
the arrangement or the technology they are
supposed to use in the assessments this can
affect the effectiveness of the arrangement.
[00:04:30:56]
Furthermore most of the studies come from the
us where their procedures tend to be different
for example the amount of extra time is more
generous in the us than in the UK while these
studies provide a wealth of evidence on whether
or not arrangements provide intended assistance
to students with disabilities or learning
difficulties it is difficult to extrapolate the
rule the results to large high-stake assessments
especially outside of the united states.
[00:04:58:72]
So this work aim to investigate whether access
arrangements are working as intended for this
they create equity and they level the playing
field for candidates with disabilities and
learning difficulties without conferring them an
unfair advantage. The strength of the current study
is that we use actual assessment
data from high-stakes examinations.
[00:05:17:92]
This ensured that access arrangements
were awarded based on the evidence of need
are not abundant and that they were the
normal way of working with the students.
[00:05:28:80]
So before talking you through the results of
our study let me tell you a bit about our data
and the methodology we used the data for the
research was provided by cameras international,
an international awarding body that offers
qualifications in more than 150 countries
in the world so in particular they provided
individual level data on the students who
requested access arrangements in international
GCSEs in the UK in the academic year 2016-17.
[00:05:57:92]
The data cover all centre delegated arrangements
this can be approved directly by the centre for
example lecture time of 25 and non-delegated
arrangements and this need a problem by the
awarding body so for example extra time over 25 a
candidate might need a particular arrangement in
[00:06:15:36]
one type of assessment but not in others or
they may need the same degree of assistance
in every subject. The data provided by
Cambridge international did not identify
the subjects in which the arrangements were
requested just the qualification and as a result
in this work we assume that the same arrangements
apply to candidates in of the subjects they took.
[00:06:36:80]
So this graph shows the most common access
arrangements for GCSE examinations
in the UK in the 2016-17 academic year. The focus
of the research was on the most frequently used
arrangements which are extra time up to
25 percent word processor with their
spell check disabled supervised resurrects
breathing assistance and writing assistance.
[00:06:57:20]
So note that a reading assistant includes the
reading pen the reader and the computer leader
and writing the systems includes scribes
and also the use of voice activated software.
[00:07:10:40]
So to investigate the impact of access
arrangements on performance we compare
the performance of candidates with and with our
access arrangements we look at each of the five
arrangements I have just mentioned in turn
and we focus on GCSE English language which
was the most popular I just used in the UK in
2016-17 then we use propensity escort machine
[00:07:33:28]
to examine the impact of the arrangements on
performance taking into account the background
characteristics of the candidates so this
strategy provides a framework to identify under
specific assumptions causal effects rather
than simple measures of association so want to
match the performance of comparable candidates
with them with how arrangements must compare
candidates that were much on the
following background characteristics
[00:07:56:96]
at the individual level they were matched on
gender school year and concurrent attainment
[00:08:01:28]
so the grade in a coursework component was used as
a measure of attainment and it can be argued that
coursework rate is a good measure of attainment
for candidates with access arrangements as it might
not be influenced by the candidates having the
arrangements in place for the other exams at the
school level their candidates were much on the
type of the school whether it was comprehensive,
[00:08:19:52]
independent etc the gender composition of the
school the socio-economic deprivation of the
school and the average school performance for each
of these arrangements the students will also match
on whether or not they have requested the other
types of arrangements considered in the research
[00:08:35:84]
so let's move to the results
[00:08:40:24]
over 99 000 students who took GCSE English
Language in June 2017 were considered in the
research. This table shows the numbers and
percentages of the students who requested
each of the five arrangements students can
be awarded more than one arrangement for
example students could have extra
time and a reader in the same exam
[00:09:00,72]
in the first step we carried out descriptive
analysis to look at the performance in
English language amongst the students with and
without the five types of access arrangements
[00:09:10,08]
So you can see in this graph that the performance
was lower for the students who had access
arrangements in place with the difference being
statistically significant in all arrangements
[00:09:19:28]
The biggest difference was for writing sorry
for reading assistance where candidates with
the arrangement achieve over one grade lower than
candidates with our systems similar differences
were found for writing assistance and extra time
difference were much smaller from the use of word
processors or supervised rest breaks however these
descriptive statistics do not account for possible
differences in the two groups of the students it's
the group with and without access arrangements.
110
[00:09:48:08]
So next we look at some background characteristics
of the students who had requested the selected
arrangements and compare them to those students
without the arrangements to check if indeed the
two groups of the students were different so this
table here shows the comparisons for extra time up
to 25 percent so in terms of concurrent attainment
the students with extra time had lower attainment
[00:10:07:92]
than their students without extra time around
half a grade lower the average deprivation was
higher for students without extra time however
this difference between both groups of those
students was very small the gender ratio
for students with and with our arrangements
showed that male students were more likely to have
access arrangements in place than female students
118
[00:10:29:36]
and the majority of the students taking the
English language qualification were in jail 11 so
they were 16 year rules regardless of whether or
not they had requested access arrangements but for
example the proportion of the students in year
12 that's the 17 year-olds was slightly higher
[00:10:44:80]
for students with access arrangements a large
proportion of the students with arrangements done
of the students without them were in selective or
independent schools and these patterns were very
similar for all the arrangements so we have seen
here that there are differences in the background
characteristics of the students with and with our
access arrangements and therefore it is important
to consider such characteristics
when looking at an exam performance
[00:11:13:60]
so next we carried out further analysis
taking into account the background
characteristics of the different groups of
the students and as I mentioned earlier we
used propensity scores to control for
the imbalances in the characteristics
[00:11:24:48]
the grade in GCSE English language was then
compared for equivalent groups of students so when
only comparable students are considered in the
analysis the differences in GCSE English language
performance between candidates with and with
how each access arrangements are much smaller so
this was the graph from before the propensity is
for machine so the difference after the propensity
is chromatin although quite small are still
significant for extra time and readiness systems
[00:11:54:48]
but the large sample size available in this
research makes statistical significant
differences likely to emerge even for small
effects so a closer look at the average grade
of two groups reveals that difference are no more
than a third of a grade so the biggest was for
reading assistance which is much smaller than
before so it's 0.32 compared to 1.37 for grade
so candidates with access agreements
performed similarly to candidates without
them when we took into account
their background characteristics.
[00:12:24:16]
So to finish just a couple of slides with
some conclusions and limitations of this work
[00:12:33:84]
comparable candidates with access arrangements
perform similarly to candidates without
arrangements the research shows that access
arrangements seem to support the students in
demonstrating their achievement and to some extent
they create equity between the groups the results
were consistent across the different access
arrangements and we didn't find any evidence
that access arrangements can fair advantage to
students so even with the access arrangements
students with arrangements perform slightly
worse than comparable students without them
[00:13:03:12]
an important issue to consider is that the effects
of disability of performance cannot be quantified
that is the access arrangements need to be
awarded on a case-by-case basis where there is
clear evidence of need this approach ensures that
disadvantaged students receive the support needed
[00:13:19,20]
and is tailored to their requirements so the
results of this study demonstrate that when these
processes follow access arrangements fulfil
their role increase the level playing field
for candidates with various needs and should
continue to be used in assessment where the
need is justified while this study helped us to
better understand the impact of arrangements on
the students performance it had some limitations
first of all we only investigated the most popular
five access arrangements secondly many students
are awarding more than one type of support
[00:13:49:44]
which is likely to have a compound effect in the
performance so future wars should investigate the
impact of multiple arrangements on performance
one of the strengths of the work was that we
match the students on a number of background
characteristics but many factors can affect
students performance and examinations
and we did not have data on all of them
[00:14:08:48]
okay that's everything from us today
thank you very much for listening
Return to top