WEBVTT

00:00:00.000 --> 00:00:09.440
Welcome to the 2012 Cambridge Assessment Conference.

00:00:10.140 --> 00:00:13.900
My name is Paul Newton, and I'm going to be hosting today.

00:00:13.980 --> 00:00:16.680
I'm going to be chairing proceedings and guiding you through a PAC program.

00:00:17.440 --> 00:00:21.220
It's going to be led by experts from the field of assessment and beyond,

00:00:21.400 --> 00:00:24.200
all of whom are going to be focusing on the theme of our conference today,

00:00:24.480 --> 00:00:25.920
which is examining risk.

00:00:27.120 --> 00:00:29.440
Okay, well now I'd like to welcome Simon Lebus.

00:00:30.000 --> 00:00:33.480
Simon Levis is the Group Chief Executive of Cambridge Assessment,

00:00:33.480 --> 00:00:36.560
which is a role that he's held here for over a decade now.

00:00:36.560 --> 00:00:38.640
Simon is a historian by training.

00:00:38.640 --> 00:00:43.080
He studied modern history and modern English philanthropy at Oxford.

00:00:43.080 --> 00:00:47.280
Before joining Cambridge Assessment, Simon worked in a variety of senior management roles,

00:00:47.280 --> 00:00:49.880
from the world of finance to the food industry.

00:00:49.880 --> 00:00:52.760
Simon will welcome you to the conference, and then I'll come back

00:00:52.760 --> 00:00:55.320
and I'll introduce our first keynote speaker.

00:00:55.320 --> 00:00:56.440
Simon, welcome.

00:01:00.000 --> 00:01:12.240
Thank you, Paul, and good morning and welcome to all of you here for our Cambridge Assessment

00:01:12.240 --> 00:01:17.600
Conference on Risk in Examinations. I think it's fair to say these conferences always get planned

00:01:17.600 --> 00:01:22.260
a long time in advance, and we certainly hadn't anticipated when we were trying to work out what

00:01:22.260 --> 00:01:23.280
theme to...

00:01:23.281 --> 00:01:27.701
choose for this conference quite how top 18 months ago quite how topical it was going to be

00:01:27.701 --> 00:01:34.701
I think though even allowing for the fact that this has been an exceptionally eventful summer

00:01:34.701 --> 00:01:40.861
the exam system is rarely out of the headlines even in normal times and I thought I'd start with

00:01:40.861 --> 00:01:45.781
a few reflections on this and try to identify some of the themes that we'll talk about in more detail

00:01:45.781 --> 00:01:51.221
during the the day so first of all perhaps the question to ask is why is the this level of

00:01:51.221 --> 00:01:57.041
interest. I was at a recording of a World Service programme recently in which Alison Wolfe was one

00:01:57.041 --> 00:02:02.321
of the contributors and she was being interviewed by Martin Rees and she observed, I thought rather

00:02:02.321 --> 00:02:07.281
disconcertingly, that in modern societies one of the main purposes of education is, I quote,

00:02:07.661 --> 00:02:13.141
sorting and selecting. And exams, of course, as we know, are the principal device by which this

00:02:13.141 --> 00:02:19.501
process occurs. Now given that, it's essential that the exam system commands widespread trust

00:02:19.501 --> 00:02:24.121
and support, and I think it's fair to say that this has been eroded during the last

00:02:24.121 --> 00:02:30.681
12 months, and that we now face a major collective challenge to restore that, not just exam boards,

00:02:30.681 --> 00:02:36.661
regulators, politicians and administrators, but also, I think, schools, colleges and teachers,

00:02:36.661 --> 00:02:41.701
as a system depends, in a sense, on there being consenting parties, and there seems to

00:02:41.701 --> 00:02:46.561
me a bit of a whiff about the current crisis of that consent being withdrawn.

00:02:46.563 --> 00:02:47.883
How do we get to this?

00:02:48.643 --> 00:02:55.223
The American sociologist Charles Perrault's 1984 study of living with high-risk technologies

00:02:55.223 --> 00:03:01.863
describes the phenomenon whereby tightly coupled systems with high levels of interactive complexity

00:03:01.863 --> 00:03:04.943
are subject to what he calls normal accidents.

00:03:05.763 --> 00:03:09.363
That is to say, accidents caused by the interaction within a system

00:03:09.363 --> 00:03:12.103
in ways that are difficult to predict or anticipate,

00:03:12.823 --> 00:03:15.883
of errors in different and mutually dependent processes,

00:03:15.883 --> 00:03:19.683
rather like the different numbers in a combination lot coming together.

00:03:20.563 --> 00:03:24.523
He argues that it is an inherent and normal, hence normal accidents,

00:03:25.003 --> 00:03:28.683
property of complex modern systems to experience such interactions

00:03:28.683 --> 00:03:32.683
and rather unsettlingly, I thought, applies the analysis to nuclear power,

00:03:33.103 --> 00:03:35.283
petrochemical plants and aircraft travel.

00:03:36.023 --> 00:03:40.403
However, much of what he says could equally be considered to be true of the exam system

00:03:40.403 --> 00:03:44.623
and in particular his description of the unpredictable way in which changes can interact

00:03:44.623 --> 00:03:48.923
seems to me to go to the heart of the risk in exam systems that we are considering today

00:03:48.923 --> 00:03:53.843
and how we seek to manage it, especially at a time when it could be argued I think that

00:03:53.843 --> 00:03:59.063
the system is already running hot. Now looked at in this light, some of the turbulence we are

00:03:59.063 --> 00:04:04.543
currently experiencing seems inevitable. I've just celebrated, as Paul mentioned, my 10th anniversary

00:04:04.543 --> 00:04:09.843
at Cambridge Assessment, and during the 10 years I've been here have had to manage the impact of a huge range

00:04:09.844 --> 00:04:14.604
of changes. To give a flavour, during this time there have been six secretaries of state,

00:04:15.044 --> 00:04:20.304
the Department for Education has reincarnated itself under different brands three times,

00:04:20.744 --> 00:04:23.944
and the way in which school performance tables are calculated has been changed no fewer than

00:04:23.944 --> 00:04:30.964
nine times. We've also seen serial major changes to A-levels and GCSEs, a drifting apart of the

00:04:30.964 --> 00:04:35.224
three-country regulatory regime, the failed introduction of the diploma as a potential

00:04:35.224 --> 00:04:40.464
replacement for A-levels, the expensive and bureaucratic introduction of the QCF, another

00:04:40.464 --> 00:04:44.684
proposed introduction of the EBAC as a replacement for GCSEs, and there are of course a number

00:04:44.684 --> 00:04:48.164
of other things that I haven't included in that list, but I think you get the idea.

00:04:49.004 --> 00:04:53.584
Now at the same time, those charged with administering this system often do not stay for long, and

00:04:53.584 --> 00:04:57.664
perhaps given the velocity of change, you can't blame them. I think my own tenure makes

00:04:57.664 --> 00:05:02.404
me something probably of a veteran. And this means, and others have observed this too, that

00:05:02.404 --> 00:05:08.344
there was a loss of policy memory something which can only serve to heighten risk this is all

00:05:08.344 --> 00:05:13.584
against a background of a system that steadily tended anyway to to grow more complex in response

00:05:13.584 --> 00:05:19.264
to a variety of social and educational pressures this has been at least partly and the worthy cause

00:05:19.264 --> 00:05:24.224
of promoting access and diversity and supporting increased learner choice but it has had some

00:05:24.224 --> 00:05:30.044
striking effects this is well illustrated in research conducted by one of my colleagues who's

00:05:30.044 --> 00:05:33.124
that reveals that there were 21,600

00:05:33.125 --> 00:05:39.225
172 possible combinations of texts in one of the English literature specifications replaced by the

00:05:39.225 --> 00:05:45.445
new English GCSE this summer, making it theoretically possible for every candidate in effect to have

00:05:45.445 --> 00:05:52.085
their own unique specification. Now this in turn creates some major administrative technical

00:05:52.085 --> 00:05:57.585
awarding and marking challenges and the question has to be asked I think as to whether the

00:05:57.585 --> 00:06:02.885
educational benefit justifies offering quite such extreme levels of choice or whether we are just

00:06:02.885 --> 00:06:08.865
designing risk into the system. There's also I think a financial dimension to this. Research

00:06:08.865 --> 00:06:15.265
published by Reform in 2011 revealed that the cost of A-levels has increased from £9.2 million

00:06:15.265 --> 00:06:23.405
in 1970 to £71 million in 2008. At the same time the number of candidates has increased 149%

00:06:23.405 --> 00:06:28.005
to just over a quarter of a million but the number of papers they are sitting has risen by a much

00:06:28.005 --> 00:06:32.885
greater amount from 644,000 to nearly 5.5 million.

00:06:32.885 --> 00:06:35.985
So despite the rise in overall cost,

00:06:35.985 --> 00:06:37.905
the amount of money available per paper

00:06:37.905 --> 00:06:42.485
has actually declined from 14 pounds 29 to 12 pounds 96,

00:06:42.485 --> 00:06:45.025
a drop of around 9%.

00:06:45.025 --> 00:06:47.485
So we find ourselves in the curious situation

00:06:47.485 --> 00:06:50.465
where we're expecting more and more from our exam system,

00:06:50.465 --> 00:06:52.865
where the results are becoming ever higher stakes,

00:06:52.865 --> 00:06:54.805
and yet where we're arguably spending

00:06:54.805 --> 00:06:56.405
proportionately less on individual.

00:06:56.406 --> 00:07:02.546
exams so what's the impact what gives us a consequence and what does it suggest also about

00:07:02.546 --> 00:07:08.446
our attitude to risk and our readiness or unreadiness to tolerate it the actual the current

00:07:08.446 --> 00:07:12.246
exam system I think actually compares quite favorably with some other high impact systems

00:07:12.246 --> 00:07:16.926
in which exposure to the risk associated with professional judgment plays a key part

00:07:16.926 --> 00:07:22.526
and our next speaker will no doubt touch on this for example studies of hospital admission show

00:07:22.526 --> 00:07:27.926
that around one in ten of all hospital patients suffer some form of adverse medical event,

00:07:28.166 --> 00:07:30.426
of which around half are preventable.

00:07:31.426 --> 00:07:37.746
Of those suffering such adverse events, around 8% die and 6% suffer permanent disablement.

00:07:38.266 --> 00:07:44.526
So that equates to a catastrophic error rate of around 1.4%, thus affecting more than one

00:07:44.526 --> 00:07:45.666
in 100 patients.

00:07:46.386 --> 00:07:51.326
Now, although there are methodological difficulties about making these comparisons, that is probably,

00:07:51.326 --> 00:07:54.166
I would suggest around 50 times greater

00:07:54.166 --> 00:07:56.426
than the error rate in the exam system,

00:07:56.426 --> 00:07:58.706
as reflected in statistics deriving

00:07:58.706 --> 00:08:01.706
from results inquiries in the appeals process.

00:08:01.706 --> 00:08:04.146
Even that error rate, however, represents

00:08:04.146 --> 00:08:08.046
a quite unacceptable cost in terms of its impact on candidates.

00:08:08.046 --> 00:08:10.906
We operate in a modern consumer culture

00:08:10.906 --> 00:08:13.546
where zero tolerance of error sits side by side

00:08:13.546 --> 00:08:16.046
with the expectation of maximum levels of choice

00:08:16.046 --> 00:08:17.586
and personalization.

00:08:17.586 --> 00:08:19.686
And our challenge in this very exacting class

00:08:19.688 --> 00:08:24.888
climate, and while subject to very high degrees of external scrutiny, it's to try and identify

00:08:24.888 --> 00:08:30.128
what we can do to reduce error at both system and candidate level, and to understand the

00:08:30.128 --> 00:08:34.548
extent of risk, because we'll never banish it totally, that's acceptable for a system

00:08:34.548 --> 00:08:39.368
designed to meet these high and quite often contradictory expectations to which we're

00:08:39.368 --> 00:08:39.908
exposed.

00:08:40.808 --> 00:08:44.648
Now, a good example, and a topical one, is the question of quality of marking raised by

00:08:44.648 --> 00:08:47.328
the recently published report by HMC.

00:08:47.328 --> 00:08:52.708
I've already referred to the multiplicity of options available in just one GCSE syllabus

00:08:52.708 --> 00:08:56.388
and there's clearly a trade-off between the range of choice we can offer

00:08:56.388 --> 00:08:59.328
and the associated complexity of the assessment model

00:08:59.328 --> 00:09:03.008
with the ability to guarantee a high and consistent quality of marking.

00:09:03.928 --> 00:09:07.608
Getting agreement from the teaching community on how to manage that effectively

00:09:07.608 --> 00:09:11.128
has therefore got to be a key part of our approach to risk management.

00:09:11.968 --> 00:09:16.108
We can only come by that and this comes back to my earlier observation about consent.

00:09:16.108 --> 00:09:20.388
if we're able to reach a shared view on how to strike the right balance

00:09:20.388 --> 00:09:22.748
between a rules-based approach to marking

00:09:22.748 --> 00:09:25.848
and making sure there is sufficient room and expertise

00:09:25.848 --> 00:09:27.988
for the application of professional judgment.

00:09:29.188 --> 00:09:31.428
Finally, I think we need to recognise the problem

00:09:31.428 --> 00:09:35.688
that an excessive preoccupation with risk management, paradoxically,

00:09:35.688 --> 00:09:38.568
can stifle innovation and educational progress.

00:09:39.068 --> 00:09:42.968
The high-stakes nature of assessment makes schools and colleges risk-averse

00:09:42.969 --> 00:09:48.849
and encourages them and indeed parents and pupils to stick with what they know not least because of

00:09:48.849 --> 00:09:54.329
the impact of accountability arrangements one consequence has been that we have ended up with

00:09:54.329 --> 00:09:59.749
a very predictable exam system and that's created disincentives for schools and colleges to innovate

00:09:59.749 --> 00:10:05.829
and experiment paradoxically of course also the resulting perception of inertia has provided

00:10:05.829 --> 00:10:10.749
politicians who because of the electoral cycle are almost never around to live with the consequences

00:10:10.749 --> 00:10:13.969
with a greater incentive to go for radical reform.

00:10:14.529 --> 00:10:17.849
The subsequent cycle in which ambitious reform proposals

00:10:17.849 --> 00:10:20.729
get watered down in the effort to make them more easily manageable.

00:10:21.269 --> 00:10:23.529
Ironically, there leads to an unhealthy combination

00:10:23.529 --> 00:10:25.969
of both disruptive and incremental change

00:10:25.969 --> 00:10:27.749
taking place at the same time.

00:10:28.109 --> 00:10:30.129
And this invariably leaves the exam system

00:10:30.129 --> 00:10:32.309
more complex and risky than before.

00:10:33.169 --> 00:10:34.889
So these are complex challenges

00:10:34.889 --> 00:10:36.789
that we all of us face in education.

00:10:37.509 --> 00:10:39.289
Today, I hope we'll have an opportunity

00:10:39.289 --> 00:10:42.089
to explore them with colleagues from a wide range of disciplines

00:10:42.089 --> 00:10:45.709
and consider what lessons we can learn from risk management more generally

00:10:45.709 --> 00:10:50.009
as we embark on what looks like being a period of very major qualification reform.

00:10:50.149 --> 00:10:53.489
So I hope you all have an interesting and lively day.

00:10:53.749 --> 00:10:54.769
It's a pleasure to have you here.

00:10:54.849 --> 00:10:55.529
Thank you very much.

00:10:55.529 --> 00:10:57.529
APPLAUSE