Ofqual’s Dr Paul Newton visited Cambridge just before Christmas to explore the issue of error and reliability in examinations. Newton suggested that the assessment profession, while more open than it was, still does not have a clear message on error. He argued that, as error is inevitable, the assessment community should not be afraid to discuss it openly.
His starting point was the work of the nineteenth century statistician Francis Ysidro Edgeworth who identified that measurement error - a product of the fact that competent judges can reasonably be expected to differ in some of their judgements - was inevitable. He then looked at what some more recent thinkers, notably Black and Wiliam, have written about error, as well as what the QCA and Awarding Bodies have said about it in recent years.
Newton weighed the arguments for and against being more open about discussing error in, and with the, public. Against doing so he noted that:
- the narrative was not fully ‘watertight’ i.e it is not fully understood or agreed upon by those within the assessment community;
- it is a very complicated argument to present through the modern media;
- it may disguise the fact that assessment results retain their value to users, as best estimates, despite the presence of error;
- it would shake confidence in a system in which many people have a great deal of faith.
However he came down in favour of engagement for the following reasons:
- that the assessment community has wrestled with the issue for 120 years and has developed lots of ways of dealing with error;
- that illuminating the subject may make it less likely that bad decisions will be made on the basis of assessments which are fallible;
- that by perpetuating a ‘myth of perfection’ the assessment profession damages itself;
- that in a new order of openness and transparency, exemplified by the Freedom of Information Act, it is simply not sustainable to refrain from engaging.
Related materials