Following our flagship annual conference, which this year focussed on the importance of effective questioning, our Group Chief Executive Simon Lebus evaluates 1880's exam question answers as to where the locusts went after the biblical plague and an unexpected analogy involving the Prince of Wales' trousers.
Our conference on the importance of effective questions - ‘Questioning Questions
’ took place yesterday. The purpose of the conference was to take a comprehensive look at question and question paper design, especially as technology, and teaching practice more generally, shift some of the emphasis onto formative assessment.
It was my last Cambridge Assessment conference before I stand down next March after fifteen years as Chief Executive. I remember on arrival in 2002 being warned about the potential online testing had for disrupting our business and also about the opportunity technology offered to develop richer and more engaging questions that would stimulate learning and improve the student experience. Fifteen years on some progress has been made in this direction but it has been fairly limited as there are many hurdles still to be overcome. One of these is the issue of cost - it is still much cheaper to repurpose a school gym as an exam hall each summer by putting in chairs, desks and an invigilator than to invest in the complex and resilient IT networks needed for the safe and foolproof delivery of public exams online. A further disincentive to investment is the astonishing rapidity with which the technology evolves; again, reflecting on the last fifteen years, we’ve moved from a largely paper-based environment, to PC, to tablet, and on to mobile, each shift involving significant expenditure and rendering much of the previous investment in what had been the ‘new’ technology redundant.
However, one particularly exciting facility technology affords is the construction of increasingly large and well-calibrated item banks which can be deployed by teachers in their day to day teaching. A good example of this is ExamBuilder
, a joint project between OCR and Hodder, which gives teachers access to a large number of old OCR exam papers from which they can construct practice tests. We have also cooperated with the Cavendish Laboratory in Cambridge which has developed Isaac Physics
using sample O Level, GCSE and A Level questions dating back to the early 1950s, along with a whole range of specially developed supporting materials, as an invaluable instructional resource.
One of the big tensions in question paper setting and the developing of exam items is the need to make sure that they are clear, to the point, and that somebody answering the question will understand exactly what it is they are being asked. This is particularly important now that exams are such a central feature of the accountability system because it means results are contested with much greater frequency and in much higher numbers than when I started fifteen years ago in what was still (largely) a pre hyper-accountability world (though this was already on its way). One unfortunate consequence has been that open-ended questions of a sort that were common thirty years ago, designed to elicit a wide range of responses to which examiners could apply their judgement are no longer always appropriate. That is reflected in the way in which question style has changed over the decades and the montage of questions at the top of this page, taken from papers in our archive, gives a bit of a sense of this. I particularly liked a question from an 1858 book of examination papers (pictured above
) about the increase in the price of wheat, oxen and eggs between 1415 and 1858 in the Economics paper as it is easy to see that it would be a fun and interesting thing to write about, though much less easy, I suspect, to mark reliably!
The ability of carelessly formulated questions to produce peculiar answers is also richly evident in our archives and a few examples from the early 1880s are shown above and below. Reflecting on the answer: ‘The Prince of Wales is said to wear his trousers only once...’ it is difficult to visualise the question. One also wonders whether the examiner reading the answer to the question on locusts awarded the candidate a bonus mark for their humour. Of course, these are extreme examples but they illustrate the way in which badly drafted questions in effect thwart the purpose of the exam as they do not give the candidate a fair opportunity to show what they are capable of.
One of the reasons we have fewer such questions in exam papers today is that a lot of research has been done on how to construct good exam questions. This is an area that Cambridge Assessment researchers have focussed on for many years and Vicki Crisp talked about this in her presentation. Although some of the issues are technical, the disciplines of good question paper writing also have much in common with those of good communication generally – short sentences, an absence of extraneous or irrelevant matter, and the avoidance of overly complex or obscure language.
This year we set just under 34,000 questions for our summer A Level and GCSE series, so making sure that questions are clear and easily understood is critical, not least as we very quickly know if they are not because of rapid feedback through social media. Good quality questions are also a powerful teaching resource. As I mentioned above, technology offers the possibility ultimately of dynamically combining learning and assessment material so as to be able to observe and monitor individual student performance over the entirety of a course of study, so that mastery of a subject discipline or skill could be certificated based on continuous observation. We are still a long way off from achieving that particular (and not necessarily wholly desirable) technological sunlit upland, but what is certain is that it will never be achieved without mastery of, what it was clear from yesterday’s conference, is the technically demanding and complex human skill of being able to formulate good, clear questions.
Group Chief Executive, Cambridge Assessment
If you weren’t able to join us in London or watch live online, you can listen to each of the expert presentations on the Cambridge Assessment website and I invite you to join the ongoing conversation by searching #CamEdLive