Heather Clements-Wheeler is a member of The Assessment Network at Cambridge and currently serves as Principal Examiner (PE) and Chair of Examiners (CoE) for two awarding organisations. Here she shares insights from her research on the implications of moving geography exams to digital screens - and how mode effects can influence outcomes, and what she finds most rewarding about her work.
Could you tell us about your current roles and the nature of your work?
I am currently a Principal Examiner (PE), and Chair of Examiners (CoE) for two UK based awarding organisations in Geography and Leisure, Travel and Tourism respectively. I also do a bit of setting, reviewing and scrutinising of question papers around the edges in geography-related subjects. Most of this is at GCSE level, but I dabble in some AS/A2 too. I am also an assessor for the Chartered College of Teaching’s Chartered Teacher Status programs where I assess a range of teaching assignments for degree programmes. Aside from these I work as an online teacher supporting learners at GCSE and AS/A-levels.
What first sparked your interest in assessment, and how did your journey into assessment progress from there?
Quite unbelievably I fell into assessment without any real direction or expectation! Having taken ill-health retirement from teaching in 2013, I was faced with the unenviable task of earning a living without being in contact with lots of germs, and marking GCSEs seemed a pretty good income so I started doing that. (I was born with Cystic Fibrosis – a degenerative illness which in 2013 had no genetic modulators to assist me in staying alive – these became available to me in 2020 and it is why I am still here.) I realised I was alright at marking (!) and was offered a team leader post within three years. I would sit in meetings for only a few days a year and it was much safer than my previous teaching roles. It just took off from there really. I did a few training events and then Mark Dawe (who was CEO of OCR at the time) introduced sponsorship for the Level 4 Certificate in the Principles and Practice of Assessment (CPPA) program offered by the Assessment Network at Cambridge (a precursor course to the Postgraduate qualification). I applied the moment I saw it. My one weakness is my love for certificates and the five degrees I had were simply not enough! I won a place and just like that I had a student card for the University of Cambridge! I did not see that coming but I have never looked back.
After completing the CPPA I did a number of setting roles for GCSE papers and then the Postgraduate Advanced Certificate in Educational Assessment came online. I enquired but by this point I was pretty unwell and travel to Cambridge was not something I could do easily or alone, so I shelved it. A year or so later I revived my interest because I felt I needed a deeper understanding of the assessment landscape. Plans were put in place for my husband to come along as my carer - and with oxygen in tow and FP95 mask on I would attend the day schools but it was going to be tough because my lung function was below 30%. Next the pandemic hit and I was locked in the shielding programme. Within weeks the website suddenly said all day schools would be online, and I knew it was my chance!
What were some of the most valuable insights you took from the Postgraduate course?
I feel really lucky that I did the qualification in the pandemic because it meant we had so many nationalities of people from around the world working in different countries and contexts – many of them were super interesting. Hearing their experiences of different educational assessment settings from around the world really helped my online teaching roles – where I teach students from different countries around the world on a daily basis. It helped me know what to say in those initial meetings about how UK assessments differed from what they were used to within their culture. I learnt more from them in terms of how to relate to students than I have in many an assessment talk or from reading around the subject. I like to think it has made me a more empathetic teacher and a more understanding assessment writer. It also gave me lots of new ideas that I had not seen in UK teaching.
This course made me an assessment thinker rather than an assessment doer."
The course itself let me explore breadth and depth and my assignments let me look at things I really needed to at the time to build my assessment literacy. I also was able to learn from some excellent people – Dr Simon Child, Tim Oates CBE, Stuart Shaw and Isabel Nisbet to name a few – who really helped me unpick the complexities of the assessment landscape through the pandemic and beyond. Their insights helped me develop all those basic principles in assessment of reliability, validity, fairness and comparability which had seemed such fixed concepts in my mind up to this point. I would recommend the course to anyone with an enquiring mind and a passion for moving their understanding of assessment from passive to active engagement.
I felt I needed this course. Data literacy was not my strong point, and I struggled on this course more than any of the other ones. I always thought I knew how to use Excel – well let me tell you, I didn’t! I am still unduly annoyed when I open a spreadsheet from a colleague, and they have used colour/highlighting to demonstrate data differences! If you know, you know. If you don’t, this is one of the many things this course will help with!
I have used this course most for understanding computer-based grading as a Principal Examiner and all those complexities of the statistics that teachers use in their Chartered College of Teaching assignments! Before I may have taken things at face value but now I nail down the nitty-gritty a bit more, therefore improving my assessment decisions. Statistical analyses can tell you a lot in a very short space of time within the bigger picture when working with schools, so understanding them at speed is vital.
As part of your MEd, you performed a critical analysis on the implications of moving geography exams onscreen. Could you tell us a bit more about this? What inspired you to conduct this analysis, and what were the key findings?
As a PE of the first geography specification to be moved onscreen I felt it was my duty to be aware of the consequences of this move. I generally am a bit of an assessment technophobe (mainly my age and my background rather than any major hatred of computers!) so I felt it was important to scope out the landscape within geography because we are quite a niche subject. We have maps and lots of source materials, and how this presents on a screen could add to construct-irrelevant variance pretty easily i.e. are we assessing what we want to assess or something else entirely like typing skills or how to make multiple windows work on a screen.
There is little research out there so far, so I wanted to see where geography teachers and assessors were at in their thinking to date. It was mainly a scoping exercise and perhaps the most interesting things I found were the absences in the data e.g. almost half of the participants felt a phased return would be most beneficial but no mention of mode effects changing outcomes was made. Mode effects are when people think less deeply when they read from a screen making outcomes different even with an identically worded question. Lots of teachers were worried about the digital divide changing outcomes for learners especially in centres of large cohorts and less wealthy locations. Lots of assessors were happy that handwriting would be less of a barrier to a candidate receiving their true score.
It really was data of two halves – for those learners on a computer day-in day-out, mode effects are removed and potentially the exam becomes more what they are used to, for learners who seldom use a computer then the mode becomes an instant barrier as they may not have had a mock onscreen, let alone have advanced typing skills needed for a high-stakes, timed assessment.
One really interesting finding was how would large cohorts using all the school computers for mocks and actual exams impact on the rest of the school during those times. Would this further add to the digital divide in some centres as Years 7-9 had no access at all to computers in this time?
What do you find most rewarding about your role?
As an assessor working with GCSE age candidates, it is so important to make sure the grades they receive are a fair reflection of their abilities.
I have amazing teams around me and we genuinely care about the true score of the learners sitting our exams. I have worked really hard to become qualified in educational assessment and to understand what makes assessments have value."
Bringing in new examiners to my teams and helping them understand the tenets of the world of assessment is very rewarding for me. Understanding why we award the marks we do and the principles that underpin our decisions is so valuable for them in their teaching roles as well so they can help their learners achieve their best.
Hearing that lightbulb moment when an examiner says to me, "that makes so much sense now, thanks for spending time with me to explain that" is what it is all about for me. That is how we create the next experts in our field and pass on what we know so far about the current assessment landscape.