So, yes. Well, we'll get started. In terms of what we hope you'll take away from this particular workshop, there are four main things. Fundamentally, we're assuming no technical knowledge in the realm of marking, so we’ll be going from the ground up.
The first aim is to help you develop your knowledge of mark scheme design – the language, the decisions, and the different elements available in mark schemes. Think of it like a palette of options you can work with. This will help you make informed design choices for your mark schemes or rubrics – terms we’ll use interchangeably here – with the ultimate goal of enhancing best practice in mark scheme development.
We’ll also explore the tensions and trade-offs involved, particularly in levels-based mark schemes. You’ll learn how to diagnose potential issues with your existing mark schemes and apply best practice principles to create effective and justifiable schemes that can evolve with your career.
The workshop has two main parts. First, we’ll cover mark scheme features – what options exist, different types of schemes, and real-world examples across various assessment areas, such as schooling and competency-based assessment. One of the benefits of this work is that, despite being from different subject areas, many principles apply universally due to the shared psychology of marking.
The second part introduces the COST model – a framework for thinking about five key aspects of mark scheme design. This model is designed to be practical and reusable in your own work. We’ll walk through each step using examples, with interactive activities throughout, both in the main session and in breakout groups.
To get us started, here’s a warm-up question. Tom Bramley, now Director of Research at Ofqual, conducted a study in 2009 on examiner agreement in GCSE and A-level papers. He looked at how often two markers would agree when marking the same piece of work. What percentage agreement do you think was found for objective items like multiple-choice or short response questions?
Feel free to pop your guesses in the chat box – from 0% to 100%. Some of you are already guessing – we’ve got a range from 50% to 100%. The actual result was 100% (rounded from around 99.97%). This high agreement rate is because these items have clear, objective answers and simple mark schemes, meaning low task complexity and minimal variation between markers.
Now, consider extended response items worth 10–20 marks. What percentage agreement do you think was found here? Again, drop your thoughts in the chat. There’s a mix of responses – around 35% to 75%. The actual figure was 75%.
That may seem high, but think about what that means – in one out of four cases, there was disagreement. If an examiner marked too harshly or too leniently, it could lead to the wrong grade for a student. This affects validity and fairness – which is why designing mark schemes that support reliable marking is critical.
This idea is well summarised in the following quote: “We may choose to accept lower levels of reliability associated with certain question types... where we believe the fundamentals of the question add value. But it may also be possible to make improvements – through refining the mark scheme or enhancing marker training.”
Ultimately, mark scheme design is about making small, thoughtful changes that can significantly impact fairness, reliability, and validity. That’s the core focus of today’s session.
Before we move on, we’ve got another activity to warm up your thinking. In your breakout groups – which we’ll move into shortly – please take around 10 minutes to introduce yourselves. Use your microphone and camera if you can, but the chat is fine too if not.
Discuss: What do you think a mark scheme has to accomplish? Consider both the technical elements (how it should be used by examiners) and the different users or stakeholders involved. Thinking about who uses the mark scheme can help clarify its different purposes.
This is your first breakout group task. Megan – are we ready to move into small groups?