00:00:00:02 - 00:00:20:03
Dr Simon Child
So. Yes. Well, we'll get started. So in terms of what you'll take away or what we hope you'll take away from this particular workshop are four main things. So fundamentally for this workshop, we're kind of assuming no or no knowledge or no kind of technical knowledge in the realm of masking. So we'll be going from the ground up.

00:00:20:03 - 00:00:42:06
Dr Simon Child
The first thing we'll be looking at is the chance to develop your knowledge of mark scheme design. So the language of the sorts of decisions, of the sorts of areas of mark schemes that that are possible, almost a palette of different options are available to you. So you know what your and what you're able to make decisions on in the design of your mark schemes or rubrics.

00:00:42:08 - 00:01:03:15
Dr Simon Child
I use those two terms interchangeably, but if, if I'm using the word mark scheme for this session, but rubrics are often used as well, really with the fundamental aim of enhancing your mark scheme best practice. So your eyes being as open as they can be to the kind of tensions that exist between the decisions, particularly for things like levels based mark schemes.

00:01:03:15 - 00:01:32:04
Dr Simon Child
So that's something that that you'll take away and really three and four are the kind of practicalities and the decisions that you're looking to make, diagnosing potential issues with mark schemes that you know or that you work with currently. And also I'm using the the best practice principles to create effective mark schemes and to and to justify them over the course of not just the means you're working with now, but over the over the course of your life and career.

00:01:32:06 - 00:01:54:07
Dr Simon Child
So the structure of the session is fundamentally two main parts. So the first one will be going through the Mark scheme, features the palettes that you have available to you, the types of mark scheme that exist. And I'll show you some real life examples in this workshop. Lots of examples from different areas from the schooling, competency based assessment and so on.

00:01:54:09 - 00:02:18:06
Dr Simon Child
One of the great things about Mark scheme design workshops in this area is that actually even though the Mark schemes and sales are in different subject areas, many of the principles are applicable across across more than one subject area. So it means that because we're dealing with, with the psychology of marking, that there are universal applications to take away.

00:02:18:08 - 00:02:40:07
Dr Simon Child
And the second part will be looking at the design of your mark schemes. So what I've called the cost model. So this is not acronym and essentially it's a way of thinking about five different aspects of Mark's scheme design with the intention of you being able to use this model again and again in your and your practice. So we'll go through each step of that model with some examples.

00:02:40:09 - 00:03:00:21
Dr Simon Child
And of course throughout the workshop there'll be some activities for you with either in the main room here or indeed in in a couple of breakout groups as well. So you will be, if you can use your microphone at this stage, there will be some opportunities to to share ideas and thoughts with, with fellow members of the workshop.

00:03:00:21 - 00:03:25:17
Dr Simon Child
So before we get going into some of the content for the session, I wanted to give you a quick question, just as a little bit of a warm up before we before we get going. So there was some research done by a researcher called Tom Bramley, who's now working at Ofqual as the director of research, and he did some research and around 2009, looking at lots of different examination papers.

00:03:25:17 - 00:03:49:09
Dr Simon Child
This is for GCSE and A-level qualifications, looking at different types of mark scheme and looking at the the amounts of agreement that there were between two different markers who were marking the same piece of work. So I wanted to ask you what you thought that the kind of the likelihood of agreement would be depending on different types of item.

00:03:49:09 - 00:04:23:00
Dr Simon Child
So what do you think for things like multiple choice questions, what are called objective items? So this could be either M6 or items that only have one word responses or very short responses. To what percentage do you think there was agreement between two examiners marking those types of those types of items? So give me a percentage of, say, from 0 to 100 of how much agreement you anticipate that would be and put that in the in the chat box, in the public chat.

00:04:23:00 - 00:04:54:04
Dr Simon Child
So Helen has already gone straight in but 90% Aleksandra Tom. Okay. So a Russell has gone as low as 50%. So I left of agreement 100%. So we've got a bit of a mix there between of 75 Rebecca's. But in the the caveat if standardization happened first. Yeah interesting point. We'll get to that later on. So there's a bit of a range that I think the lowest I saw was around 50% and the highest was was 100% and all things in between.

00:04:54:04 - 00:05:24:21
Dr Simon Child
So thank you for those those responses. The answer from this research at least, or the sample that was taken was actually 100%. It wasn't. It was rounded up to 100%. I think it was about 99.97, somewhere around that mark. And the reason for that is because even without standardization, it's likely that the Mark scheme and the the information that was provided in the Mark scheme meant that the the overall task complexity was quite low.

00:05:24:23 - 00:05:47:21
Dr Simon Child
So essentially all the examiners have to do is match the the letter or the words that is being produced by the students and match it with what is contained in the mask, even the mark schemes themselves and relatively short. So that meant there was a lot of agreement that and it's often only a clerical error of some description that would mean that there was some kind of unreliability within two examiners agreeing with each other.

00:05:47:21 - 00:06:25:18
Dr Simon Child
So that's a good case scenario for a different item type now. And these are extended response items worth between ten and 20 marks. So how much agreement do you think there would be between two examiners for for an item like this? Just your initial thoughts. So outcome for 20%. Okay. So we're getting 35 from Veronica. Okay. But I'd say if I had to give a very quick calculation of what's going through there is it's somewhere around 65 ish, 70% that people are going for.

00:06:25:18 - 00:07:14:21
Dr Simon Child
So Theresa and Sarah is going quite low at 45, Veronica at 35 as well. The actual answer is 75%. So when I revealed that, number two, you're possibly thinking, Oh, right, well, that seems a bit higher than perhaps I was expecting and so on. But if you think about what that actually might might mean for the quality of marking and therefore for the grading, if on the one out of four occasions that wasn't actually agreement between an examiner and the marker and the original marker, and there was in particular direction either marking leniently or marking harshly that can actually have some great influences or issues in relation to the validity of the assessment because it

00:07:14:21 - 00:07:39:10
Dr Simon Child
increases the possibility that a student has received the wrong grade. And even though the word they've produced could actually be of a of a higher quality than perhaps was originally judged by the by the examiner, It's really what we're trying to guess at when we design our mark schemes. And I think this was summarized quite well by this quotation here, so I'll just read it for you for, for completeness.

00:07:39:10 - 00:08:08:11
Dr Simon Child
But we may choose to accept lower levels of reliability associated with longer with certain question types such as the tenths 20 mark example shown here, where we believe that the actual fundamentals of the question type itself adds value. However, for some questions that do have lower levels of reliability, it may be possible to make improvements, for example, by refining the mark scheme or improving Mark or training that standardization point that that was raised earlier.

00:08:08:13 - 00:08:40:20
Dr Simon Child
So really what we're dealing with in terms of mark scheme design are making small incremental changes that can make a big difference in terms of the overall grading outcome of individual students, in terms of fairness, reliability and ultimately validity of your assessments. So really that's what the focus of this today. So before we move on to some more of the content, I just once again give you a little bit of a warm of activity just to think about perhaps in many ways the reasons why you're here.

00:08:40:22 - 00:08:59:14
Dr Simon Child
So in your breakout groups that that making will set up for for us in a minute or two, I want you to spend probably around 10 minutes. I want you to introduce yourselves to the group. If you if you can use your microphones and please use your microphones and camera if you can. But if you can't, that's that's absolutely fine.

00:08:59:16 - 00:09:25:07
Dr Simon Child
And you can use the chat function. If it's microphone isn't available to you. So introduce yourselves and discuss in your groups. What do you think a mark scheme has to accomplish? What do you think it's there to do? And I want you to think of this in terms of the kind of the technical elements. So how how are examiners meant to use it, but also to think about the different or is off the mark scheme as well?

00:09:25:07 - 00:09:45:10
Dr Simon Child
Who's he's going to be looking at the mark scheme and using it for some reason. And when you think about the different stakeholder groups that may actually give you some different ideas about what the purposes of a mark scheme are. So this is the initial breakout group task. Megan, Are you ready for us to move in small groups?