Video transcript
[00:00:00] Hello and welcome to this presentation. As part of Cambridge Assessment's Summit of Education 2019, which is on the topic of anticipating the future of learning. My name is Rachel Bateman and I'm a senior assessment advisor at Cambridge International. And in today's presentation, we're going to be looking at how we can use technology to improve the feedback that learners receive in the teaching and learning process. We'll be looking at the role that technology can play and we'll be looking at some real life examples from the work of Cambridge Assessment. And through those, we'll be thinking about how it's used and some of the issues raised by the use of technology. We'll be joined today by three colleagues from across Cambridge Assessment. Michelle North from OCR, Tom Booth from Cambridge English and Jo Scott from Cambridge International. To begin with, let's think about the context in which feedback is produced and received and used. If we saw this visual as a representation of what we might describe as a feedback loop, then it would be showing us a learning journey that would consist of setting out learning goals, learning taking place, eliciting some sort of evidence of the learning that's taking place, and then there it needs to be interpretation of that evidence. Through that interpretation, there's a comparison between what the learner has produced and what we intended the learning to be producing. And that comparison leads to the identification of gaps between those two things. And those gaps form the basis of the feedback that the teacher receives about the learning that the teacher provides to the learner, which should in turn form the basis of the setting of new actions of driving the learning forward to the next stage. And when we're looking at technology in this process, we shouldn't move away from this sort of model that we're familiar with in the teaching and learning process in general. In the examples that we'll be looking at in today's presentation, you can start to think about where in this process is the technology having a role? Is it in, for example, interpreting the evidence that the learner has produced? And on what basis is that interpretation being made? What's underpinning that interpretation? Is it based on secure subject understanding? It might be that technology has a role in the identification of gaps between what the learner has produced and what the objective of the learning was. It might be that the technology is involved in actually providing feedback to the learner to get some sort of mediator or go-between between the teacher and the learner. It might be that the technology has a role in setting new actions in showing the learner where their next steps are in relation to progressing in the subject area. Now, research has shown that when feedback is used well, it can have a powerful effect on a learner's progress, but not all feedback works. So let's take a moment to have a look at some of the characteristics of effective feedback. And when we're thinking about the use of technology in this process, we should still keep these characteristics in mind because they're still just as relevant. We've already mentioned the importance of subject expertise underpinning the interpretation of what the learner has produced, underpinning the provision of the feedback in working out what the most effective pathways are to overcome misconceptions and to progress. There's also the importance of the expert knowledge of the student, of where they're coming from, of where they're going to in their learning, of what's going to motivate them, of what they need right now. And as well as this knowledge of the student, the trusting relationship between the teacher and the learner is crucial because otherwise it's unlikely that the student will or the learner will want to invest the time in wanting to progress. This trust between the teacher and the learner, this trust that the learner has in the feedback they've received is really important. And as well as this, there needs to be a focus on the students using the feedback, on building time into lessons and building habits for the students to reflect on the feedback they've received and to implement it, possibly even looking back at feedback they've received in the past. So now I'm going to hand over to my colleague, Michelle North, who's going to talk to you about a project that OCR has been involved with.
[00:05:49] Thanks, Rachel. So as part of this presentation, I'm going to talk about two projects that OCR worked on with Shireland Collegiate Academy. This was actually part of a bigger relationship looking at the use of technology in schools and in education with Shireland. And so there are actually 18 projects that we collaborated on. And you can find out more about those projects at the link at the bottom of the slide. The two projects that I'm going to talk about today focused on the use of video and audio feedback in the classroom. And really for both of these projects this was built on an acknowledgement that whilst feedback is critically important to student progress, there are some key limitations of traditional bookmarking. So in particular, whilst it is necessary to mark student books, of course it's also time consuming. It can be quite a repetitive process for teachers. And actually often what we find is that those written comments on students work can be overlooked by the student or not fully digested. And I think part of that is to do with the method of delivery. Students just aren't used to engaging with the handwritten word. That can be a barrier for them, and particularly for students who perhaps have lower levels of literacy or who have English as an additional language, that can provide more barriers to them engaging with the feedback, regardless of the quality, the time, that the teachers put in to providing that feedback. And therefore, that limits students progress. So for both of these projects, this was looking at how using technology can actually improve the engagement with the feedback. So the first project that I'd like to talk about is using video feedback. Really, I'd just like to describe the process that the teacher went through here. So this is a project that was actually focused in the maths classroom. And so what the teacher was doing is he would start off having received homework in from his students in going through a handful of books to start identifying the common misconceptions that the class had with those particular activities. And having done that he would then proceed to making some videos, and as you can see from the first screen grab, this is not perhaps as complex and technological as you might think it would be. So literally what he's done here is taken his iPad, turned it upside down, had a pile of books or a couple of boxes either side of him that he's resting the iPod on top and simply has a piece of paper underneath and a pen. And what he's doing is recording himself working through particular problems and describing the different stages of those problems, what it is that students were perhaps not understanding, were perhaps getting wrong and to sort of talk them through those issues. And so he'd record maybe three, four videos that dealt with the common misconceptions he'd found in the students work. Having done that, what he's then done is created a spreadsheet as you can see in the middle image with the class list on one side, the different videos that he's created and the links to those on a YouTube channel that he'd set up, and then he's just indicating in the spreadsheet for each student which video relates to the problems that they have had in completing the homework. Having done that, what he's then using is the school system for providing feedback electronically to the whole class. And this is where I think the real time saving comes in, because rather than writing in 30 plus textbooks at the same degree of feedback, you've got one shared piece of general feedback which he can issue to every student in the class. So as you can see here, and I think if you were a maths teacher, you'll relate to this comment, you've probably written it a thousand times. The first comments is show your working out in full. So rather than having to write about 30 times, it's just issued once to the whole class. There's lots of general comments there about the approaches to work and perhaps some comments about literacy, as in this case. And then for the individual feedback, it's simply a case of here's the link to the spreadsheet, find your name, where there's a yes in the column, that means that that particular video is the one that you need to look at. And then that gives the student the opportunity to click on the link, watch the explanation, and hopefully understand what's being said. Now in terms of the delivery of this feedback, what he found in doing this project was that students still preferred to have the feedback issued to them for them to engage with that in class. And I think there are a couple of reasons for that. Probably the most important one is that what we're not doing here is replacing the relationship that the teacher has with students. And so what's really important is that the students have the opportunity to sit in class, headphones on, watch the video on their devices, but they can still actually have the teacher's attention and express the fact that perhaps they still don't understand or there was a little bit that they just need exemplifying and a bit more detail and then the teacher can come over and unpick those particular students' needs in real time. And I think that's really important. So whilst this is a time saving activity, it's definitely not something that is about replacing the teacher relationship. In terms of the impacts of this particular project, what I found interesting here was that there was no difference perceived between students of particular mathematical ability, so this wasn't a process that was preferred by high or low achieving students. Nor was there any particular gender difference. So across the board, what the teacher found was that this improved student engagement with their feedback, improved their ability to understand the feedback, because they could enrich the explanations with his time if that was needed, and that led to overall a greater level of progress in maths over this period than he had found previously. And then the second project I wanted to talk about is a similar but subtly different project. This was one rather than being carried out in the maths classroom, was more in the humanities end of the curriculum, and rather than using video feedback this focused on using audio feedback. So this gives more personalised feedback to individual students than the previous research, but here minimising the requirement for the students to engage with the handwritten word. So what what the teacher was doing here was reviewing the students books, but rather than writing at the bottom of what the student had produced, the teacher was recording feedback in a MP3 formats and issuing that to the school's virtual learning environment so that the student can access that individual audio feedback file. So rather than providing shared feedback to the class and the shared videos this is is closer to the traditional method of handwritten feedback, but is is getting over that barrier of the students' ability to engage with the written word. Again, I think this is particularly important where you do have those students with lower levels of literacy, but also for students with English as an additional language. The advantage that you've got here is that the student can obviously listen to that feedback as many times as they want to. It's a really useful resource. In fact, both of these projects provide really useful resources for when it comes to revision time because students have an archive of problem-solving, of improving their own personal profile based on what the teacher has told them. Also, if you've got students that have English as an additional language, there's an opportunity here for them to work on their language skills because they can actually transcribe the feedback themselves. And that gives them an opportunity to identify perhaps words that they don't understand. They can use programs like Google Translate so that they can engage with the feedback in their own language. Again, it's important to note that receiving the feedback in this way doesn't replace the relationship with the teacher. And again, what the teacher found is that this was more impactful when students were allowed to engage with this feedback in class and there was this clear opportunity, having immediately heard what the feedback was, to complete some kind of activity, where the pupils implement that feedback. And again, having the teacher on hand able to deal with any issues and provide additional support as required was really important to the student progress. But having recorded the feedback for the students, that frees up the teacher in that precious lesson time to be able to circulate the class rather than be stood at the front and issuing generic feedback to the group. So for both of these projects, I think there's potentially a time commitment in terms of understanding and making sure that you have the systems in place to be able to deliver the feedback by these methods. But certainly once you've overcome that first investment, I hope you can see that there would be an opportunity to save time by using technology rather than the repetitive handwriting of traditional class feedback. As I said at the start, these are just two projects from 18 bigger projects that you can find on our website. So there's a a short link: ocr.org.uk/shireland. Please do go and have a look. There's plenty of videos, packs and also information about how you can pick up the process of action learning to improve and to reflect on the practice within your own contexts. Thank you very much. And I'll hand over now to Tom.
[00:15:38] Hi, my name's Tom Booth and I'm from the new product development team at Cambridge Assessment English. Now, clearly, feedback is a hugely important area in language teaching and language learning, as it is in education more generally, and certainly in digital education. And luckily, there's quite a lot of research literature around what makes effective feedback. So I'm going to highlight a few key themes from literature around what makes good feedback and in doing so I'll also try to highlight a few examples of work that we've been doing at Cambridge English. So the first thing to say is that there's a lot of research which has shown that effective feedback can really move learning forwards. Feedback really can make a difference. But it's not true to say that any feedback is better than no feedback. In fact, it is possible to do harm with feedback. Some kinds of feedback can actually have a negative effect. So what kinds of feedback tend to be effective? Well, we think that the research tends to suggest that generally feedback which is specific tends to be quite effective, and feedback which is actionable, i.e. feedback which the donor could actually take action on tends to have the most effect. That means that feedback needs to help the learner understand where they are in relation to where they want to be and how they can bridge that gap. Some kinds of feedback which tend to be much less effective are things like when a teacher or when an app or when anything just routinely corrects every possible error in a student's piece of work. That tends to be less effective because often some of the errors may be above the level and you wouldn't expect the learner to not make that error at the level they're at. But also when every single error is corrected often it's very hard for the learner to know where to start in taking action on that feedback. Something else that tends to be less effective is general praise. So things like well done or you're great. They can be less effective. They can actually be detrimental because the learner doesn't know how to act upon that feedback. If then it doesn't know why they've been told well done or what specifically was good, then they don't know how to act upon that feedback. And yet we often find apps for language learning or digital products for language learning sometimes exemplify exactly the kinds of feedback that we know from research are not very effective. So here are three examples of quite popular language learning products. And these are all fantastic in their own way, but I would suggest that there is some room for improvement in all of these in the area of how they present and how they use feedback. So either they just say correct or wrong with no kind of actionable insights or anything that the learner could actually do to improve, or they say something like brilliant or they give a score without any kind of explanation why. Or they say good job, and there's nothing there for the learning to actually act upon. So hopefully, by contrast to that, the prototype of something that we've been working on, this is something that we worked on late last year and early this year. You can't find this anywhere. It hasn't been published anywhere. It's just a prototype that we've tested. And the aim here is to give someone give a learner an idea of their own strengths and weaknesses in English. So there is no score, there's no grade. Instead, the learner is given a visual indication at a glance of which skills they're stronger or weaker in, this one is doing okay at reading and listening, but they maybe need to work on use of English. And what I want to highlight here is towards the bottom left where it says here some activities you can try. So based on their performance, the learner can click through and there are some free activities that they can use to practice the areas they need to work on. So this learner, if they clicked on this link, they would find some grammar and vocabulary activities. And as they're not so strong in this skill, they'll get some slightly easier activities. If it was a skill they were stronger in, they would get some more challenging activities so that the activities that they're presented depend on the skill they're practicing and also their ability in that skill. So that leads quite nicely onto the idea of feedback loops. This is another key theme from the literature. Lots of people have written about the idea of feedback loops, where a learner is given some feedback but feedback on its own or just information by itself isn't enough to really be effective feedback. There needs to be this concept of a loop where the learner can take action and bridge the gap between their current state and where they want to be. An important part of this idea of feedback loops is the idea of trying again. And that's an important part of this product, which I'll show you here. This is a screenshot from a product called Wight and Improve and Write and Improve is a free product on the web, which was a collaboration between us at Cambridge Assessment English and some other colleagues in the university. So the idea is that a language learner for free can come to this website. There are various writing tasks they can try and they can attempt the task and submit their writing through the web page. And they get feedback like this. So here's an example of the kind of feedback that a student gives. It uses an artificial intelligence algorithm to generate this feedback. But what I want to highlight here is that the way the feedback is presented is really based around the idea of trying again. So rather than simply saying this bit's wrong or just correcting it, rather than crossing it out and correcting it, instead it prompts the student to try something different. So, did you forget something before this word? The learner can try something else, try to make the writing better and then resubmit it and see how making changes to their writing make a difference to the feedback they get. So that's the concept of loops. And I think this idea of feedback loops is very powerful. But one other thing to mention about that is ideally, um, it shouldn't just be a case of learners getting feedback on something, improving that and then getting feedback on something else and all these loops being completely separate from each other. Ideally, we really want the learners to be able to make connections between the bits of work that they do. So, for example, they might be doing one piece of work and they might think to themselves, oh, maybe the feedback I received several weeks ago might be relevant here and through that kind of level of development, above the level of individual loops, learners can develop at the level of self-regulation. And that's something that David Carless has written about. He describes it as feedback spirals. And I wonder whether and how technology can support and facilitate learning and development at the level of self-regulation going beyond individual feedback loops. And that's an open question which I don't think has been answered. And that leads us on to the last of the kind of things that I'd like to just highlight and pick up on from the literature, which is to say that feedback doesn't exist in a vacuum, by which I mean that to judge how effective a piece of feedback is, we need to see that feedback in the context of the learner receiving the feedback and the relationship between the learner and the person or the technology or whatever it is that gives them the feedback. So feedback really is an interaction between the learner and the giver of the feedback. One important thing to bear in mind here is I think if you think of what good teachers do, if two learners were to submit identical bits of work, a good teacher may well give those two learners quite different feedback to each other, even though their work was identical, because the feedback wouldn't just be based on that particular piece of work, but it would be keeping in mind the kind of learning trajectory of the student and the learning needs of that student, where they've come from, what they've been learning recently, what their goals are and so on. And so another really important, I think, open question to think about is how can technology contribute to that? What role can technology play in delivering feedback to learners, which moves learning forward in a way which takes into account and is conscious of learners' trajectories and learners' goals and learners' needs. And that, again, is an open question which I think is worth thinking about and discussing further. On that note, I'm going to hand over to my colleague Jo, who is from Cambridge International.
[00:25:06] Hello. We now have the opportunity to give you an idea of the way in which we use technology as a tool for feedback in our professional development qualifications. Cambridge PDQ is a practice based qualification for teachers who have been in the classroom for at least a year. The idea is that they have the opportunity to understand what it means to be an effective teacher, but then the PDQ helps them to develop that practice as they move on. It's a qualification structured around a spiral of learning and structured really into three main areas. So candidates go through guided learning led by a program leader who is trained by us. They then take the knowledge from those guided learning sessions and practice it in their classrooms, in their work based learning. And then all of this is supported by a mentor who helps them to reflect on what they've learned and helps them to compile the portfolio that then comes to Cambridge for examination. We offer the PDQ globally across four syllabuses: teaching and learning, which is to do with good foundation pedagogy, teaching with digital technologies and teaching bilingual learners, two separate qualifications which build on the knowledge of foundation pedagogy, but then lay on the different strategies that you might need to teach well with digital technologies and using bilingual strategies. And finally, we have a syllabus in educational leadership. It provides quite a broad base and our remit really within the PDQ team is to make our courses as accessible as we can for people around the globe and for Cambridge teachers around the globe. And so in order to do that in areas where it is not possible to have PDQ centres, we've started to take the PDQ online. And it's meant really that what we've needed to start to think about ways in which we can give feedback in an online sense because not all of our programmes are now delivered face to face. I mentioned at the beginning that part of our quality framework is the fact that we train a programme leader to deliver the programme in itself and in the centres, and we operate this training online. So we need to be able to demonstrate how we provide feedback within that online environment. And what we're able to do then is replicate and demonstrate how you might do that if you are then indeed an online programme leader. Some of the courses as time goes on have become blended. So where it's not possible always to have everybody in the classroom face to face in terms of the teachers who are following the PDQ, then also in some cases the programmes have an element of blended delivery, say some parts are face to face and some parts are completed online, particularly in areas where it's geographically difficult to get people together, then it offers an opportunity to be able to deliver the PDQ regardless of some of the issues that come up. And finally, in order to create maximum reach, we've recently taken the PDQs online. This provides a whole other range of issues in terms of giving feedback and creating a working dynamic for these candidates within this environment. On the next slide, I'll start to talk to you about how we've been able to do that. Of the three examples that I gave on the last slide, we use a variety of ways of offering feedback to candidates or to programme leaders. The first during our programme leader induction course is an example of a tool called Padlet, and it provides quite a lively way of programme leaders being able to post the work that they're doing and for both their colleagues and the facilitator to be able to comment upon it. You can see here that they've been set a task to think about what an effective professional development program looks like. They've gone away, found information, posted it on the padlet wall, and the facilitator then is able to read their examples and provide feedback for them. And this offers opportunities for her to stretch and develop their skills and their understanding and to encourage them, to challenge them, even to think about different things in their programme and think about different ways of approaching them. The nice thing is as well is that they can comment on each other's posts. So it's a truly collaborative environment which enables them to start in the way that they mean to go on. And sometimes this is a tool then that they replicate in their own delivery of the PDQ. In our training of programme leaders, we also use webinars a lot. So this is a way for us to be able to provide training, but also it's a way of being able to have a two way conversation in an online environment and in an asynchronous format. So it offers the opportunity for people to talk to one another in real time, but equally offers the opportunity for people to watch off-line and then raise questions perhaps in a forum. And that brings me on to the final example in our online PDQ. We rely heavily on forums, so the guided learning aspect of the course is online and ready for learners to be able to interact with whenever they want to. Because obviously this is a global qualification so time differences are an issue. They're able to interact with the material, but then reflect on their learning in the forums. And you can see here again, this is quite a lively opportunity for people to feel like they're speaking to one another, even though they're not face to face. But it also provides the opportunity in this case for the programme leader to be able to go in and structure the debate and challenge the debate and stretch the debate as well.
[00:30:51] So in today's presentation, we've seen varied examples of how technology can be used in the feedback process. We've seen how it can act as a mediator of the teachers feedback. It can present that feedback to learners in ways that are more engaging, that are more novel, that are more in keeping with how they interact with written text at the moment. And it can help them to overcome barriers to learning that may exist in different contexts. We've also seen how technology can be used to provide quite instantaneous, personalised feedback based on what the learner has done in their work. And that in the example we've been looking at has also enabled the learner to see what their next steps are and what they can do to achieve those next steps, what they can do to progress. In the last example we've seen, we've looked at how technology can enable feedback to be delivered at a scale that wouldn't otherwise be possible when learners are working in remote situations in an international context. But as Jo has touched on, that also brings challenges, particularly when we're thinking about the importance of the relationship between the teacher and the learner, between the mentor and the learner. As well as those issues we've touched on the importance of subject expertise underpinning the feedback, it needs to be based on secure subject understanding to see how the learner can overcome their misconceptions and can progress through a subject area in the most effective way. There are resource implications as well in terms of potentially the cost of the use of technology, the time needed to be invested in its use or the time saved from the use of technology. How is that time then reinvested in that teaching and learning process? And there are implications for trust in that relationship because it could be seen that the use of technology can lead to the relationship between the teacher and the learner becoming more distant. How are we reinvesting the time saved in terms of keeping that strong, trusting relationship between the teacher and the learner, which researchers suggested is a key characteristic in providing effective feedback. To close the presentation today I wanted to give you time for reflection on the things we've discussed. There are three questions on screen that hopefully will prompt your reflections and allow you to think about how applicable they are in your own schools and organisations and settings. To what extent could the examples we've shared with you today be used in your own schools and organisations? How can technology deliver feedback that moves learning forward and takes account of learners starting points and even where they're going to in their learning? And how can the ways we use technology help learners to take control of their own learning and to become increasingly self-regulating in the way that they reflect on their learning, in the way that they use feedback to progress across a subject area? Thank you very much for watching this presentation and for your time today.