Video transcript
[00:00:01] Right. Thank you very much. My name is Matthew Carroll and I'm a research officer here at Cambridge Assessment. Today I'm going to be speaking to you about work I've carried out over the last year. I've been looking at how longitudinal survey data is used in education research. To give you some background to this story, we know as education researchers that there's a whole range of big questions that people are always chipping away at. These are things like: What factors can influence people's educational outcomes? What influences the different routes people might take through education? What are young people's opinions of the education system and what are the effects of new policies? To answer these big questions, we need data and we need it on a range of things like family background and demographic variables. We need data on things like the subjects people take, the results they gain and the qualifications they gain. And we need data on more personal things like people's opinions and their ambitions. And in all of this time is a key element because we're thinking things like: how to early conditions affect later outcomes? How do things change throughout someone's school career? How do things change between years? So not only do we need data on a range of variables, we also need data from multiple ages, multiple stages of education or multiple years. So where do we get this from? Well, there's a range of places we can get data from.
[00:01:29] One of the common things we use is administrative data sets. These are things like the National Pupil Database, which is collated by the Department for Education. But there's a whole range of data sets that are collected routinely as part of the education system. And these can be really useful to carry out research with. They are comprehensive and they can have repeated observations of the same individuals over time. So that can be really useful. But we have to remember, they're not actually collected for research purposes. And this brings in some limitations. For example, they only collect data for a limited range of variables, only those that are really necessary for administrative purposes. The variables they do collect can change over time, or even the definitions of variables can change over time. And of course, data is only collected while the people are in formal education. As soon as they leave, they go missing in the data set. Now, we could also go out and collect data ourselves, carrying out surveys. Now, this can be really useful. We can target the survey questions so that we can answer our focal research questions. What this means is we acquire data on things that administrative data sets wouldn't collect, things like people's background, their opinions and so on.
[00:02:45] Of course, this isn't necessarily restricted to individuals in a particular system. But particularly when we're thinking about looking at changes over time, this can become a bit trickier with a survey. For example, if we wanted to look at how things change between the ages of 11 and 16. Well, if we could do that in a single survey, we need to go out and survey some 11 year olds and some 16 year olds. Well, the problem then is that any conclusions we draw are there because of the difference between 11 year olds and 16 year olds. Or is it because we've sampled two different groups of people? So what we say is that age or time in a single survey would be confounded by sample composition, which can introduce some challenges. An alternative, though, is to carry out a longer juvenile cohort study. Now, these are where a cohort of individuals is picked at the start and these are surveyed repeatedly over time. So what this means is that we get all the benefits of a survey, but we've got the same sample of individuals over time. So we're not going to have this confounding effect. These are huge undertakings, and so they're often linked to administrative data sets, too. So not only do we get the benefits of survey data, we can also get some of the benefits of administrative data sets if those links have been made.
[00:04:06] But because these are such a kind of huge undertaking, they're very logistically and financially challenging. to run. So although there are a few of them out there, they're relatively rare. And so for the present study, we came round to thinking that, well, as a research division, we're interested in questions about education and assessment. And we often use administrative data. We often conduct surveys, but, could we use data from longitudinal cohort studies in our research. Now, we could have gone out, started conducting research, but before we started we had some questions, and these were: What types of research are these datasets already used for? What are people already doing with this data? What are the types of methods people use to work with these data sets. What are the limitations. What should we not be doing or what is it challenging to do? And then looking forward, what else might be done? What are the options for working with these data sets. So to address these questions, we decided to carry out a literature review to look at publications that have used data from longitudinal cohort studies. And so for the rest of this talk, I'm going to be taking you through this literature review. Right. And so to carry out this research, you decided to focus on two studies in particular, and we focused on these because they are very large scale. They are recent or ongoing and they focus on young people.
[00:05:31] So I'll just give you a quick description of the focal studies first. The first one we looked at was Next Steps, which is also known as the longitudinal study of young people in England. The other one we looked at was the Millennium Cohort Study. In terms of who runs them, who owns them? Well, Next Steps was started by the Department for Education and it was taken on by the UCL Center for Longitudinal Studies. Whereas the Millennium Cohort Study was started and is run by UCLA Center for Longitudinal Studies. In terms of the geographic focus, Next Steps is focused just on England, whereas the Millennium Cohort Study is the whole of the UK. Next Step started in 2004 when the participants were age 14. Then there was a survey every year until age 20 and there was a follow up at age 25. The Millennium Cohort Study started in 2001 when the participants were just nine months old. Then there's been a survey sweep every two to three years ever since then. And that's ongoing. In terms of who the respondents are: young people were the focus of Next Steps of every sweep, and parents were also included in the first four sweeps. The Millennium Cohort Study, as it started when there were only nine months old, the young people have been included from sweep two onwards. Parents have been included in every sweep so far. And teachers were included when the children were at primary school. In terms of how many respondents they work with: Well, next step started with over 15000 and Millennium Cohort Study started with over 18000. And we see some kind of what we call sample attrition over time where simply people stop responding. So the sample does decline over time. But we've still got thousands of respondents in each survey sweep. In terms of the methods they use, they both these questionnaires and interviews. And as mentioned earlier, because these are such big undertakings that can be linked to administrative data sets and in both cases they are linked to the national pupil database and that's every sweep for Next Steps and some of the sweeps for the Millennium Cohort Study. So we picked those studies to work with and then we wanted to know what had been done using data collected by them. So we carried out a literature search. This was carried out in winter 2017, spring 2018 and was carried out using Google Scholar and Web of knowledge. And in these search engines, I searched for the survey names plus words like education or NPD to try and refine the results just to studies relevant to the field of education.
[00:08:05] I also looked at the Center for Longitudinal Studies website because they've got a list of publications and repositories of working papers. That's because they are the main player in this. When I was reading the papers, I also traced any references that I hadn't found using the search. So in total, I managed to acquire 61 papers and I read all of these and I made notes on the research questions they were asking, on the methods used, and on the findings. And then I summarised these notes to identify the broad research topics that this is trying to identify, the types of questions people are asking. I identified common research methods. I considered the limitations - what was challenging or what was not possible. And I also considered the future possibilities. And so for the rest of this, we're gonna be thinking about these broad summaries rather than the findings of individual papers. So first, think about the broad themes that I identified and I identified seven. The first of which was aspirations, expectations and outcomes. So these are papers asking things like what do young people want to do in work and education? And then what do they actually go on to end up doing? And just as a couple of examples of the types of paper in this category, Moulton et al, who looked at how children's career aspirations were influenced by family, social class, and Benton looked at whether taking entry level qualifications influenced young people's aspirations, influenced what they wanted to do.
[00:09:42] The next category I identified was attainment and progress. And this was asking questions like what influences attainment? What influences progress? And it also included the same questions about cognitive test scores. Because bear in mind, particularly for the Millennium Cohort Study, the children through most of the sweeps for which data were available were quite young and hadn't necessarily sat standardised tests. So cognitive test scores were also used. Just as a note here, attainment was included in pretty much every study I looked at in some form. But these papers were put into this category because they explicitly focused on attainment. And just again, a couple of examples: Chanfreau et al looked at how extra curricular activities influenced standardised test scores in primary school, and Strand looked at how ethnicity, gender and social class interacted to influence attainment at GCSE. The next one was subject choice and this is looking at the subjects people study. So what influences their choices and how do their choices affect educational outcomes? And just as a point here, all of the papers categorised in this category were using Next Steps data. And that's because the Millennium Cohort Study data linked to GCSE data has only very recently become available. I'm sure in the future this could also be used for similar questions. But in this category, this is all Next Steps.
[00:11:08] So again, a couple of examples. Henderson et al looked at how GCSE subject choices were influenced by family background and Anders et al looked at the next step, how your GCSE subject choices may influence whether people apply to and indeed attend university. The next theme I identified was based on setting, streaming and perceptions of ability. So these were papers asking questions like what influences the placement into ability groups like sets and streams and what are the effects of ability grouping? And again, just to point out, these were all carried out using data from the Millennium Cohort Study. That's because the teacher questionnaire, which was put out while the children were in primary school, asked explicitly about ability grouping. So, for example, Hallam and Parsons (2013) looked at how prevalent streaming was and what characteristics were associated with being in different streams. And Campbell (2017) then kind of looks at the next step, saying does stream placement influence how teachers perceive ability. The next topic was a kind of broader look at schools and educational policies. So this was a fairly broad category addressing various questions relating to schools or to educational policies. So, for example, Flouri and Midhouhas (2016) looked at how school, academic and socioeconomic composition influenced children's behaviour. And Hamden-Thompson and Galindo (2017) looked at how having a good relationship between parents and the school could influence children's academic achievement. The final two categories. Well, the first was socio economic disadvantage.
[00:12:54] Now, a bit like attainment, this was included in a whole range of papers, but some papers explicitly looked at it, looking at how it occurs, what its impacts are and how it's measured. So Layte, for example, compared different theories as to why working class children can show lower attainment in school. And Ilie et al. looked at how effective different indices of socioeconomic deprivation were, so the different ways it can be measured and described. The final category was on behaviour and personality. Now the point out here is a very common research area using these kinds of data sets, but because I was trying to relate this to education research, these papers only were those that included behaviour and personality in the context of education. So just for example, Attwood and Croll looked at how truancy linked to well-being and then went on to look at how that may impact educational outcomes. And Symonds et al. looked at the factors that may be associated with when young people disengage emotionally from schoolwork. So those are the seven themes identified, but we can also draw some overall summary points from those. So first, I'd say that some areas are very well studied already using administrative data sets, and these are things like attainment, socioeconomic deprivation and so on. These are very widely studied. So the longitudinal survey data set we used to access the really rich background data, the demographic data, this was things like parental education, family income and so on. And these variables, these new variables were used to further explore relationships are already known about, or even to look at completely new relationships. For example, the paper I mentioned looking at the effects of extra curricular activities, those kinds of things wouldn't necessarily be possible using administrative data. Some of the themes though, just couldn't be studied using administrative datasets. These are things like behaviour and personality or ability grouping, because the data sets are then used to access variables that are just not routinely collected. These are things like aspirations, risky behaviours, opinions and strengths and difficulties and so on. Another point to make is that some of the themes involved questions at quite an individual level thinking about things like attainment and subject choice, but others, scaled the data up to ask questions at a system level. So these were addressing questions about schools or regions or policies. So even though the data is collected from individuals, it doesn't mean the questions have to address kind of individual level themes. You can look at big scaled up questions as well. So if those are the questions people are asking using these data sets, the next thing I wanted to think about was how they go about asking the questions, what do they think about the methods people use. And so I made some notes on the methods and made some overall observations.
[00:16:03] This is just a brief run through some of these observations. So first, most of the studies sought to identify associations with the key outcome variable. So they're gonna pick an outcome variable of interest and this could be something like attainment or subject choice. Then it would look at the various background factors or demographic factors associated with the outcome. This is often done with a two stage methodology, so people will calculate descriptive statistics and then fit a multiple regression model or a series of regression models to account for various background factors at the same time. What this means is that many of the survey variables are mostly used as controls in regressions, and the focus of the study is on identifying fairly large scale associations. So this is quite similar to the way we would analyse a big administrative data set, for example. Related to this, many of the studies use data from just one or two time points, so they would take the background variables from the first sweep and then the outcome variable from the age of interest and use those in their regressions. Again, this is similar to how we would analyse an administrative data set or a single survey. So not too many studies utilise the fact that there could be some variables that are measured again and again and again for the same people.
[00:17:22] Some studies did, however, use more complex aspects of the data, and they used this to look at things like mechanisms underlying the associations or changes over time. But these did necessitate more complex statistical methods. So, for example, structural equation models could be used to examine all the different interactions and links between multiple variables. So, for example, Gutman and Schoon looked at how uncertainty and aspirations affected attainment. Latent class analysis was used to derive trajectories of responses to look at how things change over time. So, for example, McCulloch looked at trajectories in aspirations to attend university and duration modelling, or survival modelling, as it's known, could model transitions between different states. So Anders, for example, looked at how socio economic status could influence whether people were likely to apply at a university or not likely to apply. So more complex aspects of the data can be used, but it does require more complex statistical methods. So if that's the type of methods people use and that's the type of questions people ask. We also need to think a little about maybe what is challenging with these data sets. So just a few things to mention. First is that these studies are huge undertakings and by their very nature take a long time to collect and process and analyse the data. But education can change incredibly rapidly. So if we think Next Steps, for example, the data for that start to be collected in 2004 and everything of the changes to the education system in England since then, we've had reforms to GCSE, we've had reforms to A-levels and vocational qualifications.
[00:19:04] We've had the introduction of EBacc. We've had the introduction of Progress 8. We've had some changes to post 16 education, it's now compulsory, and we've had changes to higher education fees and funding. So huge changes since 2004. So we have to consider very carefully whether the data is still applicable to the current situation. Now, in many cases it will be. But we do have to be careful to think, does the data still apply to the current situation? The next thing to think about is how to deal with complexity of the data. Now, these are huge datasets, we've got lots of variables, lots of respondents, multiple years. It's a complex dataset and it's challenging to deal with. Some people use multiple regression as mentioned. But one of the downsides of this is when a lot of variables are included as controls, we're kind of losing some of the interpret-ability about each particular variable. Some people try and reduce the complexity by calculating an index or using something like principal components analysis to effectively reduce the dimensionality of the data. But again, we're going to lose some of the definition. So one way forward with this might be to consider the world of big data and look at kind of statistical methods intended for many predictors. But whatever happens, it is still a challenge to use this quite complex data. And the final limitation is in effect the strength actually. As mentioned, these servers are linked to administrative datasets, particularly for education the link to the national pupil database. Many studies utilise it and it is a real strength.
[00:20:50] But we have to be aware that not every sweep is linked to the NPD and increasingly people are cautious about using administrative data in research. So we know, for example, that the NPD is no longer available as it used to be. So although it is a real strength that the administrative data sets are linked, it is probably also worth considering how to work more with just the variables collected within the surveys themselves, because it may not always be the case that these linked datasets are available. And then the final thing to think about is where future research might go with these types of data sets. Now this is just some personal opinion, so you may have your own thoughts about this, but some observations I made just about where we might go first. I think there's real potential to look into some of the mechanisms that underlie these large scale associations. As I mentioned, many of the studies used regressions and effectively identified these big associations. But some of them, maybe the ones using more complex statistical methods, can identify a bit more closely the kind of mechanism, the chain of things that lead from one background variable to a particular outcome. Why might we do this? Well. Although it might not always be the case, it might help to better target particular interventions. If we're thinking why a particular group of people may not attain as highly as we think they should, maybe looking at the mechanisms might help to kind of design and target interventions to help that group for example. Similarly, we can look a bit more closely at trajectories and changes and shifts over time. Now this fact that we've got these repeated measurements, it's a really unique aspect of these data sets, and if we look a bit more closely at these changes over time, potentially we could pinpoint critical periods. So, for example, if we can better identify the time at which people make the decision about going to university, maybe again, that could be helped to target interventions. Another area to think about is whether findings with older data still apply. So as mentioned, the pace of change in education is very rapid, but we can look back at some of the older findings and say, well, is that still the case? So have tuition fees affected people's aspirations to attend university? Has EBacc changed the way people make their subject choices? And there are opportunities to revisit these questions. So the Millennium Cohort Study is ongoing, and there's a follow up survey to Next Steps, which is called Our Future. So there are still datasets being collected, being produced, that would allow older questions to be revisited. And then finally, we shouldn't just think about what is being done. It's not a limited subset of things that we can actually ask about. There's a whole range of things these these datasets could be used to ask about. So any areas of growing importance could be addressed with them. And this is already happening. For selective schooling, which has become a big issue in recent years, Jerrim and Sims just earlier this year have released a couple of papers looking at the effect of selective schooling using Millennium Cohort Study data. Similarly, education and wellbeing is a growing concern for people. And these days, let's explicitly include questions about wellbeing. So there's definitely potential for areas of growing importance to be studied with these data sets. That really brings us to the end and I just want to kind of reiterate the key conclusions. So firstly, longitudinal cohort studies are really, really valuable for education research. The data sets are usually used to acquire the rich background data or data on quite personal variables that wouldn't be collected with big administrative data sets. Some of the research areas are quite well studied using administrative data, but others can only be addressed with very rich survey data. Many of the studies use a two step methodology with descriptive statistics and then regression and the focus on data from one or two years. But some studies use more complex statistical methods to look at the mechanisms and interactions and trajectories. And finally, just looking forward, I think there's huge potential for future research using these data sets and really as a community we should be aiming to maximise the utility of this very, very valuable data. So I'm gonna end with my reference list, if you're interested in finding out more about the studies themselves. These are the studies that have been mentioned in the talk. And finally, I'd like to thank you all very much for listening.