We need research! Bringing research insights to our agile digital innovation team

by Sylvia Vitello, 27 October 2022
Group of professionals at a board room table looking at research documents

Last year Sarah Hughes, Research and Thought Leadership Lead at Cambridge University Press & Assessment, wrote about how research practices needed to fit within 'agile' ways of working. Sarah suggested that ‘agile researchers’ may have to streamline traditional (academic) research processes so that researchers can provide insights to product teams in a timely and more useable manner.

In this blog, I’m going to share my personal experience as a researcher within an agile working environment, and use this as an opportunity to reflect more deeply on Sarah’s view of ‘agile research’. How did it work for me? What went well? What was challenging?

What research support did the agile team need?

The agile team was a newly formed product innovation team, set up to develop digital high-stakes assessments. I was asked to work specifically with the History sub-team, who were at the very early stages of their product journey – the ideas stage. Conversations with History teachers and Higher Education (e.g. History undergraduate courses) highlighted that their learners had gaps in research skills. Therefore, the team was particularly keen to explore the idea of assessing research skills in a more authentic way.

Now, how could they translate that idea into a useful and effective educational product? That was the key question, and where I came in. My role was to show the team what the academic and educational literature could tell us about the pedagogy and assessment of research skills. Reflecting on my time, I can see that I broadly followed this 4-step process.

Diagram showing 4-step process, from left to right: define questions, gather literature, analyse and synthesise, report

Defining questions

In total, I worked on a set of three research questions for the design team. Each one was about an aspect of assessment, teaching, and/or learning that the team needed evidence on to make decisions. Importantly, the team were open-minded about what the evidence might show. And the wording of the research questions reflected this – they were all open-ended and non-directional.

Team goal: Developing a product to assess historical research skills

  1. What are historical research skills?
  2. How are they taught and assessed?
  3. Is there progression between and within them?

For each question, I had to make sure I understood what it meant and why it was being asked.

  • The question – did my interpretation of the question match the team's?
  • The context – e.g., which educational system(s), school stage(s), curriculum(a) did this apply to?
  • The need – what was the team's information gap? What didn't they know? Why did they need to know about this?
  • The use – how was the team going to use the information?

What helped?

  • Asking the product team – short, precise clarification emails worked particularly well
  • Doing some background reading on the topic
  • Reading planning or strategy documents from the product team 
  • Sharing my uncertainties with the Research Lead
  • Having discussions with other research colleagues

Gathering literature

Gathering the literature was the next step. But, it was never as simple as it first seemed. There were various research obstacles in my quest for answers.

  • There was often little direct research within the academic literature (i.e., peer-reviewed).
  • There was a large (sometimes overwhelming) amount of potentially-relevant information from outside academics – e.g., from teacher websites, blogs, or textbooks.
  • There was no single place that contained all the literature I might need.

What helped?

  • Being open-minded and thinking laterally about potential sources of information
  • Asking other researchers and colleagues with experiences in the area for their suggestions
  • Creating a long list of these potential research sources
  • Prioritising the options within the list, based on their (i) directness to the research question, (ii) rigour/credibility, and (iii) usefulness for the product team.
  • Keeping notes on my methods for gathering the literature

Analysing and synthesising

Within my research process, there was not a chronological separation between this step and the gathering of literature. They were interdependent and iterative.

  • I analysed and synthesised the information as I went along.
  • Sometimes my analysis of findings propelled me back into the literature to search for further evidence.

A key decision in this stage was how to collate all the information together. I chose to use Microsoft Excel as my tool. Excel is a much more versatile software than we often give it credit for. I regularly use it to keep track of written pieces of information, and to record and organise my thoughts on them. During this History work, I ended up with several Excel spreadsheets.

  • I used the rows to separate different pieces of information or evidence e.g., individual points in research papers, teaching methods, and assessment methods).
  • I used the columns to keep notes of key features or thoughts about each piece of information.
  • This became a database that I could use to compare and contrast the different pieces of information.
  • I could also re-organise information and even convert some of them into numerical data to produce graphs to visualise how common different findings were.

In addition, the analysis and synthesis process reminded me that, when it comes to education, there is (almost) never a single interpretation or conclusion to research findings.

  • There is always more information that might exist in the literature that could provide us with more insights.
  • There is always room for more primary research.
  • What is important is getting to a point where we are confident that our research insights are informed, reasoned, and going to be useful. And, equally important, these insights need to have an audit trail, which anyone can use to trace them back to the source.

What helped?

  • Using data analysis software to organise information.
  • Making regular notes of my thoughts and conclusions, especially in situ rather than retrospectively.
  • Having discussions with colleagues, especially the Research Lead, colleagues working on related areas, and members of the product team. 
  • Remembering that research is dynamic and subjective.

Reporting back to the team

It was clear from the start that the product team wanted a quick turnaround of research insights.

  • They did not have time for us to write up detailed research reports, and they didn’t need all the information we’d typically include.
  • The short timeframes (usually 2 weeks) meant that the findings were usually preliminary, rather than final. As the researcher, I felt it was my responsibility to make sure the team viewed the findings within the context of how I conducted the research.

How could I bring the research insights to the team in a way that ensured they had all the key details but also understood the research caveats?

We achieved this by having face-to-face meetings, which were devoted to talking through the findings together. I liked this approach for many reasons.

  • It enabled us to talk openly and honestly about the research.
  • It gave me the opportunity to reiterate caveats.
  • It allowed us, as a team, to positively interrogate the research findings, working together to pull them apart and put them back together again to arrive at research insights that were useful for the team and their decision-making.
  • In many cases, my interpretations changed and evolved as part of those discussions!

After the meetings, I would also give the team some of the outputs I’d produced during the analysis/synthesis stage.

  • These outputs were informal, and I’d usually have to tweak or re-organise the information a little, to ensure that someone (other than me) could follow the content.
  • I ensured that they contained information and links to all the research I’d reviewed. That way the team could look up the evidence themselves too.

What helped?

  • Asking the teams beforehand what documents they’d find helpful. How would they like to receive the research insights?
  • Discussing the findings with the product team. This is a must! 
  • Going into the reporting discussions with an open mind and full of curiosity to learn from the team.
  • Providing the team with written records of the research evidence so that they could reflect on it, reference it and revisit it during decision making

What is agile research to me?

The words 'research expertise', 'teamwork', 'curiosity and learning', 'dynamic and iterative' in a diagram around the words 'agile research'

Blog by Sylvia Vitello, Senior Research Officer

If you would like to learn more about the work of our research team, and the development of new digital high-stakes assessments, you can sign up to join our seminar, Developing Research Informed Digital Assessments on 24 November.

Before you go... Did you find this article on Twitter, LinkedIn or Facebook? Remember to go back and share it with your friends and colleagues!

Related blogs

Key bloggers

Tim Oates Blogger
Tim Oates

Director of Assessment and Research

Research Matters

Research Matters 32 promo image

Research Matters is our free biannual publication which allows us to share our assessment research, in a range of fields, with the wider assessment community.