If you are involved in research in any way, you will, at some point, need to master the systematic review. Rob Calder goes through a step-by-step approach this supposedly simple (but often not) research method.

The systematic review is, in the grand scheme of things, a recent development. It can be done without leaving the office and has become a mainstay of MSc dissertations and first-year PhD students. A systematic review is easy to access and sits (nearly) at the top of the strength of evidence checklist. When done well, it can answer important questions, when done badly it can waste a lot of time.

“I once stumbled on a whole literature on people addicted to salsa dancing. You’ll be amazed at what’s out there”

A systematic review is based on a question. Usually, but not always, a clinical question. Do NOT underestimate the importance and difficulty of determining your question. If you are at the point where you can no longer stand to think about your question, you’re probably only about half-way there. Honestly. Keep going, you will thank yourself later. Once your question has been determined, you can then collect all the published research that might answer your question, collate the results and provide as good an answer as is possible.

At its core, a systematic review tells you everything that is known on a particular topic. It will also tell you what you don’t know. Sometimes it will tell you what more you need to know in order to know more… if that makes sense. Either way, there are several stages to a systematic review:

Register your protocol

Go on, be a good scientist – as soon as you’ve finished reading this page, head over to PROSPERO and register your protocol.

Determine your question using PICO

PICO stands for Participants, Intervention, Outcome and Comparator. If you were writing an essay you could perhaps explore all the literature on an outcome (O). For a systematic review you need to ask whether an intervention (I) causes more of an outcome (O) compared with something else (C) among a particular group of people (P). For example, do not collect all the literature on ‘sneezing’ (O); rather, collect the literature on whether black pepper (I) causes more sneezing (O) compared with white pepper (C) among people with hay-fever (P). You might think your question is specific, but it will always need to be more specific.

Once nailed down, this will help you form your inclusion and exclusion criteria determining which articles are in and which are out.

“You don’t want to miss several crucial articles because one ‘quirky’ research group refers to sneezes as nose hiccoughs”

Develop your search strategy

There are billions (probably) of academic articles published each year. You need to identify only those that answer your question, whilst ignoring those that don’t. It’s a delicate balancing act. You absolutely must speak to your friendly academic librarian about this. They will look at your question and be able to advise on which databases to search and which keywords to use.

You should also consider doing a scoping review. This can help identify the odd words used in your study area. You don’t want to miss several crucial articles because one ‘quirky’ research group refers to sneezes as nose hiccoughs.

Screen your literature (titles and abstracts)

You sit in front of a computer for what might be several weeks deciding whether studies are included or excluded. Do not take your eye off the ball (/screen). Use the pomodoro technique (or coffee) if you find your mind drifting. You will probably have to call in favours and ask someone to second screen. You will probably have to repay this by second screening someone else’s review.

Screening is a draining, monotonous and occasionally thrilling task – I once stumbled on a whole literature on people addicted to salsa dancing. You’ll be amazed at what’s out there.

Screen again (full text)

Sure, you just spent 3 weeks screening. Still, you pick yourself up, get all the pdfs, and start again. It’s not even close to the last time you’ll see these papers either – they will soon be closer to you than your family or friends (if you’re doing it right). Your first screening might have identified 200 studies that refer to sneezing, hay-fever and different kinds of table condiments. Of those 200, maybe half will come close to answering your question. You will only know which when you have read all the methods sections. Three times. This stage makes you really appreciate good academic writing.

Extract the data

Having collected your final papers for inclusion, you then need to find the data that you are after and put it into a spreadsheet (or similar). The specific data you extract will have been described in your protocol on PROSPERO (because you registered your protocol, didn’t you?) and will go something like: Study, date, location, number (n), study design, participants, type of pepper, length of sneeze, sneeze measurement, result of study, other details.

It’s okay to change this list as you go and as your understanding develops but keep a record of what changes you have made and why. Report these changes when you write it up, update PROSPERO, transparent reporting matters.

Assess the risk of bias

By the time you get to this stage you a) won’t want to do this b) will think you don’t have time to do this c) will actually quite enjoy doing this and d) do have the time as it turns out. The purpose of assessing risk of bias is to avoid putting small and badly designed studies on the same level as massive randomised controlled trials.

You might have 100 studies on sneezing, but if only three can be relied on, or have a low risk of bias, then you want to focus on those three rather than the ramshackle 97 that were funded by the white pepper industry using their friends as participants.

Start with Hoy and colleagues, the Newcastle-Ottawa scale, COREQ and RoB before getting progressively confused and fascinated by the whole issue of bias. You can then thrill your friends with the details, if you have any friends left at this point.

Synthesise the data

If all of your studies are very very similar, you can ‘pool’ all the data and analyse them as if they were a single study (this is called a meta-analysis – more on which later). If, however, all your included studies are quite different, you will have to either describe the data, talk through it, or select an alternative approach. Learn how to type heterogeneity.

I once read a systematic review of just one paper, and while I wouldn’t particularly recommend this approach, one of the important results of any review is how much research exists. If there is very little evidence on pepper-related sneezing, then it can be important to publish that finding to encourage more research.

Write it up

Okay, so you now know whether to use white or black pepper to induce the nose hiccoughs in your younger sibling this Christmas. But if you don’t tell the rest of the world, what was the point? However, academic publishing is another story.

There are some superb resources for systematic reviewing. You must look at the PRISMA statement. This will tell you what you need to do and how you need to report your review. It is unlikely that you will get your review published without a PRISMA flowchart and checklist.

Cochrane specialises in systematic reviews on a scale and to a level of detail that you just can’t imagine. Their website will help give some insight into how they do this. It is also a good place to look for inspiration for your search term

The opinions expressed in this post reflect the views of the author(s) and do not necessarily represent the opinions or official positions of the SSA.

The SSA does not endorse or guarantee the accuracy of the information in external sources or links and accepts no responsibility or liability for any consequences arising from the use of such information.