Literature Review Pitfalls
One of the activities common to all researchers – from undergraduates to post-docs – is conducting literature reviews. The volume of research literature has been growing at an unprecedented rate in recent years. The number of new scientific papers published each year now exceeds 2.5 million1 , and the problem isn’t limited to published research. Preprint servers – which are increasingly becoming an important resource for researchers – have grown by over 300% since 20152.
The amount of research available on any given subject can make the early stages of the literature review even harder, especially when it comes to formulating search strategies and systematically screening the results. We’ll focus here on things to look out for once you’ve identified and downloaded a folder of papers that you think could be useful to your own research.
1. Skim reading: tempting but not always effective.
If you’re a student or a researcher, the chances are you’ll screen your search results using the title and the abstract, or by skim-reading the entire paper. But with hundreds of papers to review, even this can be easier said than done. And while skim-reading as a technique is something we all rely on to keep up, knowledge acquired in this way is often quickly forgotten – which either means important information ends up getting missed, or the paper has to be re-read later on. This often results in researchers skipping the screening and exploratory phase of a literature review and diving straight into the fine print of a paper which comes with its own set of problems. So how can you effectively screen your search results without quickly feeling overwhelmed and frustrated?
2. Abstracts aren’t always what they seem.
Often the abstract is recommended as a starting point. Some abstracts are well structured, provide a good overall summary of the paper, and can be a more effective resource than skim-reading the full paper at this stage. The problem is that the quality of abstracts can vary widely. They might lack enough detail for experienced researchers, or be too long or too technical for those new to a subject. This inconsistency can also be quite a barrier if you’re trying to methodically assess a batch of papers with limited time available. Ideally, what’s needed at this point is a summary snapshot of the research in an accessible format that highlights the objectives, methods, and key findings. This way, you get a good overview of a collection of papers to draw more meaningful observations and comparisons. It’s best not to assume that abstracts will always give you a true reflection of the contents of a paper; often they focus on just one section of the paper such as the methodology.
3. Starting at the beginning isn’t always the right thing to do.
Once you’ve narrowed down your search results to a collection of papers that look like the most relevant and informative for your work, the next task is to determine the validity and reliability of their findings which is usually achieved by reading the paper in full. But this typically takes even an experienced researcher 45 minutes, so if you have a pile of 20 papers but you don’t have a spare 15 hours, this is the point you’re most likely to get frustrated and give up altogether. The temptation to cut corners and is to cite a source without reading it is high, but this can really backfire later on.
The best way of reducing the chances of this happening is by adopting a systematic approach. Exploring some of the sections, figures and tables in a non-linear way can be a helpful way of getting to grips with a paper before reading from start to finish for the first time.4 Even if you’ve developed critical eye and some useful screening techniques, paying close attention to study design, sample size and potential author bias can take time to perfect. And trying to tackle all of these elements in one go by diving straight into the paper will generally leave you feeling overwhelmed.
4. Not all papers are created equally
For postgraduate students and early career researchers, another challenge at the literature review stage is becoming adept at reading a wide range of styles – from qualitative research, systematic reviews and meta-analyses to economic analyses and many other subject-specialist papers. Non-statisticians will sometimes need to quickly analyse and draw conclusions from complex data, and significant statistical findings can often get overlooked at this point.
Extracting statistical information from the text can make it easier to understand and highlight any potential errors or anomalies. It can be helpful for researchers to study this type of information in an environment that is completely separate from the rest of the paper for greater clarity and to aid focus.
And depending on the discipline, some sections of a paper will be more significant to the researcher than others. So again, a systematic way of breaking down any research paper into its constituent parts and highlighting the key elements of these parts is a good way of removing unnecessary distractions, particularly during the early stages of reading a paper in full.
5. Paying enough attention to cited sources.
It’s tempting to leave checking the sources a paper or chapter cites to late on in the literature review – mainly because of how laborious this activity can be. But that’s when the potential to propagate misinformation can creep in. Without going back to the original paper to check assertions made in later papers, researchers can leave their work open to being discredited. Not only that, but the opportunity to identify gaps in a piece of research by systematically reviewing the findings of all the papers it references (and then reviewing their findings) is pivotal to a high-quality literature review. Many researchers will spend days following citation trails back to original ideas – their work being all the richer for it – but an equal number will try to create shortcuts at this stage and miss out on significant findings. Unless you’re using an enhanced PDF reader, or reading the article online, few published research papers – and virtually no preprints – will give you direct links to follow these citation trails, which is an understandable source of frustration.
Technologies that make in-text citation (eg from a PDF of a research paper) clickable are now available and can be useful, but this still only address part of the problem. Researchers also need to be able to quickly identify the main findings of any cited source so that they can verify the claims of a paper while reading it. Being able to quickly scan summaries of cited papers rather than also attempting to tackle them in full, would not only make the review process much more efficient but could really improve the quality and accuracy of the output. Find out more on this in our upcoming article: How to do an effective literature review with Scholarcy.
- Canadian Science Publishing. (2019). 21st Century Science Overload | Canadian Science Publishing. [online] Available at: http://blog.cdnsciencepub.com/21st-century-science-overload/
- Johnson, R. and Chiarelli, A., 2021. The Second Wave of Preprint Servers: How Can Publishers Keep Afloat?. [online] The Scholarly Kitchen. Available at: <https://scholarlykitchen.sspnet.org/2019/10/16/the-second-wave-of-preprint-servers-how-can-publishers-keep-afloat/>
- Downloads.smccd.edu. (2019). [online] Available at: https://downloads.smccd.edu/file/getting_students_to_read.pdf?s=/sites/downloads/canada&du=/sites/downloads/canada/Shared%20Documents/Ca%C3%B1ada%20College%20Inside/CIETL/getting_students_to_read.pdf
