How to evaluate the quality and veracity of an article

Analysing how a study builds on, confirms or contrasts with previous work is key to appraising its reliability

So you’ve got a literature review to write, and the deadline is approaching. Perhaps you don’t know how to begin. Or maybe you’ve started and got stuck. Perhaps you’ve used one of the many discovery tools out there to find relevant papers, but now you have to read and filter them all. You’ve got a folder full of articles, and you’re learning that PDF doesn’t actually stand for ‘print, download, forget’….

As graduates and post-graduates ourselves, we’ve been there too. And it’s why we built Scholarcy. In this post, our goal is to help you learn some new techniques to quickly evaluate those papers you downloaded.

Now, much has been written on how to search the literature¹, and there are some great guides on how to organise your sources and how to structure your writing². But when they suggest you identify gaps or weaknesses in the literature or areas of controversy, how do you actually do this? While there are some excellent guides³ on how to read a paper⁴, they tell you what to do rather than how to do it. When they suggest you verify the paper’s main assumptions, theoretical frameworks, and contributions, how do you go about doing that?

Interestingly, some recommend not reading the abstract⁵, as this can provide a biased interpretation of the paper, given that author will often write the abstract in such a way as to show the paper in the best light. Jennifer Raff⁵ recommends reading the introduction and background first to identify:

the main problem that the researchers are trying to solve
prior work on the subject and its possible limitations

In the background

With Scholarcy’s flashcard technology, getting this background context is a simple process, as we make all citations clickable:

Shows clickable citations in Scholarcy linked to the full text of cited papers — Clickable citations in Scholarcy

This will take you direct to the source in your browser, where you can use our Chrome Extension to highlight potential limitations instantly:

Identification of study limitation in Scholarcy — Scholarcy identifies study limitations

Keshav⁴ suggests looking up background reading for information on unfamiliar terms or acronyms. In Scholarcy, acronyms are automatically expanded, and background knowledge to get you up to speed is just one click away:

Key concepts extracted by Scholarcy with links to their definitions — Links to definitions of key terms

Follow the citation trail

Keshav also recommends following up references if you are unfamiliar with the topic. In the Scholarcy flashcard, this is super easy, as you can export all references to Zotero, which will locate the full texts for you, or you can click on the LibKey or UnPaywall links for individual references:

Extracted bibliography with links to full texts — Links to full texts of cited works and scite.ai analyses

It’s all in the method

Next, Raff advocates studying the methods in detail. What was done and with whom? Our flashcards help with this by isolating mentions of study participants and data analysis techniques used:

Scholarcy extracts details of study participants and methods — Scholarcy highlights data on study participants

Data analysis methods extracted with links to definitions — Links to definitions of statistical tests

If you are not sure what specific methods such as Kruskal-Wallis, Benjamini-Hochberg or Dunn’s tests are, you can get more information on them by clicking on them. We can see that the Benjamini-Hochberg test is a procedure to identify the false discovery rate which, if not corrected for, could cause the results to be inaccurately interpreted.

Definitions of statistical tests - false discovery rate — Links to Wikipedia for definitions

Standing on the shoulders of giants

Both Raff and Keshav suggest comparing the results to previous work in the field. The ‘comparative analysis’ section of the Scholarcy flashcard makes this easier to do – again, all citations are clickable, and our integration with scite.ai shows whether others have confirmed or questioned those findings. Pay attention here though, as some of these may be self-citations, or to work by the same research group as the current study.

Showing how a study builds on or contrasts with previous research

In conclusion, discovery tools will get you so far, but for appraising individual studies, AI tools that distil and connect the key information can really accelerate the process. You need more than a summary – you also need background information on the results of previous work, the current findings in the context of what has gone before, and a quick overview of limitations and future work. Is the study is standing on the shoulders of giants? Scholarcy flashcards help you to decide.

References

vom Brocke, J., Simons, A., Riemer, K., Niehaves, B., Plattfaut, R., & Cleven, A. (2015). Standing on the Shoulders of Giants: Challenges and Recommendations of Literature Search in Information Systems Research. Communications of the Association for Information Systems, 37, pp-pp. https://doi.org/10.17705/1CAIS.03709
University of Edinburgh (2021). A general guide on how to conduct and write a literature review. Available from: https://www.ed.ac.uk/institute-academic-development/study-hub/learning-resources/literature-review
BMJ (nd). How to read a paper. Available from: https://www.bmj.com/about-bmj/resources-readers/publications/how-read-paper
Keshav, S. (nd). How to read a paper. Available from: https://web.stanford.edu/class/ee384m/Handouts/HowtoReadPaper.pdf
Raff, J. (2016). How to read and understand a scientific paper: a guide for non-scientists. Available from: https://blogs.lse.ac.uk/impactofsocialsciences/2016/05/09/how-to-read-and-understand-a-scientific-paper-a-guide-for-non-scientists/