Sifting and evaluating academic articles using Scholarcy

Getting a clear insight into what an academic article is about takes time. It’s perhaps not surprising that researchers are good readers, but effective research involves two quite different kinds of reading: reading at scale, and detailed evaluation.

Researchers read at scale to carry out the initial scan of an article. Using the standard figure for adult reading speeds of 300 words per minute, and given an average length of article as 4,133 words (based on a count of 61,000 articles in PubMed Central), it will take a researcher around 14 minutes to read (that is, to scan) one paper. However, this figure is based on reading fiction or a newspaper article, and is unrepresentative of the reading a researcher does. Reading speeds vary significantly depending on the type of content, and reading academic content in detail is a very different process. A highly cited article by Satish Keshav (2007) suggests a three-pass approach for detailed reading for academic purposes: the three passes comprise 10 minutes, one hour, and then a further hour (or more, for less experienced readers) – that is, over two hours for each paper!

Why should it take so long to read a paper in detail? There are several reasons:

• Academic papers are not written to be entertaining: they use a rather stilted language with little redundancy, and they have little or no narrative interest;
• To save space, academic authors assume considerable background knowledge of the subject;
• Papers assume the reader is familiar with the articles cited;
• When a researcher reads a paper in detail, they are looking for errors. Research articles are not to be trusted! Although published articles are peer-reviewed, this does not mean they necessarily follow best practice for sample size, bias, and problem-framing.

How can Scholarcy help? Scholarcy has a host of features for comparing and evaluating articles, helping you to decide which articles to read in detail, and then assisting you with the detailed read. Let’s look at our collection of articles in our library on the topic of water desalination. The default dashboard view in Scholarcy shows several articles alongside each other. Your goal is to assess these articles: which of them is central to my argument? What are their main arguments? Which of them agrees or disagrees with my findings? Scholarcy can help with all of these questions, and more.

The tools for evaluation include:

Headlines

‍Viewable from the dashboard (above), the headlines enable you to grasp the main idea of each paper. Trying to capture the gist of a paper in one sentence is an excellent way to approach it. Once you have viewed the title, the author, year of publication, and headline, you can click on any article to open what is called the Scholarcy Flashcard. Here you have several ways to find out more about the paper.

Highlights

In the Highlights section, Scholarcy provides four or five key sentences from the article that provide a quick overview of the content:

‍Blue and orange highlighting

Wherever the full text is displayed, Scholarcy applies its highlighter to indicate two types of statement. You can see at a glance which type of statement is being used:

• Factual statements are highlighted in blue

• Findings and contributions made by the authors are highlighted in orange.

‍
Key terms

If there are significant technical terms in the article, these are typically shown as links to the relevant definition in Wikipedia. So, for example, in the above screenshot, “power generation” and “microbial desalination cell” are linked to articles in Wikipedia.

‍

References

Often it is necessary to review the cited papers for an article to get a clear idea of the main argument. Scholarcy simplifies this for you, by giving you both a convenient way to access the full text, and some “signposts” that enable you to get an idea about the referenced articles even before you read them.

‍
For accessing the article, the orange icons provide a link to the full text, and a link to Libkey, a service that provides access via your institution (if you are a registered user of an institution).

To get a quick overview of the cited article, clicking on the Scholarcy icon shows the key findings from the article.

The above example shows citation context from scite.ai, and the key findings of the cited article from Scholarcy.

Combining Scholarcy with reference management software

One useful feature of Scholarcy for those needing to quickly review, say, the methods, results, and key findings from a whole collection of articles on a subject, is the ability to export your flashcards to Zotero.

You will then see not only the reference of the article in Zotero, but also the key highlights, findings, and a structured summary of that article.  This gives you more context to your references and you can quickly generate a report containing the flashcard contents for every article.

By using all these tools either separately or in combination, Scholarcy can make the literature review process much more effective – and time-efficient.