Revamping the scientific paper: Paper Backbone, Key References and the Reproducibility Score

This short article aims to introduce some ideas and concepts that could be helpful in drafting an evolution of the scientific article. The PDF format, while currently being the only format that is widely accepted by academic journals for consideration for publication, suffers from some major issues.

The problem with PDFs

The first major limitation of PDFs resides in its static format: the PDF does not allow for modifications, it can only be replaced by another document. In fact, would the information in a paper become outdated, the author would have to create another article with the updated findings and submit it for publication once again. The traditional PDF does not “evolve” over time as new information comes in and this poses a major hindrance to those authors that would want to show the history of their scientific work. The history of a research study, consisting of the ideation stage, the development phase and the testing of the hypothesis, should always be retrievable when accessing an article. Having a similar paper history would also make it easier to demonstrate the amount of time that has been devoted to a study.

A glaring example of areas of improvement is that of review articles. The number of journals and articles being published in the last decades has grown exponentially, thus leading to the generation of a massive amount of reviews, aiming to summarize recent developments within specific fields. This process, however, has also led to the creation of redundant material, increasing the body of literature but bringing only marginal improvements to the body of knowledge. In addition, these reviews are, in some cases, not fully comprehensive. That means that the parameters used for analysis and comparisons of research articles are often different, creating discrepancies in the evaluation of research works . Provided that most results could be identified and compared between each other on the basis of different “attributes”, compiling these works into databases would make the process of searching for information much more efficient. In relation to this, a previous post on Perpetual Reviews aims to propose a novel version of living reviews. These reviews are dynamic in nature, as they are not based on a static PDF file, they are crowdsourced, as anyone could input data into the database (once legitimacy and suitability of data are verified) and they are simple to navigate. More complex constructs could also be built on top of these living reviews, such as Compute to Data (a concept borrowed from Ocean) where readers can run statistical analysis based on the data contained in a Perpetual Review, and Perspectives, where readers can draw their personal conclusions based on the compiled data, and later publish them as individual Perspectives.

The static nature of the PDF format is also detrimental for the research article, where protocols and processes are described. An example is made below, in order to illustrate those limitations.

A research group develops a study using a protocol, called protocol A, and publishes the results. The same research group finds out, months later, that better results could be obtained with a modified protocol, called protocol B, and wants to publish updated results. Here two problem arises. The first problem is that a new study with only slights improvements to already obtained results would hardly be published. Would that happen, a second problem would consist in having the authors spending additional time writing another manuscript to substantiate the new results. This process is bothersome for authors and it would still have a high chance of not being accepted by the journal for publication.

Fig.1 Schematic of static and dynamic research articles

What would already bring an improvement is having research papers with trackable changes and versions (v1, v2, etc. similar to software updates). This is expected to bring several benefits, including full transparency on how a protocol or a study has evolved over time. Moreover, this is intended to foster collaboration, where other research groups might be able to “upload” a new version of the same study with a different protocol, aiming for improvements of specific elements of a study.

A Paper Backbone structure tracking papers history

The ability to be able to submit new versions of a study leads to the introduction of a second concept, that of the Paper Backbone. The concept of paper backbone draws from that of versions history but aims to be specific to research articles. Usually, when a new study from the same research group gets published, that study, unless it's a brand new research line, has some links to previous studies done by the same research group. This is indeed specified in the article, usually though a reference mentioning the previous study. This citation, however, is buried among tens of other references, thus making it difficult for the reader to link that publication to the previous study, unless they go through the whole article. This information should be clearly visible on the front page or summary of the article, as that is an important knowledge that should be readily available to readers.

Fig. 2 Schematic example of the Paper Backbone

The benefits of introducing a Paper Backbone are several and they are mostly evident in multidisciplinary studies comprising. An illustrative example could be made about the development of a biosensor with applications in medical diagnostics. A biosensor is comprised of different elements, namely:

The materials that are responsible for the “sensing behavior”
The electronics that translates biological signals into understandable electrical signals
The structure of the device comprising all components

The development of such complex devices needs to be done in stages, thus it's usually the materials and the sensing element that are developed first, while the rest follows after that first element has been developed. It is not unusual for big research groups to do everything “in-house”, that means developing all of these elements using the multidisciplinary competences of the research group. What happens usually is that the individual elements are published as standalone research articles, and then a gradual implementation of other elements leads to the final prototype. There is currently no way to bundle these studies together in a way that makes it easy for the reader to see the development steps, apart from simple referencing. The same applies to those groups that do not do everything “in-house” but rather borrow some protocols or designs from previous studies to implement them in the design of their device. Similarly, there should be a way for the reader to immediately recognize what is the origin of the individual elements, what is original and what is not original.

Not all references are made equal

In order to be able to build a Paper Backbone, the Key References of an article should be identified. Nowadays, one of the metrics used to measure the “impact” of a paper is the number of citations that the paper gets. However, citations are not all equal. References in fact can have greatly different weights, and the way of considering citations today is rather simplistic as it does not consider the nuances of the impact of each individual citation. Here the term “impact” of a reference is defined based on how important that reference was in the development of the study that cited it. To illustrate this concept a comparison can be made between:

A reference that has been included in the introduction when providing some background about the motivation to carry out the study.
A reference that is cited when a specific procedure from that study is being used during the development of a device, analysis of a sample etc.

At a first glance, it should be apparent how these two citations clearly do not have the same weight in the development of a research article. However, as it stands today, they are in fact considered on the same level. The concept of Key References aims to change the status quo by asking the author to clearly identify those articles that were foundational to the development of the study, before the article is submitted. Key references should be displayed in the front page or summary of the article, allowing readers to quickly identify where the founding elements of a study are coming from. The discrimination between simple and key references should also be reflected in the number of citations, either introducing a separate count for key references or adding some sort of multiplier that allows for a better discrimination of the impact of key references.

Introducing a Reproducibility Score

The last metric that should be evaluated when assessing the impact of a scientific article, is the Reproducibility Score. The reproducibility score is a much more challenging feature to build, when compared to Paper Backbone and Key References, but an essential one to foster progress in the scientific community and reduce wrongdoings. It has already been vastly reported how most of the research studies that are published in literature are in fact not reproducible, leading to the so-called “reproducibility crisis”. In the context of an evolution of the scientific paper, a reproducibility badge and a reproducibility score would greatly help assessing which studies are in fact reproducible in order to help other researchers to base their assumptions on a solid ground.

Fig. 3 Schematic representation of the Reproducibility Score concept

A reproducibility score for a research article would be calculated once another research group reproduces the methods of the article and publishes the results. Would the results be confirmed, the paper will receive a positive score in the report. Once a minimum of studies have been performed, reaching statistical significance, a score will be calculated. A threshold after which a reproducibility badge is issued would then be set. The idea behind this concept is to create a sort of consensus mechanism that allows to evaluate the validity of scientific claims among researchers. Several mechanisms could be also put in place to prevent malicious activities, but a detailed description of the Reproducibility Score concept is beyond the scope of this article.

Closing Remarks

The concepts introduced in this article are still in the ideation stage and will face many challenges before they could be considered for implementation at large scale. Running a pilot study where these concepts get tested, however, could already give important information about the challenges that would most probably be faced before validation is reached. The academic research field has remained mostly unchanged for the past 50 years. However, the appearance of independent initiatives and innovative ideas shows how this sector could be ripe for disruption. And it is ultimately the researchers responsibility, as curators of science, to make sure the wave of innovation that would likely sweep across academia, will change the way science is done for the better.

Preprint

Preprint

Revamping the scientific paper: Paper Backbone, Key References and the Reproducibility Score

Revamping the scientific paper: Paper Backbone, Key References and the Reproducibility Score