The National Institutes of Health uses small groups of scientists to judge the quality of the grant proposals that they receive, and these quality judgments form the basis of its funding decisions. In order for this system to fund the best science, the subject experts must, at a minimum, agree as to what counts as a “quality” proposal. We investigated the degree of agreement by leveraging data from a recent experiment with 412 scientists. Each of these scientists acted as primary reviewers for three of 48 NIH R01 grant proposals, half of which hadbeen funded and half unfunded. Across all dimensions of NIH’s official rubric, we find low agreement among reviewers in their judgments of scientific merit. For judgments of Overall Impact, which has the greatest weight in funding decisions, we estimate that three reviewers yield a reliability .2, and 12 reviewers would be required to bring this reliability up to .5. Supplemental analyses found that reviewers are even less reliable in the language they use to describe proposals.

How many reviewers are required to obtain reliable evaluations of NIH R01 grant proposals?