Author-suggested reviewers: gender differences and influences on the peer review process at an ecology journal
Summary
- Peer review is the primary method by which journals evaluate the quality and importance of scientific papers. To help editors find suitable reviewers, many journals allow or require authors to suggest names of preferred and nonpreferred reviewers. Though authors should know best who is qualified to review their papers, they also have a strong incentive to suggest reviewers that they expect to review their paper positively.
- In this study, we examine the reviewers that are suggested as preferred and nonpreferred by authors, the use of these author suggestions by editors, and the influence of author suggestions on the peer review process and outcomes at the journal Functional Ecology. In particular, we examined how gender of the participants (author, editor and reviewer) influences the role of preferred reviewers in the peer review process.
- Even when not required by the journal, most authors suggest preferred reviewers, but few suggest nonpreferred reviewers. Most author-preferred reviewers are male, but the proportion of women among author suggestions increased over the 11 years, from a low of 15% in 2004 to a high of 25% in 2014.
- Male and female authors did not differ in how likely they were to suggest preferred reviewers, but the proportion of women among author suggestions was higher for female authors (~28%, averaged across years) than for male authors (~21%). Women that were suggested as author-preferred reviewers were more likely to be selected by editors than were men suggested by authors.
- There was no evidence that editor gender, seniority or length of service as an editor for Functional Ecology affected the probability that they used author suggestions. Of reviewers invited to review, those that were author-suggested were more likely to respond to the editors' review invitations but were not more likely to agree to review.
- Most strikingly, author-preferred reviewers rated papers more positively than did editor-selected reviewers, and papers reviewed by author-preferred reviewers were much more likely to be invited for revision than were papers reviewed by editor-selected reviewers. This difference was not influenced by the gender of the participants in the process.
- Suggesting preferred reviewers benefits authors because preferred reviewers rate papers significantly more positively than do editor-selected reviewers, improving the chances that a paper will be published. Journals and journal editors should recognize that preferred reviewers rate manuscripts differently than do editor-selected reviewers, and be aware that this difference can have large effects on editor decisions.
A Lay Summary is available for this article.
Introduction
Peer review is the primary method by which journals, granting agencies and other organizations evaluate the quality and importance of research contributions (Bornmann 2011; Lee et al. 2013). At scientific journals, peer review of research manuscripts is typically performed by a small number of people (“reviewers” or “referees”) chosen by editors for their subject and/or methodological expertise. These reviewers judge the merits and significance of a manuscript, including its presentation, research execution, data analysis, results and the inferences drawn from the research. Yet peer review faces growing challenges and is increasingly criticized for its flaws and limitations (Benos et al. 2007; Trafimow & Rice 2009). The growth of the scientific literature and the increasing selectivity of top journals in most fields have increased the peer review “burden” on scientists that serve as reviewers (Cooper 2009; Stahel & Moore 2014; Blume & Schmaling 2016), reducing the proportion of review requests that scientists accept (Fox, Burns & Meyer 2016) and increasing the work required by editors to find suitable reviewers (Hochberg et al. 2009; De Mesnard 2010; Fox & Burns 2015). Peer review is also widely argued to be subject to many systematic, albeit subtle, biases (Opthof, Coronel & Janse 2002; Ross et al. 2006; Lee et al. 2013; Kaatz, Gutierrez & Carnes 2014). In particular, reviewer and editor assessments of manuscripts can be influenced by characteristics of the author(s), such as their gender, rank, institutional affiliation or geographic location (reviewed by Lee et al. 2013).
To help editors find suitable reviewers, many journals allow or even require authors to suggest names of preferred and nonpreferred reviewers, and editors can use these lists to invite or exclude reviewers. Because authors generally know their field better than the editors handling their manuscript, authors should be better at identifying people who are most able to assess the quality and importance of their specific research paper (Tonks 1995). However, since reviewer comments are the major determinant of manuscript acceptance by a journal, authors also have an incentive to choose reviewers that will review their paper positively. Multiple studies have shown that author-suggested reviewers rate manuscripts more positively or recommend acceptance more often than do reviewers selected by editors (Scharschmidt et al. 1994; Hurst, Howard & Wedzicha 2005; Schroter et al. 2006; Wager, Parkin & Tamber 2006; Rivara et al. 2007). This has led to concerns that author-suggested reviews are “biased, inflated, unreliable and invalid” (Marsh, Bond & Jayasinghe 2007) and has led some editors to abandon or discourage the use of author-suggested reviewers (Moore, Neilson & Siegel 2011) or to place less weight on reviews submitted by author-suggested reviewers.
The influence of editor, reviewer and author gender on peer review outcomes remains a topic of debate (reviewed in Fox et al. 2016a; Fox, Burns & Meyer 2016). However, it is clear, at least for ecology journals, that author and editor gender predict the proportion of women invited to review; female editors invite more women to review than do male editors (Buckley et al. 2014; Fox et al. 2016a) and women are invited to review more often if manuscript authors are women than if the authors are men for at least one ecology journal (Functional Ecology; Fox, Burns & Meyer 2016). Editor and reviewer gender also influence some additional aspects of the peer review process, such as reviewer responses to review invitations (Fox et al. 2016a). Though we know of no studies addressing the influence of gender on author suggestion of reviewers, our prior observations of gender differences lead us to predict that male and female authors may also differ in the number or gender of reviewers they suggest when submitting a manuscript, and that these differences might influence the process of peer review.
In this paper, we examine the suggestion of preferred and nonpreferred reviewers by authors, the use of these author suggestions by editors, and the influence of preferred reviewers on the process and outcome of peer review at the journal Functional Ecology. In particular, we examine how gender of the participants (author, editor and reviewer), number of authors and seniority of editors influence the role of preferred reviewers in the peer review process.
Materials and methods
Terminology and Variables
Throughout this paper, preferred reviewers refer to the individuals suggested by authors as those that they recommend as reviewers of their manuscript. Nonpreferred reviewers refers to the individuals that authors request be excluded as reviewers (sometimes called excluded or opposed reviewers). Author gender refers to the gender of author who handled the manuscript submission process and who presumably provided the list of reviewer suggestions (the submitting author in Fox, Burns & Meyer 2016). This is most often the first author of the paper (Fox, Burns & Meyer 2016) and almost always (>99% of the time) the corresponding author.
For all analyses, we include only first submissions of “standard papers”, which includes all typical research studies (empirical or theoretical), but excludes review papers, commentaries, perspectives, editorials and other types of papers not considered typical research manuscripts.
Data Sets
Papers submitted to Functional Ecology are first screened by a Senior Editor for their suitability for the journal. Those that pass this screening are sent to a handling editor for consideration. Handling editors decline some manuscripts without review. If they decide to send a manuscript for review, they select a set of potential reviewers and submit these names to the editorial office (henceforth, selected reviewers). A subset of these reviewers are then invited to review by the editorial office (henceforth, invited reviewers).
We have detailed data from ScholarOne Manuscripts (previously Manuscript Central) on the reviewers selected by editors, those actually invited to review and review scores for all papers submitted between 1 January 2004 to 30 June 2014 (inclusive; 6720 standard papers) (details in Fox et al. 2016a). We also have complete details of the peer review process, including dates (the date of review invitations and reviewer responses), peer review scores and editorial decisions (whether papers were sent for review, whether revision was invited) for all papers submitted during this period.
We have the names of all preferred and nonpreferred reviewers suggested by authors for all manuscripts submitted between 1 January 2004 and 31 December 2014 (inclusive). Until mid-2010, authors had the option of suggesting preferred reviewers, but it was not required by the journal. From mid-2010 onwards (starting with manuscript number 2010–00501), authors were required to suggest at least three preferred reviewers. However, there has never been a requirement to suggest nonpreferred reviewers. Note also that we cannot distinguish whether suggested reviewers would have been independently identified as potential reviewers by editors had they not been suggested by the author. In total (2004–2014), 20 048 reviewers were suggested by authors as preferred and 1162 reviewers were listed as nonpreferred for standard papers submitted to Functional Ecology. Additional sample size details are presented in Supporting information and in Fox, Burns & Meyer (2016).
Authorship on submitted papers was identified and categorized for all standard papers submitted between 1 January 2010 and 30 June 2014 (n = 3528 papers). Because author data in ScholarOne are author-entered, the database is incomplete and has many errors. We thus extracted author details from the cover pages of submitted manuscripts. Our data set includes 14 280 author entries, of which ~97% of the non-Asian names were assigned a gender. Further details on the author data set are presented in Fox et al. 2016a.
Gender of authors and reviewers was determined using the online data base genderize.io and internet searches; details are presented in Supporting information and in Fox et al. (2016a) and in Fox, Burns & Meyer (2016). Editor seniority at the time they handled a manuscript was calculated as the number of years between the editor's year of PhD graduation and the year they handled the particular manuscript.
Analyses
For analyses of author behaviour (e.g. suggestion of reviewers), and the fate of manuscripts (revision invited vs. reject) each manuscript represents a single data point. For most analyses of editor and reviewer behaviour, each reviewer entry represents a single data point; the individual either agrees to review or not and then submits a single review score in a certain period of time.
Most of the variables analysed are binary, e.g. whether authors suggest preferred reviewers (yes/no), the decision on a manuscript (revise/reject), and the gender of authors/reviewers (male/female). For analyses in which our dependent variables are binary, we used logistic regression (SAS Proc Logistic or SAS Proc Glimmix, depending on the modelling options needed) with models of the form: DependentVariable = Year + IndependentVariables + Interactions. Year is the submission year and was included in all models (except those sorted by year) as a categorical variable. Dependent variables that were not binary, such as time from acceptance of review invitation to submission of a review and peer review scores, were analysed using general linear models (SAS Proc GLM). All categorical variables were treated as fixed effects, except for HandlingEditorIdentity which was treated as a random effect. Details of the models fitted are presented in the Results section and/or in figure legends.
To test whether review scores submitted by individual reviewers were on average better when they were author-suggested, we identified 634 unique individuals who have reviewed for the journal at least once when they were suggested as a preferred reviewer and also at least once when they were not suggested by authors. We calculated mean scores (one mean when author-suggested and one mean when not author-suggested) and then compared these scores with a nonparametric (Wilcoxon signed-ranked) test. Because the journal scoring criteria has varied among years, we removed the among-year variation by calculating least-squares means using the linear model ReviewerScore = Year + ReviewerIdentity (SAS Proc GLM).
Additional analyses are described in the Results when presented.
Results
Author Suggestion of Preferred and Nonpreferred Reviewers (2004–2014)
The proportion of authors suggesting preferred reviewers was just over 50% in 2004–2006, then increased quickly to >70% by 2008, and eventually reached 100% after the journal began requiring that preferred reviewers be suggested by authors in late 2010 (Fig. 1a; variation among years; logistic regression, 
= 99·5, P < 0·001). In contrast, the proportion of authors suggesting nonpreferred reviewers varied little across years, and did not increase in later years (2010–2014; Fig. 1a; note that the journal has never required that nonpreferred reviewers be suggested at submission; 
= 14·7, P = 0·14). The number of preferred and nonpreferred reviewers suggested by authors fluctuated across years (of those that suggested any), but in 2013–2014 authors were listing similar numbers of preferred and nonpreferred reviewers as they had listed a decade previous (Fig. S1, Supporting information). Interestingly, authors that suggest preferred reviewers were much more likely to suggest nonpreferred reviewers; considering only papers submitted before the journal began requiring that authors suggest preferred reviewers (2004 to mid-2010), 12·3% of authors that suggest preferred reviewers also suggest nonpreferred reviewers, whereas only 3·6% of authors suggest nonpreferred reviewers if they did not suggest preferred reviewers (logistic regression; 
= 64·0, P < 0·001).
The proportion of women among author-suggested preferred reviewers increased fairly consistently over the 11 years for which we have data, from a low of 14·9% in 2004 to a high of 24·7% in 2014 (Fig. 1b; logistic regression, model: NumberOfWomenSuggested / TotalSuggestions = Year; 
= 99·5, P < 0·001). However, we could detect no significant increase in the proportion of women among nonpreferred reviewers suggested by authors (Fig. 1b; 
= 14·7, P = 0·14), though the low proportion of authors that suggest nonpreferred reviewers and the large variation in the number that they suggest would likely obscure changes in gender ratio, if any, over the period for which we have data.
Influence of Author Gender on the Suggestion of Preferred/nonpreferred Reviewers (2010–2014)
Beginning in the second half of 2010, authors were required to submit the names of at least three preferred reviewers. Restricting our analysis to just the papers submitted in 2010 before the journal instituted this requirement, and for which we have author gender information (starting 1 January 2010), we see no evidence that male and female authors differ in how likely they are to suggest preferred (79·7% of men and 75·7% of women suggested preferred reviewers; logistic regression, PreferredReviewersSuggested = AuthorGender; 
= 0·93, P = 0·34, N = 440 papers) or nonpreferred reviewers (12·8% of men and 9·0% of women suggested nonpreferred reviewers; 
= 1·36, P = 0·24). Of authors that submitted suggestions, men and women did not differ in the number of preferred reviewers they suggested (men suggested 3·2 ± 0·1 and women suggested 3·0 ± 0·1 preferred reviewers; F1,343 = 1·2, P = 0·28) or nonpreferred reviewers they suggested (men: 1·4 ± 0·2, women: 1·6 ± 0·3; F1,49 =0·29, P = 0·59). Examination of the number of reviewers suggested by male and female authors (if they suggested any) is presented in the Supporting information (including Fig. S2).
There was no evidence that men and women differ in their tendency to submit the names of nonpreferred reviewers after the journal began requiring that preferred reviewers be suggested (11·4% of men and 10·1% of women submit nonpreferred reviewers; logistic regression, NonPreferredReviewers = Year + AuthorGender + Interaction; Year: 
= 2·58, P = 0·63; AuthorGender: 
= 1·07, P = 0·30; Interaction: 
= 2·22, P = 0·69). Interestingly, we found no evidence that requiring authors to suggest preferred reviewers increased the likelihood that they also suggested nonpreferred reviewers (analysis for 2010 for which we have pre- and post-data; 
= 0·03, P = 0·86) or the number of nonpreferred reviewers suggested (of those that suggested any; F1,72 = 0·19, P = 0·66).
The proportion of women among author-suggested preferred reviewers was higher for female authors (26·9 ± 1·6%, averaged across years) than for male authors (21·3 ± 2·0%; Fig. 2). However, there was no statistically significant difference between male and female authors in the proportion of women among nonpreferred reviewers (16·6 ± 6·1% for female authors, 21·4 ± 4·5% for male authors) (Fig. S3, Supporting information). Examination of variation in author-suggested reviewers with geographic locality and number of authors is presented in the Supporting information (including Fig. S4).
= 2·45, P = 0·65; AuthorGender: 
= 7·8, P = 0·005; Interaction: 
= 1·34, P = 0·85.Editor Usage of Author-suggested Reviewers
For papers for which at least one preferred reviewer was suggested, about one out of every four names selected as prospective reviewers by the handling editor was an author-preferred reviewer, and this proportion has remained fairly constant over time (20–27%; logistic regression, Proportion = Year: 
= 13·8, P = 0·18, with HandlingEditorIdentity included as a random effect). The proportion of invited reviewers that were author-suggested (if any were suggested) has fluctuated more (20–29%) and been generally increasing over the period of the study, averaging ~22% in 2004–2007 but increasing to on average 28% in 2011–2014, being highest (29%) in the final year (2014) for which we have data (logistic regression, Proportion = Year: 
= 21·0, P = 0·02). However, because the proportion of papers for which preferred reviewers are suggested increased substantially over the period of study (Fig. 1a), the overall proportion of editor-selected reviewers that were author-suggested (across all papers) increased substantially over the 11 years, from ~13% in 2004–2006 to 26% in 2012–2014 (
= 218·0, P < 0·001). Likewise, the proportion of invited reviewers that were author-suggested, across all papers, increased substantially, from just 13% in 2004–2007 to over 28% in 2011–2014 (
= 196·3, P < 0·001)
There was no difference between male and female editors in the probability that they selected at least one preferred reviewer (considering only papers for which authors suggested preferred reviewers; HandlingEditorGender added to models described above: 
= 0·11, P = 0·75) or in the proportion of selected or invited reviewers that were author-suggested (
= 13·5, P = 0·94 and 
= 0·14, P = 0·71, respectively). There was also no evidence that more senior editors (seniority = years since PhD; 
= 0·28, P = 0·60) or editors that had served longer on the Functional Ecology editorial board (
= 3·30, P = 0·07) included a higher/lower proportion of author-suggested reviewers among the editors' selected reviewers.
The probability that an editor selected a specific author-preferred reviewer as a potential reviewer depended on the gender of the reviewer; averaged across years, author-suggested female reviewers were selected by editors 46·2 ± SEM 2·5% of the time, whereas author-suggested male reviewers were selected just 41·8 ± 2·1% of the time (Fig. 3; this is the probability that a specific author suggestion was chosen by an editor, not the proportion of editor selections that were author-preferred; least-squares means from model PreferredReviewersSelected / PreferredReviewersSuggested = Year + ReviewerGender + HandlingEditorGender + Interactions, with HandlingEditorIdentity included as a random effect; Year: 
= 17·2, P < 0·001; ReviewerGender: 
= 7·03, P = 0·008; HandlingEditorGender: 
= 0·59, P = 0·44). This difference in usage of male and female suggested reviewers did not depend on editor gender (nonsignificant ReviewerGender-x-HandlingEditorGender interaction, 
= 2·17, P = 0·14).
Influence of Author-suggested Reviewers on the Peer Review Process
Considering only papers submitted before the journal began requiring that authors suggest preferred reviewers, we observed that papers were more likely to be sent for peer review if the authors suggested preferred reviewers (Fig. 4a; statistics in figure legend). However, this difference was only present in 2004–2006 and then disappeared (explaining the highly statistically significant interaction with Year). Of papers for which authors suggested at least one preferred reviewer, the number of reviewers suggested did not influence whether a paper was sent for peer review. Interestingly, during this same period papers were also more likely to be sent for peer review if the authors listed nonpreferred reviewers (i.e. reviewers they asked not be invited to review), though this effect was small and only marginally significant, and is not consistent across years (Fig. 4b).
= 170·6, P < 0·001; PreferredReviewersSuggested: 
= 11·2, P < 0·001; Interaction: 
= 19·8, P < 0·001. (b) Year: 
= 176·9, P < 0·001; NonPreferredReviewersSuggested: 
= 4·00, P = 0·046; Interaction: 
= 5·77, P 0·45.Prospective reviewers invited to review for the journal were more likely to respond to the review request if they were suggested as a preferred reviewer by authors than if they were not suggested by the authors, though the difference was small, averaging just 1·5 percentage points different across years (Fig. 5a). When adding reviewer gender to the analysis, we see that women were less likely than men to respond to review invitations (as described in Fox et al. 2016a), but there was no evidence of an interaction between reviewer gender and preferred reviewer status on the probability that they responded to the review invitation (PreferredReviewer*ReviewerGender interaction: 
= 1·77, P = 0·18).
= 27·9, P < 0·001; PreferredReviewer: 
= 4·75, P = 0·03; Interaction: 
= 5·02, P = 0·66. (b) logistic regression, RespondeeAgreed[y/n] = Year + PreferredReviewer[y/n] + Interaction; Year: 
= 292·2, P < 0·001; PreferredReviewer: 
= 0·66, P = 0·42; Interaction: 
= 16·2, P = 0·09.Of invitees that responded to the review invitation, there was no evidence that preferred reviewers were more or less likely to agree to review (Fig. 5b). Though women were more likely than men to agree to review if they responded to the review invitation (as described in Fox et al. 2016a), we found no evidence of an interaction between reviewer gender and preferred reviewer status on the probability that invitees agreed to review (
= 0·25, P = 0·62). Examination of reviewer times to respond and review are presented in the Supporting information.
Influence of Author-suggested Reviewers on Peer Review Outcomes
Author-suggested preferred reviewers consistently rated papers more positively (~16% better rating, averaged across years) compared to reviewers not suggested by authors (note that a lower review score is better; Fig. 6a). The significant among-year variation in reviewer scores reflects changes in the scoring criteria provided to reviewers by the journal; the difference in mean score given to papers between categories of reviewers remains fairly similar across years, despite changes to the scoring criteria. There was no evidence that reviewer gender (details in Fox, Burns & Meyer 2016), or its interaction with preferred reviewer status (F1,8155 = 1·39, P = 0·24), affected peer review scores. This analysis (Fig. 6a) includes all data points. Limiting our analysis to just papers that were reviewed by at least one author-preferred and one other reviewer, we get similar results (large differences between review scores submitted by author-preferred vs. other reviewers; Wilcoxon signed-rank test; P < 0·001). Only 16 papers were reviewed by both a nonpreferred and other reviewer, an inadequate sample size to test whether non-preferred reviewers submit less positive scores than do editor-selected reviewers.
= 19·4, P = 0·036; NumberOfPreferredReviewers: 
= 51·4, P < 0·001; Interaction: 
= 24·6, P = 0·22, with NumberOfPreferredReviewers treated as a class variable with three levels, 0, 1 and 2–3. When including the mean of the peer review scores as a covariate in (b), the preferred reviewer effect remains significant: Year: 
= 129·4, P < 0·001; NumberOfPreferredReviewers: 
= 14·7, P < 0·001; Interaction: 
= 23·9, P = 0·25; MeanReviewScore: 
= 866·7, P < 0·001.Considering only reviewers that reviewed at least once when they were author-suggested and at least once when they were not author-suggested (N = 634), we tested whether they submitted more positive reviews for papers when they were author-suggested. For reviewers that reviewed under both circumstances, submitted reviews were on average more positive when the reviewers were suggested as preferred than when they were not suggested by authors (mean review score [lower is better] = 2·2 ± 0·7 when suggested as preferred and 2·5 ± 0·4 when not suggested, respectively; Wilcoxon signed-rank test, P < 0·001).
Papers for which reviews were obtained from author-preferred reviewers were substantially more likely to be invited for revisions (rather than rejected). Of papers sent for peer review, 52% (averaged across years) were invited to submit a revision if reviewed by two or more preferred reviewers, 36% were invited to revise if they were reviewed by one preferred reviewer, but only 28% were invited to revise if they were not reviewed by any preferred reviewers (Fig. 6b). Inclusion of review scores in the analysis did not account for the entire difference between author-suggested and editor-suggested reviewers; papers reviewed by author-preferred reviewers continue to have a higher probability of being invited for revision (statistics in legend of Fig. 6b) though the effect sizes are much smaller after controlling for review scores (least-squares means: two or more preferred reviewers = 41·5 ± 3%, one preferred reviewer =32·0 ± 1%, no preferred reviewers = 30·6 ± 1%).
Interestingly, of papers sent for review, papers were more likely to be invited for revision if authors submitted nonpreferred reviewer names (38·3% vs. 30·0%, averaged across years; 
= 5·13, P = 0·02). This difference in the proportion of papers invited for revision disappears when the mean peer review score for the paper is included in the model (
= 2·5, P = 0·12).
Discussion
Most scientific journals allow authors to suggest preferred or nonpreferred reviewers for their manuscript, and editors commonly invite some of the authors' suggestions to serve as reviewers. In this study, we examined author suggestions, how these are used by editors, and how journal usage of author-suggested reviewers influences the process and outcomes of peer review at the journal Functional Ecology. The proportion of women among author-suggested reviewers gradually increased over the period of the study, though women remained a minority among author suggestions as of 2014. There was no difference between male and female authors in how likely they were to suggest preferred or nonpreferred reviewers, but male authors suggested fewer women among their preferred reviewers. Interestingly, men suggested as potential reviewers by authors were less likely to be selected by editors than were women suggested by authors.
The proportion of invited reviewers that were author-suggested, averaged across all papers, has been increasing over time, likely because the proportion of authors suggesting preferred reviewers increased. We saw no evidence that more senior editors or editors that had served longer on the Functional Ecology editorial board included a higher or lower proportion of author suggestions among their selected reviewers. Of reviewers invited to review, those that were author-preferred were more likely to respond to the editors' review invitations, but were not more likely to agree to the invitation, and took longer to submit their reviews. Notably, author-preferred reviewers consistently rated papers more positively than did editor-selected reviewers and papers for which reviews were obtained from author-preferred reviewers were substantially more likely to be invited for revision.
Author Suggestion of Preferred and Non-preferred Reviewers
Over the period of this study, most authors of papers submitted to Functional Ecology suggested at least some preferred reviewers, even in the years before the journal started requiring authors to suggest reviewers (more than 50% each year from 2004–2009), consistent with data from at least one other journal (e.g. Moore, Neilson & Siegel 2011). The proportion of authors voluntarily suggesting reviewers increased over the period of our study, with ~72% of authors suggesting reviewers for their paper in 2009, the year before the journal instituted the requirement that authors suggest preferred reviewers. In contrast, few authors listed nonpreferred reviewers, and the proportion listing nonpreferred reviewers did not change over time, also consistent with at least one other journal (e.g. Moore, Neilson & Siegel 2011).
More interesting are the gender differences in reviewer suggestions at Functional Ecology. The proportion of preferred reviewers that were women steadily increased over the 11 years of the study, from a low of 15% in 2004 to a high of 25% in both 2013 and 2014. Notably, though, female authors suggested more women as prospective reviewers than did male authors – 28% of preferred reviewers suggested by female authors were women, whereas for male authors only 21% of suggested reviewers were women (averaged across years; see Fig. 2). Though only seven percentage points different, it means that female authors suggest nearly a third more female reviewers than do male authors. These results – the increase in the number of female authors suggested over time and the difference in preferences between male and female authors – mirror differences observed for reviewer selection by editors at Functional Ecology (Fox et al. 2016a).
Why do female authors suggest, and female editors select, a higher proportion of female reviewers than do male authors or editors? One possibility is that authors exhibit conscious or subconscious biases and favour reviewers of their own gender. However, we should be cautious when speculating on what specific gender composition we expect among author-suggested and editor-selected reviewers because the gender composition of the scientific community varies substantially with age (e.g. Ceci et al. 2014); thus, the expected gender distribution of reviewers depends on whether we consider all scientists, primarily senior scientists, or some distribution weighted by age or seniority.
In a previous paper (Fox et al. 2016a), we suggested two alternative hypotheses that we think are more likely to explain both the gender difference in author-suggested reviewers and editor-selected reviewers. First, differences in the proportion of women in different subfields of the ecological community (West et al. 2013) can generate covariation between editor, reviewer and author gender even if reviewers are suggested by authors and selected by editors based only on their expertise in the research area most directly relevant to the study paper and irrespective of gender. For example, women are substantially better represented in studies of plant mating systems than in studies of fish or snake ecology (data at eigenfactor.org), such that we expect papers on plant mating systems to be more commonly authored, reviewed and edited by women compared to papers on fish or snake ecology. Alternatively, authors may often suggest as reviewers people with whom they have had, or currently have, positive social and professional relationships. Social and professional networks are structured by gender and age (McPherson, Smith-Lovin & Cook 2001). This professional network structure is evident in authorship data; women are more likely to coauthor papers with other women, and men with men, compared to expectations if the genders sort randomly (Shah et al. 2013; Long et al. 2015). At Functional Ecology, the proportion of women authors on a paper is higher for papers with female compared to male last authors (Fox, Burns & Meyer 2016). Thus, authors suggesting reviewers with whom they have (or had) positive social or professional interactions would lead them suggesting a higher proportion of reviewers of their own gender relative to expectations if reviewers were suggested at random from the entire pool of suitable people. Testing the relative contribution of subject area structuring vs. social network structuring in explaining gender differences in reviewer selection is beyond the scope of this current manuscript.
Editorial Use of Author–suggested Reviewers and the Peer Review Process
We found no evidence that male and female editors at Functional Ecology differed in how likely they were to use author-preferred reviewers, and no evidence that the seniority of the editor or experience on the journal editorial board influenced the usage of author-preferred reviewers. Interestingly, men suggested by authors as preferred reviewers were less likely to be selected as actual reviewers than were women suggested as preferred reviewers (41·8 vs. 46·2% respectively), but that this difference did not depend on editor gender. Additional research on other journals is necessary to see if this holds true more generally. We suspect this difference reflects an effort by editors to identify more female reviewers, but our data do not allow us to test this hypothesis. At least one previous study found that papers were more likely to be sent for peer review if authors suggested preferred reviewers (Hurst, Howard & Wedzicha 2005). We found evidence of this for papers submitted to Functional Ecology, and also detected a marginally significant influence of authors suggesting nonpreferred reviewers. However, both effects varied significantly among years; given the inconsistency, and the overall small magnitude of the differences, we are sceptical these differences are real. If the difference is real, it is unlikely a causal relationship; instead, it may be that scientists with more experience both write better papers and know (and thus suggest) more prospective reviewers in their subject area. Alternatively, editors that are handling difficult papers, such as those at the edge of (or outside) their expertise or those for which they are “on the fence” in their decision, may be influenced by the presence of suggested reviewers when deciding whether to send a paper for review. We find this unlikely to be a common occurrence, but one previous result suggests that difficulty recruiting reviewers can affect editor decision – papers submitted to this journal are more likely to be declined (after review) if the handling editor has greater difficulty recruiting reviewers, irrespective of review scores (Fox & Burns 2015).
Previous studies have shown that author-suggested reviewers are not more likely to agree to review (Hurst, Howard & Wedzicha 2005; Helton & Balistreri 2011). We find that prospective reviewers were more likely to respond to review invitations if they were author-suggested, but not that they were more likely to agree to review (if they responded). Reviewer gender influenced invitee responses (as reported previously; Fox et al. 2016a) but we found no evidence of an interaction between reviewer gender and author-preferred status on responses. Interestingly, author-suggested preferred reviewers took longer to complete their reviews compared to other reviewers (results in Supporting information). This contrasts with results of previous studies that found no difference between author-suggested and other reviewers in the time taken to complete their reviews (Earnshaw et al. 2000; Wager, Parkin & Tamber 2006; and results by Sara Schroter as described in Grimm 2005), though one study (Rivara et al. 2007; which had a very small sample size) observed a similar (but not statistically significant) difference to what we observed. If the difference we observe is real, it may be because preferred reviewers are more likely to be well-known scientists that are busier than average and thus take longer to review (Fawzi 2012). Alternatively, preferred reviewers may write longer or better quality reviews, possibly because they are more familiar with the subject area and type of experiment being described, or because they are more likely to know the researchers and write more constructive reviews. However, the few studies that have compared reviewer quality (Schroter et al. 2006; Wager, Parkin & Tamber 2006; Rivara et al. 2007; Kowalczuk et al. 2015) or tone (Wager, Parkin & Tamber 2006) have found no consistent difference between reviews submitted by author-suggested vs. other reviewers, though they commonly observe trends in the predicted direction that may not be statistically significant due to small sample sizes (Wager, Parkin & Tamber 2006; Rivara et al. 2007; Kowalczuk et al. 2015).
Author-preferred Reviewers, Review Scores and the Fate of Submitted Manuscripts
The most striking effect observed in our study is that author-suggested preferred reviewers gave more positive ratings to papers submitted to Functional Ecology than did other reviewers in every year for which we have data. This is consistent with previous studies; in every study we could find, author-preferred reviewers rated papers (or grant proposals) more positively than did reviewers not suggested by authors (Scharschmidt et al. 1994; Earnshaw et al. 2000; Jayasinghe, Marsh & Bond 2003; Hurst, Howard & Wedzicha 2005; Schroter et al. 2006; Wager, Parkin & Tamber 2006; Marsh, Bond & Jayasinghe 2007; Rivara et al. 2007; Bornmann & Daniel 2009, 2010; Helton & Balistreri 2011; Moore, Neilson & Siegel 2011; Kowalczuk et al. 2015). Likewise, reviewers listed as nonpreferred by authors rate papers less positively than do other reviewers (Moore, Neilson & Siegel 2011), though few studies have examined nonpreferred reviewers, likely because they are rarely invited by journals (at Functional Ecology, only 20 nonpreferred reviewers have reviewed for the journal during the period for which we have data).
One possible explanation for the difference in review scores between types of reviewers may be that authors are good at identifying people that generally write positive reviews, and list them as preferred reviewers (and vice-versa for nonpreferred reviewers). However, we find that reviewers were generally more positive when reviewing papers for which they had been suggested by authors as preferred reviewers than when those same reviewers reviewed different papers for which they were not listed as preferred reviewers. This is despite reviewers not being informed by the journal whether they have been suggested by authors. The same result was found by Moore, Neilson & Siegel (2011). This suggests that preferred reviewers are not just more lenient reviewers overall. Preferred reviewers may be overly positive relative to other reviewers because they are more enthusiastic about the subject of the paper, because they more often share the perspective of the author or, more nefariously, because they are positively inclined towards the authors. Alternatively, preferred reviewers may be, on average, more informed about the subject and their review scores may more correctly (compared to other reviewers) reflect the true quality and significance of the paper.
We also find that editors at Functional Ecology are more likely to invite revision of a manuscript (rather than reject it) if it is reviewed by author-preferred reviewers. This effect is partly, but not entirely, explained by differences in review scores. Though few studies have looked at the relationship between reviewer types (editor vs. author-suggested) and decisions, a few have found that editors more often disagree with the author-preferred reviewer than with the other reviewer, and that scores submitted by editor-selected (rather than author-suggested) reviewers better predicted the final fate of the manuscript (Hurst, Howard & Wedzicha 2005; Marsh, Bond & Jayasinghe 2007; Kowalczuk et al. 2015; but see Schroter et al. 2006; Moore, Neilson & Siegel 2011). Editors may trust reviewers that they have selected more than they trust author-suggested reviewers when there is disagreement between the reviewers. Though author-preferred reviewers may be less trusted by Functional Ecology editors, this is inadequate to explain the effect of author-suggested reviewers on editorial decisions; papers reviewed by one or more author-preferred reviewers remain more likely to be invited for revision even after accounting for variation in peer review scores. Instead, it seems likely that review content, rather than just scores per se, contributes to the difference in outcomes. Previous studies have demonstrated that reviews submitted by author- and editor-suggested reviewers are similar in quality (Schroter et al. 2006; Wager, Parkin & Tamber 2006; Rivara et al. 2007; Kowalczuk et al. 2015), so it's unlikely that differences in review quality explain the difference in editorial decisions. Instead, the difference may be in review tone, with author-preferred reviewers writing generally more enthusiastic reviews irrespective of the review scores given to papers.
At least one previous study has suggested that the odds of acceptance for a manuscript is higher if authors exclude reviewers (Goldsmith et al. 2005; further discussion of these results in Grimm 2005). We found a similar result for Functional Ecology – authors were more likely to be invited to submit a revision of their paper if they submitted names of nonpreferred reviewers. Though the difference was only marginally statistically significant, the effect size was quite large – 8 percentage points is an increase of ~27% in the revision invitation rate for papers with vs. without nonpreferred reviewers listed. However, this disappears when the mean peer review score for the paper is included in the model, suggesting that excluding nonpreferred reviewers affects peer review outcomes by eliminating more negative reviews. When invited to review, author-excluded reviewers tend to be more negative in their recommendations (Moore, Neilson & Siegel 2011). We suspect that many authors, especially experienced authors, know who their detractors are, and know which members of their community are likely to be most critical of their work; as Lowell Goldsmith notes, “People know their assassins” (Grimm 2005). Our data, and that of Goldsmith et al. (2005), suggest that excluding these critical reviewers can substantially increase the likelihood of success in the peer review process.
Conclusions
The majority of authors suggest preferred reviewers when given the option to do so. Though women are underrepresented as suggested reviewers (relative to, for example, their representation among authors), the representation of women among author-suggested reviewers, and among reviewers actually used by the journal (Fox et al. 2016a), has been steadily increasing. Whether and how many reviewers are suggested differs little between male and female authors, but female authors consistently suggest more women to be reviewers than do male authors, consistent with gender differences in editor behaviour. Intriguingly, preferred reviewers that are female are more often invited by editors to review than are preferred reviewers that are male, opposite the pattern commonly observed for invitations of other kinds, such as to participate in symposia (Schroeder et al. 2013) or to write editorials and commentaries (Nature 2012; Pettorelli et al. 2013). Though our analysis covers just a single ecology journal, and the generality of these observations across journals and across academic disciplines is still unclear, the current analysis plus results of Fox et al. (2016a) and of Fox, Burns & Meyer (2016) demonstrate clearly that gender differences in behaviour influence, albeit subtly, many aspects of the peer review process, and thus that diversity (gender, but also age and geographic diversity) in the peer review system should be promoted to foster a more valid and inclusive process.
It's clear from our results and other studies that author-suggested preferred reviewers rate papers more highly than do editor-selected reviewers, and that use of preferred reviewers increases the probability a paper will be accepted. We cannot distinguish whether preferred reviewers are rating papers more positively than they should, or whether editor-selected reviewers are less positive than they should be (possibly due to inadequate expertise). From the author perspective, our results clearly demonstrate that authors should submit names of preferred reviewers if the option to do so is available; regardless of why preferred reviewers rate papers higher, papers reviewed by preferred reviewers are scored more positively during peer review and are more likely to be accepted for publication. Journals, by contrast, should recognize that author-suggested reviewers rate papers much more highly than do editor-selected reviewers. Editors should thus consider who proposed the reviewer (editor or author), and possible biases, when evaluating reviewer comments and scores. Highly selective journals – those whose decisions emphasize broad general interest, significance and novelty – might consider eliminating the practice of allowing or requiring authors to suggest their own reviewers.
Acknowledgements
We thank the British Ecological Society (BES), owners of the journal Functional Ecology, for permitting us to use their peer review database for this project. Emma Sayer and members of the Fox laboratory at the University of Kentucky (Jacqueline Dillard, William Licht, Josiah Ritchey and Jesse Salazar) provided comments on an earlier draft of this paper. This work was reviewed and approved by the Internal Review Board at the University of Kentucky, IRB 14-0570-P4S. This research was funded in part by the British Ecological Society, the Kentucky Agricultural Experiment Station and the School of Information Science at the University of Kentucky.
Data accessibility
Data for this project are deposited in the Dryad Digital Repository: http://dx.doi.org/10.5061/dryad.37312 (Fox et al. 2016b) The deposited data allow recreation of most results in the paper. However, because the data set contains information on human subjects, the data available at Dryad are anonymized and lack variables that allow parts of the data set to be de-anonymized. Please see the metadata accompanying the Dryad submission for additional details.
