September 11, 2017 From rOpenSci (https://deploy-preview-304--ropensci.netlify.app/blog/2017/09/11/software-review-update/). Except where otherwise noted, content on this site is licensed under the CC-BY license.
The R package ecosystem now contains more than 10K packages, and several flagship packages belong under the rOpenSci suite. Some of these are: magick for image manipulation, plotly for interactive plots, and git2r for interacting with git
.
rOpenSci is a community of people making software to facilitate open and reproducible science/research. While the rOpenSci team continues to develop and maintain core infrastructure packages, an increasing number of packages in our suite are contributed by members of the extended R community.
In the early days we accepted contributions to our suite without any formal process for submission or acceptance. When someone wanted to contribute software to our collection, and we could envision scientific applications, we just moved it aboard. But as our community and codebase grew, we began formalizing standards and processes to control quality. This is what became our peer review process. You can read more about it in our recent blog post.
As our submissions have grown over the past couple of years, our standards around peer review have also changed and continue to evolve in response to changing community needs and updates to the R development infrastructure.
Although a large number of packages submitted to CRAN could also be part of rOpenSci, our submissions are limited to packages that fit our mission and are able to pass a stringent and time intensive review process.
Here, we summarize some of the more important changes to peer review at rOpenSci over the past year. The most recent information can always be found at </software-review/>.
Our Aims and Scope document what types of packages we accept from community contributors. The scope emerges from three main guidelines. First, we accept packages that fit our mission of enabling open and reproducible research. Second, we only accept packages that we feel our editors and community of reviewers are competent to review. Third, we accept packages for which we can reasonably endorse as improving on existing solutions. In practice, we don’t accept general packages. That’s why, for instance, our “data munging” category only applies to packages designed to work with specific scientific data types.
We’ve refined our focal areas from
to
You will note that we’ve removed data visualization. We’ve had some truly excellent data visualization packages come aboard, starting with plotly. But since then we’ve found data visualization is too general a field for us to confidently evaluate, and at this point have dropped it from our main categories.
We’ve also added geospatial and text analysis as areas where we accept packages that might seem more general or methods-y than we would otherwise. These are areas where we’ve built, among our staff and reviewers, topic-specific expertise.
Given that we accept packages that improve on existing solutions, in practice we generally avoid accepting more than one package of similar scope. We’ve also added clarifying language about what this entails and how we define overlap with other packages.
We now strongly encourage pre-submission inquiries to quickly assess whether the package falls into scope. Some of these lead to suggesting the person submit their package, while others are determined out-of-scope. This reduces effort on all sides for packages that be out-of-scope. Many authors do this prior to completing their package so they can decide whether to tailor their development process to rOpenSci.
To see examples of what has recently been determined to be out-of-scope, see the out-of-scope label in the onboarding repository.
As always, we’d like to emphasize that even when packages are out-of-scope, we’re very grateful that authors consider an rOpenSci submission!
Our packaging guide contains both recommended and required best practices. They evolve continually as our community reaches consensus on best practices that we want to encourage and standardize. Here are some of the changes we’ve incorporated in the past year.
object_verb()
function naming scheme to avoid namespace conflicts.devtools::check()
on all major platforms, that each package should have a test suite that covers all major functionality.xml2
package in general, but XML
package may be needed in some cases.Standards changes often take place because we find that both editors and reviewers are making the same recommendations on multiple packages. Other requirements are added as good practices become accessible to the broader community. For instance, CI and code coverage reporting have gone from recommended to required as the tooling and documentation/tutorials for these have made them more accessible.
Editors
As the pace of package submissions increases, we’ve expanded our editorial team to keep up. Maëlle Salmon joined us in February, bringing our team to four. With four, we need to be more coordinated, so we’ve moved to a system of a rotating editor-in-chief, who makes decisions about scope, assigns handling editors, and brings up edge cases for discussion with the whole team.
Welcome @ma_salmon to our editorial team for open peer review of #rstats software https://t.co/KsL0SF1b6K 1/2
— rOpenSci (@rOpenSci) February 16, 2017
The process our editors follow is summarized in our editors' guide, which will help bring editors up to speed when we further expand our team.
Automation
As submissions increase, the entire process benefits more from automation. Right now most steps of the review system are manual - we aim to automate as much as possible. Here’s a few things we’re doing or planning on:
goodpractice
on submissions and reporting back to the issue. Other rOpenSci specific checks, e.g., checking rOpenSci policies, are likely to be added in to this system.Other changes
JOSS harmonization/co-submission: For authors wishing to submit their software papers to the Journal of Open Source Software after acceptance, we have also begun streamlining the process. Editors check to make sure that the paper clearly states the scientific application, includes a separate .bib
file and that the accepted version of the software is deposited at Zenodo or Figshare with a DOI. Having these steps completed allows for a fast track acceptance at JOSS.
Reviewer template and guide: We now have a reviewer template - making reviews more standardized, and helping reviewers know what to look for. In addition, we have an updated reviewer guide that gives high level guidance, as well as specific things to look for, tools to use, and examples of good reviews. In addition, the guide gives guidance on how to submit reviews.
Badges: We now have badges for rOpenSci review. The badges show whether a package is under review or has been approved. Packages that are undergoing review or have been approved can put this badge in their README.
Get in touch with us on Twitter (@ropensci, or in the comments) if you have any questions or thoughts about our software review policies, scope, or processes.
To find out more about our software review process join us on the next rOpenSci Community Call
We hope to see you soon in the onboarding repository as a submitter or as a reviewer!