Introduction

riskmetric provides a workflow to evaluate the quality of a set of R packages that involves four major steps. The workflow can help users to choose high quality R packages, improve package reliability and prove the validity of R packages in a regulated industry. The four steps include:

  1. Finding a source for package information pkg_ref()
    this can be any source of package metadata, such as a
    • package repository webportal from which we can scrape metadata
    • built and installed package in a local library
    • directory containing package source code
  2. Assessing the package under validation criteria assess()
    collect a set of metadata for scoring risk, this might include
    • a profile of the number of active users over the last month
    • the availability of documentation for package functions
    • the use of a unit testing framework
    • the test coverage
  3. Scoring assessment criteria score()
  4. Summarize scores into an aggregate risk metric summarize_risk()

Package Reference Objects

Before we get started assessing packages, we need a place to aggregate all of our package metadata. To handle this, we use the pkg_ref class. As package metadata is requested, it will be housed within this object for reuse, avoiding repeatedly having to scrape, download, parse or derive the same metadata multiple times.

library(riskmetric)
pkg_ref("riskmetric")
#> <pkg_install, pkg_ref> riskmetric v0.1.0.9000
#> $path
#>   [1] "/home/user/username/R/3.6/Resources/library/riskmetric"
#> $source
#>   [1] "pkg_install"
#> $version
#>   [1] '0.1.0.9000'
#> $name
#>   [1] "riskmetric"
#> $description...
#> $help...
#> $help_aliases...
#> $news...

A package reference defines one of many different ways of gathering information about a package and its development practices. The targets of this reference could be a source code directory, installation in a package library, or simply a url in a web-hosted repository from which metadata can be scraped. Each piece of metadata is captured through a callback function, which can be uniquely handled for each source of package information.

Furthermore, pkg_ref objects can be coerced to a tibble for easier, parallel assessment of packages.

library(dplyr)
package_tbl <- pkg_ref(c("riskmetric", "utils", "tools")) %>%
  as_tibble()
#> # A tibble: 3 x 3
#>   package    version    pkg_ref            
#>   <chr>      <chr>      <list<pkg_ref>>    
#> 1 riskmetric 0.1.0.9000 riskmetric<install>
#> 2 utils      3.6.1      utils<install>     
#> 3 tools      3.6.1      tools<install>

Core Verbs

In riskmetric, the core operations of package validation are handled in a series of verbs. Once package reference objects have been defined, they can be processed by chaining these foundational operations. These verbs, as well as many of the features of package assessment, are designed to be highly extensible to make it easier for package validation to be a community effort.

See the extending-riskmetric vignette for more details

Creating a Package Reference

We’ll use the tibble from above as our example

package_tbl <- pkg_ref(c("riskmetric", "utils", "tools")) %>%
  as_tibble()

Assess

The first component of the validation process is an assessment. This is used for iterating over assessment functions which serve to fetch any necessary package metadata and produce an atomic value reflective of each assessment. For example, when assessing whether a package’s NEWS files are up-to-date, it may return a logical vector indicating which NEWS files have entries for the current package version.

The assess function iterates over a list of assessment functions, adding a column per assessment. By default, it will use all the available assessment functions (all functions in the riskmetric exports beginning with assess_*). Once applied, an assessment produces a pkg_metric object.

assess will use all available riskmetric assessment functions by default. A subset of functions, or additional user-defined functions, can be used by passing a list of functions to the assessments argument.

Score

After gathering the available metric metadata, every metric object is scored. Scoring a metric translates the assessment atomic values into a single numeric score. For example, we might score the number of available NEWS files by giving a package a perfect score (1.0) only when all NEWS files have been updated for the latest version, or alternatively we might score it as the fraction of NEWS files that are up-to-date.

Keeping the scoring separated from the assessment allows us to easily iterate on how scores are defined and allows for the scoring function to be easily overwritten.

Summarizing Package Risk

What we ultimately want is a single numeric value which indicates the “risk” involved with using a given package. For the dplyr-savvy, this will look quite familiar. We’ve defined a default summarizing function summarize_risk. It accepts the full data.frame and performs a default risk assessment assuming all the available riskmetric assessments have been performed.

How you can help…

As you can see, the package is currently quite bare-bones and nobody would reasonably choose packages based solely on the existence of a NEWS file.

Our priority so far has been to set up an extensible framework as the foundation for a community effort, and that’s where you come in! There are a few things you can do to get started.

  1. Check out the extending-riskmetric vignette to see how to extend the functionality with your own metrics
  2. Take part in the discussion about which metrics are captured and how they are measured
  3. Propose a new metric on the riskmetric GitHub where we can further discuss new metric proposals
  4. Help us to develop new metrics and package functionality