R Package Validation

R/Pharma Workshop, 2020 <br>

<div class="row">
<div class="column">
  <img class="profile" src="images/andy-nicholls.jpg" alt="Andy Nicholls">
  <div class="profile-name">Andy Nicholls</div>
</div>

<div class="column">
  <img class="profile" src="images/marly-gotti.jpg" alt="Marly Gotti">
  <div class="profile-name">Marly Gotti</div>
</div>
</div>

]

---

R/Pharma Workshop, 2020 <br>
Marly Gotti

]
.pull-farright[

]

---

# Outline

- What is a *validated* software?

- Introduction to the `riskmetric` package

- Introduction to the Risk Assessment Shiny Application

- Example using `riskmetric`

- Example using the Risk Assessment App

- Work in groups (~20 mins)

---

# What is a validated software?

---

# Validation

- No statistical software can be purchased as "pre-validated" software in the regulatory context. Validation is entirely on the end user to implement. **Commercial statistical analysis softwares are not a "pre-validated" software.**

- Given the dominance of some of these softwares in the industry, many pharmaceutical companies have substantial evidence collected over a long period of time, **we make the decision to use certain procedures based on their history of extensive previous use.**

- Why can't we apply the same thought process for the use of R packages? **R packages have been extensively used.**

<br><br><br><br>
<small>
References: [PSI Special Interest Group: Application and Implementation of Methodologies in Statistics (AIMS)](https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjOxuW7uYfsAhVEwFkKHVUzBOQQFjAAegQIBRAB&url=https%3A%2F%2Fwww.psiweb.org%2Fdocs%2Fdefault-source%2Fdefault-document-library%2Fwant-to-use-r-here-are-some-top-tips-before-you-start-20171027.docx%3Fsfvrsn%3Dab5dfdb_0&usg=AOvVaw1JWwlz0z_k0V5UPlBZxUUD).
</small>

---

layout: false
class: inverse, middle, center
# Which R packages can we <br> add to our safe list?

---

# The **riskmetric** package

- Some important questions:

- Is the package on CRAN? 
  - Does it have unit tests?
  - What is its test coverage?
  - Does the package have documentation?
  - How many times has the package been downloaded?
  - Who are the authors?

- `riskmetric`: Measures the risk of using an R package. It is calculated based on a number of metrics meant to evaluate development best practices, code documentation, community engagement, and development sustainability. This risk does *not* represent
  - the risk of damaging the system in which the package is installed, or
  - the risk that the internal (statistical) functions are correct.

- Example:
  - low risk package: `ggplot2` (risk = 0.21)
  - High risk package: `tidymodels/stacks` (risk = 0.9). Does this mean we can't use it?

---

layout: false
class: inverse, middle, center
# The Risk Assessment Shiny Application

---

# The Risk Assessment Shiny Application

- Framework to provide user feedback on the risk calculated by riskmetric

- Inherits the advantages of shiny: no need to code, interactive way to measure risk

- Allows authorized personnel to comment on the metrics

- Historical comments

- Reporting tool

- Adopting R within a GxP R server

---

[`http://github.com/pharmaR/riskmetric`](http://github.com/pharmaR/riskmetric)

---

# Example using **riskmetric**

```r
# devtools::install_github("pharmar/riskmetric")

library(riskmetric)

metrics <- pkg_ref(c("ggplot2", "stacks")) %>%
  as_tibble() %>%
  pkg_assess() %>%
  pkg_score()

metrics
```

```
## # A tibble: 2 x 16
##   package version pkg_ref          pkg_score news_current has_vignettes has_bug_reports… bugs_status license
##   <chr>   <chr>   <list<pkg_ref>>      <dbl> <pkg_scor>   <pkg_scor>    <pkg_scor>       <pkg_scor>  <pkg_s>
## 1 ggplot2 3.3.2   ggplot2<install>     0.217 1            1             1                0.4         NA     
## 2 stacks  0.0.0.… stacks<install>      0.833 0            0             0                0.0         NA     
## # … with 7 more variables: export_help <pkg_scor>, downloads_1yr <pkg_scor>, has_website <pkg_scor>,
## #   has_maintainer <pkg_scor>, has_news <pkg_scor>, has_source_control <pkg_scor>, covr_coverage <pkg_scor>
```

---

# Example using **riskmetric**

```
# devtools::install_github("pharmar/riskmetric")

library(riskmetric)

metrics <- pkg_ref(c("ggplot2", "stacks")) %>%
  as_tibble() %>%
  pkg_assess() %>%
  pkg_score()

metrics
 
```

<br>

<table class="table table-striped table-hover" style="font-size: 15px; margin-left: auto; margin-right: auto;">
  <thead>
   <tr>
    <th style="text-align:left;"> package </th>
    <th style="text-align:left;"> version </th>
    <th style="text-align:right;"> risk </th>
    <th style="text-align:right;"> downloads_1yr </th>
    <th style="text-align:right;"> bugs_status </th>
    <th style="text-align:right;"> has_maintainer </th>
    <th style="text-align:right;"> news_current </th>
    <th style="text-align:right;"> has_vignettes </th>
    <th style="text-align:right;"> export_help </th>
    <th style="text-align:left;"> ... </th>
   </tr>
  </thead>
 <tbody>
   <tr>
    <td style="text-align:left;"> ggplot2 </td>
    <td style="text-align:left;"> 3.3.2 </td>
    <td style="text-align:right;"> 0.2174392 </td>
    <td style="text-align:right;"> 0.9907296 </td>
    <td style="text-align:right;"> 0.4 </td>
    <td style="text-align:right;"> 1 </td>
    <td style="text-align:right;"> 1 </td>
    <td style="text-align:right;"> 1 </td>
    <td style="text-align:right;"> 1 </td>
    <td style="text-align:left;"> ... </td>
   </tr>
   <tr>
    <td style="text-align:left;"> stacks </td>
    <td style="text-align:left;"> 0.0.0.9000 </td>
    <td style="text-align:right;"> 0.8333333 </td>
    <td style="text-align:right;"> 0.0000000 </td>
    <td style="text-align:right;"> 0.0 </td>
    <td style="text-align:right;"> 1 </td>
    <td style="text-align:right;"> 0 </td>
    <td style="text-align:right;"> 0 </td>
    <td style="text-align:right;"> 1 </td>
    <td style="text-align:left;"> ... </td>
   </tr>
 </tbody>
 </table>

---

layout: false
class: inverse, middle, center
# An example using <br> the **Risk Assessment App**

[`https://github.com/pharmaR/risk_assessment`](https://github.com/pharmaR/risk_assessment)

---

# Lets measure some risks!

- Breakout rooms.

- We will use the following set of packages: `c("dplyr", "lme4", "bshazard")`.

- Classify the packages:
  - Classify the packages by risk: low, medium, high. 
  - Which metrics were more important in your decision?
  - Which other information would have helped to reach your decision? (e.g. other metrics, read vignettes, Google the authors)
  - How would you go about reducing the risk?

- Time: ~15min.

---
layout: false
class: inverse, middle, center
# Break