R Validation Hub

Updates & Pharma Repository

Juliane Manitz and Coline Zeballos
on behalf of the R Validation Hub team

2023-10-25

🗓️ Agenda

  • Introduction & Updates10 min; Juliane Manitz

  • Repositories Workstream 10 min; Coline Zeballos

Slides Available!

pharmar.github.io/events-rpharma2023

👋 Who We Are

The R Validation Hub is a collaboration to support the adoption of R within a biopharmaceutical regulatory setting (pharmaR.org)

  • R Consortium Working Group which grew out of R/Pharma 2018

  • Led by participants from ~10 organizations

  • With frequent involvement from health authorities

  • And subscribers from ~60 organizations spanning multiple industries

📣 Updates

👷‍♂️ New Opportunities

  • New R validation hub lead is Doug Kelkhoff
  • More ways to get involved

White Paper

Guidance on compliant use of R and management of packages

Andy Nicholls et al.

New! Communications

Connecting validation experts across the industry

Juliane Manitz

New! Repositories

Building a public, validation-ready resource for R packages

Coline Zeballos

{riskmetric}

Gather and report on risk heuristics to support validation decision-making

Eric Milliman

{riskassessment}

A web interface to {riskmetric}, supporting review, annotation and cataloging of decisions

Aaron Clark

New! {riskscore}

An R data package capturing risk metrics across all of CRAN

Aaron Clark

🗝 Key Policy Updates!

If nothing else, take this home!

  • The FDA appears to accept .R files through their eSUB portal1 .
  • The FDA has released a draft of a new Computer Software Assurance2 guideline that seems to be increasingly the basis for their evaluation of R.

🤝 Affiliate: CAMIS

Comparing Analysis Method Implementations in Software

  • A cross-industry group formed of members from PHUSE, PSI, and ASA.

  • Released a white paper providing guidance on appropriate use of stats methods, e.g.

    • Don’t default to the defaults
    • Be specific when drafting analysis plans, including precise methods & options

  • Currently building a repository of methods comparisons at psiaims.github.io/CAMIS.

  • Check out their contributors page to learn how you can get involved!

📜 Workstream Report

Case Studies

7 Companies Shared their Approach to Package Validation

Commonalities

  • Categorized risk (hi/med/lo)
  • Heavily weight unit testing
  • Base & recommended packages “trusted”

Differences

  • Risk stratification, e.g, coverage cutoff

  • Managing risk

    • human-in-the-middle review
    • restricted package subset
    • adding bespoke testing

Themes

  • Time & resource intensive
  • Requires unique intersection of expertise
  • Challenges of lifecycle management of ecosystem

{riskmetric} Roadmap

  • Ease of use:
    Wrapper functions for a complete workflow, prettier outputs
  • Metric completeness:
    Implement metrics for as many package sources as possible. Chain sources together to create more complete assessments
  • Modular addititions:
    Allow users to easy add custom assessments, create optional assessments based on community packages, e.g. oysteR, srr, pkgstats
  • Focusing on metrics and scoring:
    Making custom weighting more robust and convenient. Guidance materials on weighting specific assessments based on community feedback and our own views on best practices.

{riskassessment} App

“Best App” Award at Shiny Conf 2023

  • Face lifts:
    Improved Report Builder & Database View. Better dependency inspection

  • Admin tools:
    Org-level customization using config file. Allow admin user role management

  • Source file explorer (original code provided by GSK):
    Allows to browse test files with brand-new add-on to browse help and source code of a function in the same module

  • Automatic risk sssessment:
    Write custom code to automatically categorize packages into low/medium/high risk

Check out the Risk Assessment Demo App

📦 Repositories

Repositories Workstream

Supporting a transparent, open, dynamic, cross-industry approach of establishing and maintaining a repository of R packages.

  • Taking ample time to engage stakeholders

    • Validation leads across the industry
    • Active health authority involvement
    • Analytic environment admins and developers

  • Considering the possibilities

    • Mapping needs to solutions that meet the industry where it is
    • …while building the path for it to move forward

How did we get here?

  • Our whitepaper is widely adopted

  • But implementing it is inconsistent & laborious

    • Variations throughout industry pose uncertainty
    • Sharing software with health authorities is a challenge
    • Health authorities, overwhelmed by technical inconsistencies, are more likely to question software use

  • We feel the most productive path forward is a shared ecosystem

Work to-date

Building consensus in package evaluation and distribution…

  1. Who needs a repository anyways?
  2. Stakeholder engagement 3mo
  3. Product refinement and proof-of-concept planning 1mo
  4. POC development 2mo

What does a solution look like?

Closing the CRAN gap for the Pharma Use Case

  • Reproducibility guidelines
  • Standard, public assessment of packages
  • Avenues for communicating about implementations, bugs, security

The Proposal so Far

🧗 “Leaps of Faith”

  • A “Golden” Base Image
    to establish ground truth for testing.
  • Rethinking requirements
    testing, external vetting (CRAN) and adoption are sufficient for Scripted Testing needs - are new requirements necessary?
  • Expectations of Public Communication
    industry-standard communication channels.
  • Nearly all meaningful assessment can be automated
    edge cases (malicious code, methods debate) are better handled by transparent community engagement.

The Proposal so Far

Join the R validation Hub!

We’re excited to champion new ways of working that bring Pharma’s together.