In vjcitn/BiocBBSpack: support for build system

Introduction

Building and distributing packages for Linux, macOS, and Windows in a continuous integration framework is a hallmark of the Bioconductor project. The Bioconductor Build System has evolved over years and has proven robust and resilient as the project has grown in scope, and as the underlying languages and infrastructures have evolved.

The package BiocBBSpack is an exploration of how recently developed tools in R and Bioconductor can contribute to the performance and maintainability of the build system.

Simple example

With an example we illustrate some of the most basic concepts.

The PackageSet class

suppressPackageStartupMessages({
library(BiocBBSpack)
})
coreset() # a simple collection of relatively lighteight packages
ps1 = PackageSet(coreset(), biocversion="3.10")
ps1
ps1 = add_dependencies(ps1)
ps1

The large number of dependencies for this small group of packages is characteristic. One reason for working on this package is to provide a framework for experimenting with package management processes to measure scalability. We would like to pass the responsibility for efficient installation of essential packages to R or Bioconductor utilities that are designed for this purpose. In this case, our current understanding is that the installation of dependencies can be parallelized by passing a suitable value for parameter Ncpus to install.packages via BiocManager::install.

Populating a folder with git clones of sources in a PackageSet

tf = tempfile()
dir.create(tf)
populate_local_gits(ps1, tf)
dir(tf)

For a large collection of packages this can be time-consuming but can be regarded as a one-time-charge, because updates can be conducted as needed via git pull.

Identifying an out-of-date source clone

Here we artificially back-version a package and show how to find it, comparing its version value to that in the Bioconductor repository. This task could be carried out at the git level, but that can occur in a more mature version if necessary.

okdesc = readLines(paste0(tf, "/parody/DESCRIPTION"))
baddesc = gsub("Version.*", "Version: 1.0", okdesc)
writeLines(baddesc, paste0(tf, "/parody/DESCRIPTION"))
local_gits_behind_bioc(tf)
writeLines(okdesc, paste0(tf, "/parody/DESCRIPTION")) # restore

Summary of simple example

• We use a character vector of package names to specify the build task. The PackageSet class manages relevant information about these packages:
  • package names,
  • dependencies as identified via Depends, Imports, LinkingTo, or Suggests; the latter are essential for building vignettes,
  • additional information as returned by BiocPkgTools::biocPkgList()
• We use the concepts of gits and gitspath to refer to collections of cloned package source repositories held in a single folder.
  • populate_local_gits will attempt to clone sources into a specified folder,
  • local_gits_behind_bioc will list packages for which the version specified in DESCRIPTION is earlier than the one reported in BiocPkgTools::biocPkgList(),
  • crucially, there is a slot in the PackageSet class specifying the Bioconductor version relevant for object use.

Conceptual framework

Now that we have seen a little bit of the functionality of BiocBBSpack, let's try to spell out the concepts that must be covered in a build system support package.

Generating current source set

• base source provider: a collection of git repositories owned by Bioconductor
• package set: a list of packages and metadata, to be managed with PackageSet class
• gitspath: a folder where all repositories corresponding to elements of a package set are cloned; these clones are local_gits
• lagging_local_gits: clones whose version tag in DESCRIPTION is earlier than the corresponding one in the base source provider
• make_all_current_in_gitspath: task of running git pull so that there are no lagging_local_gits

Provisioning builder infrastructure

• ubuntu package list: list of arguments to apt-get install to achieve a capable build machine
• manual infrastructure spec: directions to add infrastructure (typically runtime libraries and header sets) to achieve a capable build machine
• full CRAN package set: PackageSet of all CRAN packages needed for a build
• current-full-R: installation of R with all packages required to build any Bioconductor package installed

Generating logs and tarballs

Package build attempts may fail, and logs must be kept and conveyed. When build succeeds, logs and tarballs are generated

• current log-tarball set: convergent result of iteration of the build system -- we iterate because a given attempt may fail owing to resource contention or other random challenges
• lagging-log-tarballs: logs and tarballs for builds of source packages whose version is earlier than that at the base source provider

Generating binaries

• current binary set: binary builds of packages from current log-tarball set
• lagging-binaries: binary builds with versions earlier than that at the base source provider

vjcitn/BiocBBSpack documentation built on Nov. 21, 2020, 1:18 p.m.