In mtmorgan/RedisParam: Provide a 'redis' back-end for BiocParallel

knitr::opts_chunk$set(
    eval =  RedisParam::rpalive()
)

Getting started

RedisParam implements a BiocParallel backend using redis, rather than sockets, for communication. It requires a redis server; see ?RedisParam for host and port specification. redis is a good solution for cloud-based environments using a standard docker image. A particular feature is that the number of workers can be scaled during computation, for instance in response to kubernetes auto-scaling.

Use

Ensure that a redis server is running, e.g., from the command line

$ redis-server

Manager and workers from a single R session

On a single computer, in R, load and use the RedisParam package in the same way as other BiocParallel backends, e.g.,

library(RedisParam)
p <- RedisParam(workers = 5)
result <- bplapply(1:7, function(i) Sys.getpid(), BPPARAM = p)
table(unlist(result))

Independently-managed workers

For independently managed workers, start workers in separate processes, e.g.,

Sys.getpid()
library(RedisParam)
p <- RedisParam(jobname = "demo", is.worker = TRUE)
bpstart(p)

Start and use the manager in a separate process. Be sure to use the same jobname =.

Sys.getpid()            # e.g., 8563
library(RedisParam)
p <- RedisParam(jobname = 'demo', is.worker = FALSE)
result <- bplapply(1:7, function(i) Sys.getpid(), BPPARAM = p)
unique(unlist(result)) # e.g., 9677

Independently started workers can be terminated from the manager

rpstopall(p)

Session info {.unnumbered}

This version of the vignette was built on r Sys.Date() with the following software package versions:

sessionInfo()

mtmorgan/RedisParam documentation built on July 22, 2023, 6:27 a.m.