SnowfastParam-class: Fast Simple Network of Workstations (SNOW)-style Parallel...

In kuwisdelu/matter: Out-of-core statistical computing and signal processing

Fast Simple Network of Workstations (SNOW)-style Parallel Backend

Description

This class provides a enhanced version of the SnowParam parallel backend from BiocParallel based on the newer PSOCK cluster implementation from the parallel package.

Usage

## Instance creation
SnowfastParam(workers = snowWorkers(),
    tasks = 0L, stop.on.error = TRUE, progressbar = FALSE,
    RNGseed = NULL, timeout = WORKER_TIMEOUT,
    exportglobals = TRUE, exportvariables = TRUE,
    resultdir = NA_character_, jobname = "BPJOB",
    force.GC = FALSE, fallback = TRUE, useXDR = FALSE,
    manager.hostname = NA_character_, manager.port = NA_character_, ...)

## Additional methods documented below

Arguments

workers	Either the number of workers or the names of cluster nodes.
tasks	The number of tasks per job. See SnowParam for details.
stop.on.error	Enable stop on error. See SnowParam for details.
progressbar	Enable text progress bar. See SnowParam for details.
RNGseed	The seed for random number generation. See SnowParam for details.
timeout	Time (in seconds) allowed for workers to complete a task. See SnowParam for details.
exportglobals	Export base::options() from manager to workers? See SnowParam for details.
exportvariables	Automatically export variables defined in the global environment and used by the function? See SnowParam for details.
resultdir	Job results directory. See SnowParam for details.
jobname	The name of the job for logging and results. See SnowParam for details.
force.GC	Whether to invoke the garbage collector after each call to FUN. See SnowParam for details.
fallback	Fall back to using SerialParam if the cluster is not started and the number of workers is no greater than 1? See SnowParam for details.
useXDR	Should data be converted to network byte order when serializing from the manager to the workers? The default (FALSE) assumes all nodes in the cluster are little endian. Passed to makePSOCKcluster.
manager.hostname	Host name of the manager node.
manager.port	Port on the manager with which workers communicate.
...	Additional arguments passed to makePSOCKcluster.

Details

SnowfastParam is a faster but somewhat more limited version of SnowParam. Like SnowParam, it uses simple network of workstations (SNOW)-style parallelism so that it is available on all operating systems.

The workers are initialized in parallel, so cluster startup can often be significantly faster than SnowParam if utilizing a large number of workers.

Because the workers are started using makePSOCKcluster from the parallel package rather than using BiocParallel's startup script, some features of BiocParallel-managed backends such as logging are unsupported or only partially available.

The default parameter useXDR=TRUE assumes that all nodes are little endian so that data can be sent to workers without the overhead of conversion to network byte order. This should result in overall faster performance for large datasets on compatible clusters.

SnowfastParam is intended to be used as a drop-in replacement for SnowParam in most situations and as a more stable alternative to MulticoreParam for long-running jobs in graphical environments like RStudio where forking the R process should be avoided.

Value

An parallel backend derived from class BiocParallelParam.

Methods

BiocParallelParam methods:

bpstart(x):

Start the cluster.

bpstop(x):

Stop the cluster.

Additional methods documented in BiocParallelParam.

Author(s)

Kylie A. Bemis

See Also

SnowParam, MulticoreParam

Examples

x <- replicate(1000, runif(200), simplify=FALSE)

bp <- SnowfastParam(workers=4, tasks=20, progressbar=TRUE)

ans <- chunkLapply(x, sum, BPPARAM=bp)

kuwisdelu/matter documentation built on Oct. 19, 2024, 10:31 a.m.