knitr::opts_knit$set(root.dir = tempdir())
The Rcwl
package is aimed to be a simple and user-friendly way to manage command line tools and build data analysis pipelines in R using Common Workflow Language (CWL). It can be a bridge between heavy bioinformatics tools and pipeline to R/Bioconductor. More details about CWL can be found at http://www.commonwl.org.
if (!requireNamespace("BiocManager", quietly = TRUE)) install.packages("BiocManager") BiocManager::install("Rcwl")
The development version is also available to download from Github.
BiocManager::install("hubentu/Rcwl")
library(Rcwl)
The main class and constructor function is cwlParam
, which wrap a command line tool and its parameters in a cwlParam
object. Let's start with a simple example, echo hello world
.
First, we load the package and then define the input parameter for "echo", a string without a prefix. Just an id
option required.
input1 <- InputParam(id = "sth")
Second, create a cwlParam
object with baseCommand
for the command to execute and InputParamList
for the input parameters.
echo <- cwlParam(baseCommand = "echo", inputs = InputParamList(input1))
Now we have a command object to run. Let's send a string "Hello World!" to the object. Without defining the outputs, it will stream standard output to a temporary file by default.
echo$sth <- "Hello World!" echo
Let's run it. A list including the command executed, temporary output and logs. The output directory is the current folder by default, but it can be changed by setting outdir
option. All standard out and standard error stream can also be printed by setting stderr = ""
.
r1 <- runCWL(echo, outdir = tempdir()) r1 readLines(r1$output)
The writeCWL
function will write the cwlParam object to a CWL file for the command and YML for the inputs. Then it will invoke cwl-runner
by default to execute the two files. Thus the command tool cwl-runner
is required to be installed and available in the system path.
For the input parameters, three options need to be defined usually, id, type, and prefix. The type can be string, int, long, float, double, and so on. More detail can be found at: https://www.commonwl.org/v1.0/CommandLineTool.html#CWLType.
Here is an example from CWL user guide. Here we defined an echo
with different type of input parameters by InputParam
. The stdout
option can be used to caputre the standard output stream to a file.
e1 <- InputParam(id = "flag", type = "boolean", prefix = "-f") e2 <- InputParam(id = "string", type = "string", prefix = "-s") e3 <- InputParam(id = "int", type = "int", prefix = "-i") e4 <- InputParam(id = "file", type = "File", prefix = "--file=", separate = FALSE) echoA <- cwlParam(baseCommand = "echo", inputs = InputParamList(e1, e2, e3, e4), stdout = "output.txt")
Then we give it a try by setting values for the inputs.
echoA$flag <- TRUE echoA$string <- "Hello" echoA$int <- 1 tmpfile <- tempfile() write("World", tmpfile) echoA$file <- tmpfile r2 <- runCWL(echoA, outdir = tempdir()) r2$command
A similar example to CWL user guide. We can define three different type of array as inputs.
a1 <- InputParam(id = "A", type = "string[]", prefix = "-A") a2 <- InputParam(id = "B", type = InputArrayParam(items = "string", prefix="-B=", separate = FALSE)) a3 <- InputParam(id = "C", type = "string[]", prefix = "-C=", itemSeparator = ",", separate = FALSE) echoB <- cwlParam(baseCommand = "echo", inputs = InputParamList(a1, a2, a3))
Then set values for the three inputs.
echoB$A <- letters[1:3] echoB$B <- letters[4:6] echoB$C <- letters[7:9] echoB
Now we can check whether the command behaves as we expected.
r3 <- runCWL(echoB, outdir = tempdir()) r3$command
The outputs, similar to the inputs, is a list of output parameters. Three options id, type and glob can be defined. The glob option is used to define a pattern to find files relative to the output directory.
Here is an example to unzip a compressed gz
file. First, we generate a compressed R script file.
zzfil <- file.path(tempdir(), "sample.R.gz") zz <- gzfile(zzfil, "w") cat("sample(1:10, 5)", file = zz, sep = "\n") close(zz)
We define a cwlParam
object to use "gzip" to uncompress a input file.
ofile <- "sample.R" z1 <- InputParam(id = "uncomp", type = "boolean", prefix = "-d") z2 <- InputParam(id = "out", type = "boolean", prefix = "-c") z3 <- InputParam(id = "zfile", type = "File") o1 <- OutputParam(id = "rfile", type = "File", glob = ofile) gz <- cwlParam(baseCommand = "gzip", inputs = InputParamList(z1, z2, z3), outputs = OutputParamList(o1), stdout = ofile)
Now the gz
object can be used to uncompress the previous generated compressed file.
gz$uncomp <- TRUE gz$out <- TRUE gz$zfile <- zzfil r4 <- runCWL(gz, outdir = tempdir()) r4$output
Or we can use arguments
to set some default parameters.
z1 <- InputParam(id = "zfile", type = "File") o1 <- OutputParam(id = "rfile", type = "File", glob = ofile) Gz <- cwlParam(baseCommand = "gzip", arguments = list("-d", "-c"), inputs = InputParamList(z1), outputs = OutputParamList(o1), stdout = ofile) Gz Gz$zfile <- zzfil r4a <- runCWL(Gz, outdir = tempdir())
To make it for general usage, we can define a pattern with javascript to glob the output, which require node
to be installed in your system PATH.
pfile <- "$(inputs.zfile.path.split('/').slice(-1)[0].split('.').slice(0,-1).join('.'))"
Or we can use the CWL built in file property, nameroot
, directly.
pfile <- "$(inputs.zfile.nameroot)" o2 <- OutputParam(id = "rfile", type = "File", glob = pfile) req1 <- list(class = "InlineJavascriptRequirement") GZ <- cwlParam(baseCommand = c("gzip", "-d", "-c"), requirements = list(), ## assign list(req1) if node installed. inputs = InputParamList(z1), outputs = OutputParamList(o2), stdout = pfile) GZ$zfile <- zzfil r4b <- runCWL(GZ, outdir = tempdir())
We can also capture multiple output files with glob
pattern.
a <- InputParam(id = "a", type = InputArrayParam(items = "string")) b <- OutputParam(id = "b", type = OutputArrayParam(items = "File"), glob = "*.txt") touch <- cwlParam(baseCommand = "touch", inputs = InputParamList(a), outputs = OutputParamList(b)) touch$a <- c("a.txt", "b.gz", "c.txt") r5 <- runCWL(touch, outdir = tempdir()) r5$output
The CWL can work with docker to simplify your software management and communicate files between host and container. The docker container can be defined by the hints
or requirements
option.
d1 <- InputParam(id = "rfile", type = "File") req1 <- list(class = "DockerRequirement", dockerPull = "r-base") doc <- cwlParam(baseCommand = "Rscript", inputs = InputParamList(d1), stdout = "output.txt", hints = list(req1)) doc$rfile <- r4$output
r6 <- runCWL(doc)
The tools defined with docker requirements can also be run locally by disabling the docker option. In case your Rscript
depends some local libraries to run, an option from cwltools
, "--preserve-entire-environment", can be used to pass all environment variables.
r6a <- runCWL(doc, docker = FALSE, outdir = tempdir(), cwlArgs = "--preserve-entire-environment")
The CWL also can work in high performance clusters with batch-queuing system, such as SGE, PBS, SLURM and so on, using the Bioconductor package BiocParallel
. Here is an example to submit jobs with "Multiicore" and "SGE". A more detailed example can be found (https://hubentu.github.io/others/Rcwl_RNASeq.html).
library(BiocParallel) sth.list <- as.list(LETTERS) names(sth.list) <- LETTERS ## submit with mutlicore result1 <- runCWLBatch(cwl = echo, outdir = tempdir(), inputList = list(sth = sth.list), BPPARAM = MulticoreParam(26)) ## submit with SGE result2 <- runCWLBatch(cwl = echo, outdir = tempdir(), inputList = list(sth = sth.list), BPPARAM = BatchtoolsParam(workers = 26, cluster = "sge", resources = list(queue = "all.q")))
We can connect multiple tools together into a pipeline. Here is an example to uncompress an R script and execute it with Rscript
.
Here we define a simple Rscript
tool without using docker.
d1 <- InputParam(id = "rfile", type = "File") Rs <- cwlParam(baseCommand = "Rscript", inputs = InputParamList(d1)) Rs
Test run:
Rs$rfile <- r4$output tres <- runCWL(Rs, outdir = tempdir()) readLines(tres$output)
The pipeline includes two steps, decompressed by GZ
and compiled by Rs
. The input file is a compressed file and the output would be the output Rout
from Rs
.
First we need to define the direct inputs and outputs from GZ
and Rs
.
i1 <- InputParam(id = "cwl_zfile", type = "File") o1 <- OutputParam(id = "cwl_cout", type = "File", outputSource = "Compile/output")
For the input "cwl_zifle", it refers to the GZ
input zfile
. The output "cwl_cout" will be the outcome of Rs
output Rout
.
The cwlStepParam
is used to define inputs
and outputs
from previous step. Then it connects with the two steps with Step
function. The run
option refer to the corresponding cwlParam
object and the In
option in steps should be linked to the input parameters defined by cwlStepParam
. In the end, we use +
to connect all steps.
cwl <- cwlStepParam(inputs = InputParamList(i1), outputs = OutputParamList(o1)) s1 <- Step(id = "Uncomp", run = GZ, In = list(zfile = "cwl_zfile")) s2 <- Step(id = "Compile", run = Rs, In = list(rfile = "Uncomp/rfile")) cwl <- cwl + s1 + s2 cwl
Let's run the pipeline.
cwl$cwl_zfile <- zzfil r7 <- runCWL(cwl, outdir = tempdir()) readLines(r7$output)
The scattering feature can specifies the associated workflow step or subworkflow to execute separately over a list of input elements. To use this feature, ScatterFeatureRequirement
must be specified in the workflow requirements. Different scatter
methods can be used in the associated step to decompose the input into a discrete set of jobs. More details can be found at: https://www.commonwl.org/v1.0/Workflow.html#WorkflowStep.
Here is an example to execute multiple R scripts. First, we need to set the input and output types to be array of "File", and add the requirments. In the "Compile" step, the scattering input is required to be set with the scatter
option.
i2 <- InputParam(id = "cwl_rfiles", type = "File[]") o2 <- OutputParam(id = "cwl_couts", type = "File[]", outputSource = "Compile/output") req1 <- list(class = "ScatterFeatureRequirement") cwl2 <- cwlStepParam(requirements = list(req1), inputs = InputParamList(i2), outputs = OutputParamList(o2)) s1 <- Step(id = "Compile", run = Rs, In = list(rfile = "cwl_rfiles"), scatter = "rfile") cwl2 <- cwl2 + s1 cwl2
Multiple R scripts can be assigned to the workflow inputs and executed.
cwl2$cwl_rfiles <- c(r4b$output, r4b$output) r8 <- runCWL(cwl2, outdir = tempdir()) r8$output
The function plotCWL
can be used to visualize the relationship of inputs, outputs and the analysis for a tool or pipeline.
plotCWL(cwl)
Here we build a tool with different types of input parameters.
e1 <- InputParam(id = "flag", type = "boolean", prefix = "-f", doc = "boolean flag") e2 <- InputParam(id = "string", type = "string", prefix = "-s") e3 <- InputParam(id = "option", type = "string", prefix = "-o") e4 <- InputParam(id = "int", type = "int", prefix = "-i", default = 123) e5 <- InputParam(id = "file", type = "File", prefix = "--file=", separate = FALSE) e6 <- InputParam(id = "array", type = "string[]", prefix = "-A", doc = "separated by comma") mulEcho <- cwlParam(baseCommand = "echo", id = "mulEcho", label = "Test parameter types", inputs = InputParamList(e1, e2, e3, e4, e5, e6), stdout = "output.txt") mulEcho
Some input parameters can be predefined in a list, which will be converted to select options in the webapp. An upload
parameter can be used to defined wether to generate an upload interface for the file type option. If FALSE, the upload field will be text input (file path) instead of file input.
inputList <- list(option = c("option1", "option2")) app <- cwlShiny(mulEcho, inputList, upload = TRUE) runApp(app)
We can wrap an R function to cwlParam
object by simply assigning the R function to baseCommand
. This could be useful to summarize results from other tools in a pipeline. It can also be used to benchmark different parameters for a method written in R. Please note that this feature is only implemented by Rcwl
, but not available in the common workflow language.
fun1 <- function(x)x*2 testFun <- function(a, b){ cat(fun1(a) + b^2, sep="\n") } assign("fun1", fun1, envir = .GlobalEnv) assign("testFun", testFun, envir = .GlobalEnv) p1 <- InputParam(id = "a", type = "int", prefix = "a=", separate = F) p2 <- InputParam(id = "b", type = "int", prefix = "b=", separate = F) o1 <- OutputParam(id = "o", type = "File", glob = "rout.txt") TestFun <- cwlParam(baseCommand = testFun, inputs = InputParamList(p1, p2), outputs = OutputParamList(o1), stdout = "rout.txt") TestFun$a <- 1 TestFun$b <- 2 r1 <- runCWL(TestFun, cwlArgs = "--preserve-entire-environment") readLines(r1$output)
The runCWL
function wrote the testFun
function and its dependencies into an R script file automatically and call Rscript
to run the script with parameters. Each parameter requires a prefix from corresponding argument in the R function with "=" and without a separator. Here we assigned the R function and its dependencies into the global environment because it will start a new environment when the vignette compiled.
The Rcwl
package can be utilized to develop pipelines for best practices of reproducible research, especially for Bioinformatics study. Multiple Bioinformatics pipelines, such as RNASeq alignment, quality control and quantification, DNASeq alignment and variant calling, have been developed based on the tool in an R package RcwlPipelines
, which contains the CWL recipes and the scripts to create the pipelines. Examples to analyze real data are also included.
The package is currently available in github.
To install the package.
BiocManager::install("hubentu/RcwlPipelines")
More recipes will be collected in this package and we would like to invite community to submit more pipelines to this package.
Plenty of Bioinformatics tools and workflows can be found from github in CWL format. They can be imported to cwlParam
object by readCWL
function, or can be used directly.
Most of the Bioinformatics software are available in docker containers, which can be very convenient to be adopted to build portable CWL tools and pipelines.
sessionInfo()
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.