R/tab_wf_cwl.R

In systemPipeR/systemPipeS: systemPipeShiny: An Interactive Framework for Workflow Management and Visualization

## UI
#' @importFrom shinyAce aceEditor
#' @importFrom shinyWidgets radioGroupButtons
wf_cwlUI <- function(id){
    ns <- NS(id)
    tagList(
        tabTitle("Common Workflow Language Files (optional)"),
        renderDesc(id = ns("desc"),
        '
        #### To use this tab

        ***

        **This tab helps you to check if the CWL files can be rendered to the correct command line code.**

        You need 3 files to work on this tab: a CWL defination file, a CWL
        input file and a SPR targets file.

        - If you have **Add to task** a targets file from the **Targets** tab,
        the table will be displayed here, otherwise an example table will be
        provided here.
        - Examples of CWL file and the input file are provided or you can upload
        your own.

        Click the "**Parse**" button to see the parsing results below.

        If you plan to use all defaults, you can just **skip** to
        prepare these CWL files in SPS.

        #### CWL file

        ***

        In the latest version of SPR, all command line tools and their running
        commands are defined in the [Common Workflow Language{blk}](https://www.commonwl.org/) (CWL) files.

        There are two files required to run command line in CWL:

        - *.cwl* is the command line file and it defines the command line
        - *.yaml* or *.yml* is the input file, and it defines the input variables
        that are used in *.cwl* file.

        Both are in [yaml{blk}](https://yaml.org/) format.

        #### CWL in SPR

        ***

        1. SPR has its own R-based CWL parser, so users do not need to install
        pyhton and node.js. However, the CWL files may be parsed slightly
        different in SPR than the native `cwl-runner`. So it is good to use
        this tab to test your CWL files to see if they can be parsed as expected.
        2. When a workflow template is generated by SPR, the default input file
        *.yml* is been copied and when you check this file, you often find that
        this file has some strings look like `_xx_`. They are *targets file*
        variables. When the `renderWF` function is run in SPR, these variables
        will be replaced by the designated columns in the targets file:

        For example, we have a record in my *.yml*

        ```
        fq1:
          class: File
          path: _FASTQ_PATH1_
        ```

        We have a targets file with columns like this

        | FileName1 | FileName2 |...|
        |:---:|:---:|:---:|
        | f1_1      | f2_1      |...|
        | f1_2      | f2_2      |...|
        | f1_x      | f2_x      |...|
        | ...       | ...       |...|

        When we run `renderWF`

        ```
        args <- renderWF(args, inputvars = c(FileName1 = "_FASTQ_PATH1_"))
        ```
        We am saying to replace the `_FASTQ_PATH1_` records in *.yml* with `FileName1`
        column from my targets file. Internally, SPR will treat the input yaml as the following
        when parse the CWL commands:

        ```
        fq1:
          class: File
          path: [f1_1, f1_2, f1_x, ...]
        ```

        #### Read more
        Read more how CWL works in SPR on [our website{blk}](https://systempipe.org/sp/spr/cwl/cwl_and_spr/)
        '),
        spsHr(),
        box(
            title = "Parse CWL files",
            closable = FALSE, collapsible = FALSE,
            status = "info",
            width = 12,
            class = "center-block",
            HTML(
                "
                <ul>
                  <li>
                    Choose which targets columns to replace CWL input variables from
                    the box below and hit <b>Parse</b>.
                  </li>
                </ul>
                "),
            fluidRow(
                class = "center-block text-center",
                actionButton(ns("parse"),
                             label = "Parse CWL",
                             icon("angle-right")) %>%
                    bsHoverPopover(
                        "Parse the CWL file",
                        "When you have loaded all 3 files by using examples or uploads,
                        select which variables you want to replace and parse.",
                        "bottom"
                    )
            )
        ),
        box(
            title = "Choose which targets columns to replace CWL input variables:",
            class = "center-block", width = 12, closable = FALSE, collapsible = TRUE,
            fluidRow(id = ns("replace-box"))
        ),
        box(
            class = "center-block",
            closable = FALSE, width = 12, collapsible = TRUE,
            title = "Parse Results",
            verbatimTextOutput(ns("parse_out"))
        ),
        spsHr(),
        bsplus::bs_accordion(id = ns("cwl_panel")) %>%
            bsplus::bs_set_opts(panel_type = "info") %>%
            bsplus::bs_append(
                "Targets file",
                fluidRow(
                    h3("Load targets table"),
                    tags$ul(
                        tags$li(
                            id = ns("targets_eg"),
                            class = "text-warning",
                            "You have not setup a workflow environment, loaded an external example for you."
                        ),
                        tags$li(
                            id = ns("targets_default"),
                            class = "text-success",
                            "Workflow folder detected, loaded the default targets file for you."
                        ),
                        HTML(
                        '<li>If you would like to change the targets table content, please use
                        step <span class="text-info">2. Prepare the targets file</span>.
                        Remember to click on <b>Add to task</b> to submit the change.
                        </li>'
                        )
                    ),
                    column(
                        3,
                        box(
                            closable = FALSE, width = 12,
                            radioGroupButtons(
                                inputId = ns("source_targets"),
                                label = "Choose your targets file source:",
                                selected = "eg",
                                choiceNames = c("Upload", "Default/Example"),
                                choiceValues = c("upload", "eg"),
                                justified = TRUE, status = "primary",
                                checkIcon = list(
                                    yes = icon("ok", lib = "glyphicon"),
                                    no = "")
                            ),
                            dynamicFile(id = ns("targets_upload")),
                            selectizeInput(
                                inputId = ns("targets_delim"),
                                label = "File delimiter",
                                choices = c(Tab="\t", `,`=",", space=" ",
                                            `|`="|", `:`=":", `;`=";"),
                                options = list(style = "btn-primary")
                            ),
                            clearableTextInput(
                                ns("targets_comment"), "File comments", value = "#")
                        )
                    ),
                    box(
                        closable = FALSE, width = 9,
                        DT::DTOutput(ns("targets"))
                    )
                )
            ) %>%
            bsplus::bs_append(
                "CWL running file",
                fluidRow(
                    column(
                        3,
                        box(
                            closable = FALSE, width = 12,
                            radioGroupButtons(
                                inputId = ns("source_cwl"),
                                label = "Choose your CWL file source:",
                                selected = "eg",
                                choiceNames = c("Upload", "Example"),
                                choiceValues = c("upload", "eg"),
                                justified = TRUE, status = "primary",
                                checkIcon = list(
                                    yes = icon("ok", lib = "glyphicon"),
                                    no = "")
                            ),
                            dynamicFile(id = ns("cwl_upload"))
                        )
                    ),
                    shinyAce::aceEditor(
                        outputId = ns("ace_cwl"),
                        theme = "Chrome",
                        value = "",
                        placeholder = "yaml format",
                        mode = "yaml"
                    )
                )
            ) %>%
            bsplus::bs_append(
                "CWL input file",
                fluidRow(
                    column(
                        3,
                        box(
                            closable = FALSE, width = 12,
                            radioGroupButtons(
                                inputId = ns("source_cwl_input"),
                                label = "Choose your CWL input file source:",
                                selected = "eg",
                                choiceNames = c("Upload", "Example"),
                                choiceValues = c("upload", "eg"),
                                justified = TRUE, status = "primary",
                                checkIcon = list(
                                    yes = icon("ok", lib = "glyphicon"),
                                    no = "")
                            ),
                            dynamicFile(id = ns("cwl_input_upload"))
                        )
                    ),
                    shinyAce::aceEditor(
                        outputId = ns("ace_cwl_input"),
                        theme = "Chrome",
                        value = "",
                        placeholder = "yaml format",
                        mode = "yaml"
                    )
                )
            )
    )
}

## server
#' @importFrom shinyAce updateAceEditor
#' @importFrom shinyWidgets sendSweetAlert
wf_cwlServer <- function(id, shared){
    module <- function(input, output, session){
        ns <- session$ns
        # setup
        temp_folder <- tempdir()
        ## targets file
        observe({
            shinyjs::toggleElement(
                "targets_eg",  anim = TRUE, animType = "fade",
                condition = !as.logical(shared$wf$flags$env_ready)
            )
            shinyjs::toggleElement(
                "targets_default",  anim = TRUE, animType = "fade",
                condition = shared$wf$flags$env_ready
            )
        })
        observeEvent(input$source_targets, {
            shinyjs::toggleElement(id = "targets_upload", anim = TRUE, condition = input$source_targets =="upload")
        })
        # path resolve
        targets_path <- reactiveVal(NULL)
        observeEvent(c(shared$wf$targets_path, shared$wf$flags$targets_ready,
                       input$source_targets), {
            req(input$source_targets)
            if(emptyIsFalse(shared$wf$targets_path)){
                targets_path(shared$wf$targets_path)
            } else {
                targets_path(file.path("data", "targetsPE.txt"))
            }
            if(input$source_targets == "upload") targets_path(targets_upload_path()$datapath)
        })
        targets_upload_path <- dynamicFileServer(input, session, id = "targets_upload")
        # load
        data_targets <- reactive({
            loadDF(
                choice = input$source_targets,
                upload_path = targets_upload_path()$datapath,
                delim = input$targets_delim,
                comment = input$targets_comment,
                eg_path = targets_path()
            )
        })
        output$targets <- DT::renderDT({
            data_targets <- data_targets()
            targets_columns(names(data_targets)) # get col names for parsing
            DT::datatable(
                data_targets,
                style = "bootstrap",
                class = "compact"
            )
        })
        ## cwl file
        observeEvent(input$source_cwl, {
            shinyjs::toggleElement(id = "cwl_upload", anim = TRUE, condition = input$source_cwl =="upload")
        })
        data_cwl <- reactiveValues()
        upload_path_cwl <- dynamicFileServer(input, session, id = "cwl_upload")
        observeEvent(c(upload_path_cwl(), input$source_cwl), {
            req(input$source_cwl)
            if(input$source_cwl == "eg"){
                data_cwl$path <- file.path("data", "hisat2.cwl")
            } else {
                data_cwl$path <- upload_path_cwl()$datapath
            }
        })
        observeEvent(data_cwl$path, {
            shinyAce::updateAceEditor(
                session, editorId = "ace_cwl",
                value = {
                    shinyCatch(
                        readLines(data_cwl$path) %>%
                            paste(collapse = "\n"), blocking_level = "error")
            })
        })
        ## cwl input file
        cwl_input_path <-
            observeEvent(input$source_cwl_input, {
                shinyjs::toggleElement(id = "cwl_input_upload", anim = TRUE, condition = input$source_cwl_input =="upload")
            })
        data_cwl_input <- reactiveValues()
        upload_path_cwl_input <- dynamicFileServer(input, session, id = "cwl_input_upload")
        observeEvent(c(upload_path_cwl_input(), input$source_cwl_input), {
            req(input$source_cwl_input)
            if(input$source_cwl_input == "eg"){
                data_cwl_input$path <- file.path("data", "hisat2.yml")
            } else {
                data_cwl_input$path <- upload_path_cwl_input()$datapath
            }
        })
        observeEvent(data_cwl_input$path, {
            shinyAce::updateAceEditor(
                session, editorId = "ace_cwl_input",
                value = {
                    shinyCatch(
                        readLines(data_cwl_input$path) %>%
                            paste(collapse = "\n"), blocking_level = "error")
                })
        })
        # get real time updates on cwl input file
        cwl_input_vars <- reactive({
            input$ace_cwl_input %>%
                str_split("\n") %>%
                unlist() %>%
                {.[str_detect(., ".*:")]} %>%
                {.[str_detect(., ":\\s{0,}_[a-zA-Z0-9_]+_\\s{0,}$")]} %>%
                str_extract("_[a-zA-Z0-9_]+_")
        }) %>% debounce(1000)

        # setup for parsing
        targets_columns <- reactiveVal("")
        observeEvent(c(cwl_input_vars(), targets_columns()), {
            req(emptyIsFalse(targets_columns()))
            removeUI(
                selector = glue('#{ns("replace-box")} div'),
                multiple = TRUE
            )
            for(i in seq_along(cwl_input_vars())){
                insertUI(
                    glue('#{ns("replace-box")}'),
                    where = "beforeEnd",
                    ui =  column(3, selectizeInput(
                        inputId = ns(paste0("replace-", i)),
                        label = cwl_input_vars()[i],
                        choices = c("Not required", targets_columns()),
                        options = list(style = "btn-primary")
                    ))
                )
            }
        })
        ## parsing
        observeEvent(input$parse, {
            cwl_file <- file.path(temp_folder, "wf.cwl")
            cwl_input <- file.path(temp_folder, "input.yml")
            targets <- targets_path()
            writeLines(input$ace_cwl, cwl_file)
            writeLines(input$ace_cwl_input, cwl_input)

            args <- shinyCatch({
                # parse replacement
                replace_cols <- lapply(seq_along(cwl_input_vars()), function(i){
                    input[[paste0("replace-", i)]]
                }) %>% unlist()
                not_required <- replace_cols == "Not required"
                inputvars <- cwl_input_vars()
                names(inputvars) <- replace_cols
                inputvars <- inputvars[!not_required]
                if(length(names(inputvars)) != length(unique(names(inputvars))))
                    stop("Each target column can only be used once for inputvars")
                inputvars <- if(emptyIsFalse(inputvars[1])) inputvars else NULL
                # parsing
                args <- systemPipeR::loadWorkflow(targets = targets, wf_file = "wf.cwl",
                                                  input_file = "input.yml", dir_path = temp_folder)
                systemPipeR::renderWF(args, inputvars = inputvars)
            }, blocking_level = "error")
            output$parse_out <- renderPrint({
                systemPipeR::cmdlist(args) %>% unlist() %>% unname()
            })
        })
    }
    moduleServer(id, module)
}