The LongTable Class

In CoreGx: Classes and Functions to Serve as the Basis for Other 'Gx' Packages

library(formatR)
knitr::opts_chunk$set(echo = FALSE)

library(CoreGx)
library(data.table)

Why Do We Need A New Class?

The current implementation for the @sensitivity slot in a PharmacoSet has some limitations.

Firstly, it does not natively support dose-response experiments with multiple drugs and/or cancer cell lines. As a result we have not been able to include this data into a PharmacoSet thus far.

Secondly, drug combination data has the potential to scale to high dimensionality. As a result we need an object that is highly performant to ensure computations on such data can be completed in a timely manner.

Design Philosophy

The current use case is supporting drug and cell-line combinations in PharmacoGx, but we wanted to create something flexible enough to fit other use cases. As such, the current class makes no mention of drugs or cell-lines, nor anything specifically related to Bioinformatics or Computation Biology. Rather, we tried to design a general purpose data structure which could support high dimensional data for any use case.

Our design takes the best aspects of the SummarizedExperiment and MultiAssayExperiment class and implements them using the data.table package, which provides an R API to a rich set of tools for high performance data processing implemented in C.

Anatomy of a LongTable

Class Diagram

knitr::include_graphics('LongTableClassDiagram.png')

We have borrowed directly from the SummarizedExperiment class for the rowData, colData, metadata and assays slot names. We also implemented the SummarizedExperiment accessor generics for the LongTable.

Object Structure and Cardinality

There are, however, some important differences which make this object more flexible when dealing with high dimensional data.

knitr::include_graphics('LongTableStructure.png')

Unlike a SummarizedExperiment, there are three distinct classes of columns in rowData and colData.

The first is the rowKey or colKey, these are implemented internally to keep mappings between each assay and the associated samples or drugs; these will not be returned by the accessors by default. The second is the rowIDs and colIDs, these hold all of the information necessary to uniquely identify a row or column and are used to generate the rowKey and colKey. Finally, there are the rowMeta and colMeta columns, which store any additional data about samples or drugs not required to uniquely identify a row in either table.

Within the assays the rowKey and colKey are combined to form a primary key for each assay row. This is required because each assay is stored in 'long' format, instead of wide format as in the assay matrices within a SummarizedExperiment. Thanks to the fast implementation of binary search within the data.table package, assay tables can scale up to tens or even hundreds of millions of rows while still being relatively performant.

Also worth noting is the cardinality between rowData and colData for a given assay within the assays list. As indicated by the lower connection between these tables and an assay, for each row or column key there may be zero or more rows in the assay table. Conversely for each row in the assay there may be zero or one key in colData or rowData. When combined, the rowKey and colKey for a given row in an assay become a composite key which maps that to

Building a LongTable

he current implementation of the buildLongTable function is able to assemble a LongTable object from two sources. The first is a single large table with all assays, row and column data contained within it. This is the structure of the Merck drug combination data that has been used to test the data structure thus far.

filePath <- '../data/merckLongTable.csv'
merckDT <- fread(filePath, na.strings=c('NULL'))
colnames(merckDT)

knitr::kable(head(merckDT)[, 1:5])

knitr::kable(head(merckDT)[, 5:ncol(merckDT)])

We can see that all the data related to the treatment response experiment is contained within this table.

Single Assays Table

To build a LongTable object from this file:

rowDataCols <- list(
    c(cell_line1="cell_line", BatchID="BatchID"))
colDataCols <- list(
    c(drug1='drugA_name', drug2='drugB_name',
     drug1dose='drugA Conc (uM)', drug2dose='drugB Conc (uM)'),
    c(comboName='combination_name'))
assayCols <- list(viability=paste0('viability', seq_len(4)),
                  viability_summary=c('mu/muMax', 'X/X0'))
longTable <- buildLongTable(from=filePath, rowDataCols,
                            colDataCols, assayCols)

longTable

This function will also work if directly passed a data.table or data.frame object:

longTable1 <- buildLongTable(from=merckDT, rowDataCols, colDataCols, assayCols)

paste0('All equal? ', all.equal(longTable, longTable1))

Multiple Assay Tables

The second option for building a LongTable is to pass it a list of different assays with a shared set of row and column identifiers. We haven't had a chance to testing this functionality with real data yet, but do have a toy example.

assayList <- assays(longTable, withDimnames=TRUE, metadata=TRUE, key=FALSE)
assayList$new_viability <- assayList$viability  # Add a fake additional assay
assayCols$new_viability <-  assayCols$viability  # Add column names for fake assay
longTable2 <- buildLongTable(from=assayList, lapply(rowDataCols, names), lapply(colDataCols, names), assayCols)

longTable2

We can see that a new assay has been added to the LongTable object when passed a list of assay tables containing the required row and column IDs. Additionally, any row or column IDs not already in rowData or colData will be appended to these slots automatically!

LongTable Object

As mentioned previously, a LongTable has both list and table like behaviours. For table like operations, a given LongTable can be thought of as a rowKey by colKey rectangular object.

To support data.frame like sub-setting for this object, the constructor makes pseudo row and column names, which are the ID columns for each row of rowData or colData pasted together with a ':'.

Row and Column Names

head(rownames(longTable))

We see that the rownames for the Merck LongTable are the cell line name pasted to the batch id.

head(colnames(longTable))

For the column names, a similar pattern is followed by combining the colID columns in the form 'drug1:drug2:drug1dose:drug2dose'.

data.frame Subsetting

We can subset a LongTable using the same row and column name syntax as with a data.frame or matrix.

row <- rownames(longTable)[1]
columns <- colnames(longTable)[1:2]
longTable[row, columns]

Regex Queries

However, unlike a data.frame or matrix this subsetting also accepts partial row and column names as well as regex queries.

head(rowData(longTable), 3)
head(colData(longTable), 3)

For example, if we want to get all instance where '5-FU' is the drug:

longTable[, '5-FU']

This has matched all colnames where 5-FU was in either drug1 or drug2. If we only want to match drug1, we have several options:

all.equal(longTable[, '5-FU:*:*:*'], longTable[, '^5-FU'])

data.table Subsetting

In addition to regex queries, a LongTable object supports arbitrarily complex subset queries using the data.table API. To access this API, you will need to use the . function, which allows you to pass raw R expressions to be evaluated inside the i and j arguments for dataTable[i, j].

For example if I want to subset to rows where the cell line is VCAP and columns where drug1 is Temozolomide and drug2 is either Lapatinib or Bortezomib:

longTable[.(cell_line1 == 'CAOV3'),  # row query
          .(drug1 == 'Temozolomide' & drug2 %in% c('Lapatinib', 'Bortezomib'))]  # column query

We can also invert matches or subset on other columns in rowData or colData:

subLongTable <-
  longTable[.(BatchID != 2),
            .(drug1 == 'Temozolomide' & drug2 != 'Lapatinib')]

To show that it works as expected:

print(paste0('BatchID: ', paste0(unique(rowData(subLongTable)$BatchID), collapse=', ')))
print(paste0('drug2: ', paste0(unique(colData(subLongTable)$drug2), collapse=', ')))

Accessor Methods

rowData

head(rowData(longTable), 3)

head(rowData(longTable, key=TRUE), 3)

colData

head(colData(longTable), 3)

head(colData(longTable, key=TRUE), 3)

assays

assays <- assays(longTable)
assays[[1]]

assays[[2]]

assays <- assays(longTable, withDimnames=TRUE)
colnames(assays[[1]])

assays <- assays(longTable, withDimnames=TRUE, metadata=TRUE)
colnames(assays[[2]])

assayNames(longTable)

Using these names we can access specific assays within a LongTable.

assay

colnames(assay(longTable, 'viability'))
assay(longTable, 'viability')

colnames(assay(longTable, 'viability', withDimnames=TRUE))
assay(longTable, 'viability', withDimnames=TRUE)

CoreGx documentation built on Nov. 8, 2020, 4:50 p.m.