PRG: PRG - generate a Precision Recall Gain object to show...

In Simon-Coetzee/StatePaintR: StatePaintR: a tool for creating and documenting reproducible chromatin state annotations from StateHub

Description

PRG - generate a Precision Recall Gain object to show accuracy of states

Usage

PRG(states, comparison, state.select = NULL, comparison.select = NULL)

Arguments

states	A GRangesList produced by PaintStates or a list of GRanges, or GRangesList that has a "score" column, and optionally a "state" column, that is filtered on if the state.select argument is used
comparison	A GRanges object that indicates TRUE or FALSE for a specific genomic intervals. The names of the the states object must be reflected in the columns of the comparison object, Where each interval is indicated as being validated with 1 or negative with 0
state.select	A character vector that will filter the states object if the states object has a "state" column. For example if the output of PaintStates has segments with 'EAR' and 'PAR', and one wants to select only 'EAR', then state.select = c("EAR") would only compare the 'EAR' scores in states to the comparison set.
comparison.select	a list of numeric vectors that can be used to select a subset of comparison for each element in the list of states

Details

This code incorprates Precision-Recall-Gain analysis from https://github.com/meeliskull/prg/tree/master/R_package.
From the abstract of the referenced paper: "Precision-Recall analysis abounds in applications of binary classification where true negatives do not add value and hence should not affect assessment of the classifier's performance. Perhaps inspired by the many advantages of receiver operating characteristic (ROC) curves and the area under such curves for accuracy-based performance assessment, many researchers have taken to report Precision-Recall (PR) curves and associated areas as performance metric. We demonstrate in this paper that this practice is fraught with difficulties, mainly because of incoherent scale assumptions – e.g., the area under a PR curve takes the arithmetic mean of precision values whereas the F_beta score applies the harmonic mean. We show how to fix this by plotting PR curves in a different coordinate system, and demonstrate that the new Precision-Recall-Gain curves inherit all key advantages of ROC curves. In particular, the area under Precision-Recall-Gain curves conveys an expected F1 score on a harmonic scale, and the convex hull of a Precision-Recall-Gain curve allows us to calibrate the classifier's scores so as to determine, for each operating point on the convex hull, the interval of beta values for which the point optimises F_beta. We demonstrate experimentally that the area under traditional PR curves can easily favour models with lower expected F1 score than others, and so the use of Precision-Recall-Gain curves will result in better model selection."

Value

a list that contains the precision gain and recall gain, the convex hull, and the area under the precision recall gain curve (auprg).

References

Flach, P., & Kull, M. (2015). Precision-Recall-Gain Curves: PR Analysis Done Right. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 28 (pp. 838-846). Curran Associates, Inc.

See Also

http://www.cs.bris.ac.uk/~flach/PRGcurves/

Examples

load(system.file("extdata", "heart.states.rda", package = "StatePaintR"))
load(system.file("extdata", "vista.heart.enhancers.rda", package = "StatePaintR"))
PRG(states = heart.states,
   comparison = heart.enhancers,
   state.select = c("EAR", "EARC", "AR", "ARC"))

Simon-Coetzee/StatePaintR documentation built on May 9, 2019, 1:31 p.m.