New format_points() reconstructs per-instance scores
and labels from a table of performance values calculated at a set of
thresholds - true and false positive rates, or recall and precision - so
that a summary table can reach evalmod() and
mmdata() after all. The rows between two thresholds become
instances that share a score, and the even spread precrec
already applies to a tied run is the non-linear interpolation of Davis
and Goadrich (2006), so nothing new was needed in the curve code and a
complete table round-trips exactly. np and nn
are required: rates do not carry the class totals, and the totals also
place the chance-level baseline. The interpolation is exact only across
a gap one instance wide, so an area read off a coarse table remains an
estimate - format_points() says so when it runs, and
Prepare your data says so at more length. Requested in #6 and
#15, and tracked in #32.
Prepare your data replaces its section on what cannot be used as input with one on what a table of summary points can and cannot give you.
Prepare your data gains a section on which direction the
score runs. precrec ranks the highest score first, and a
quantity that runs the other way - a p-value, a distance, an error term
- is handled by negating it. There is no argument for this and there was
nothing written down either, so the question had no answer anywhere on
the site. The section also notes the one place negation does not carry:
prob_metrics() reads the values of the scores rather than
their order, so it needs a genuine probability. Reported in
#14.
Prepare your data gains a section on what cannot be used
as input. precrec needs the per-instance scores and labels;
precision and recall already calculated at a set of thresholds, or a
table of true and false positive rates, cannot be used. The counts such
a table implies are recoverable when the totals are known, so the limit
is not arithmetic: it is that the precision-recall interpolation is only
correct across a gap one instance wide, and over the coarse gaps a
threshold table leaves it reintroduces the very error
precrec exists to remove. The limit was deliberate but
undocumented, and two people arrived with such a table four years apart.
Reported in #6 and #15.
Correct the pROC argument table in Coming from
pROC or ROCR, which paired
direction = with posclass =. The two do
different things: posclass names the positive label, which
is pROC’s levels, while direction
is the negation above.
metric_table() reported a precision on the row that
calls nothing positive and an NPV on the row that calls everything
positive. Neither has a denominator - precision is
TP / (TP + FP) and NPV is TN / (TN + FN) - and
both were filled in from the neighboring row, so they followed the
top-ranked or bottom-ranked instance, which is an instance the rule does
not predict. A threshold above every score reported
precision = 1 beside sensitivity = 0, and
0 instead if the top-ranked instance happened to be a
negative.
Both are now NA, with the metrics derived from them: the
false discovery rate, the false omission rate, and
markedness at both ends. mcc,
lift, odds, chisq and the
positive likelihood ratio already reported NA on the same
rows.
This matters most for metric_table(at = ), whose purpose
is scoring a threshold chosen somewhere else: a threshold transferred
between datasets lands outside the score range routinely.
The long form as.data.frame(evalmod(mode = "basic"))
returns is unchanged, and so are the curves. The inherited precision is
what anchors the precision-recall curve at recall 0, and it
is the right value there
auc(), average_precision() and
prbe() are unaffected, which the tests now assert.best_cutoff() could return the rank 0
row of metric_table(), the rule that calls nothing
positive. There is no threshold that predicts nothing, so that row’s
score is NA and it is not an operating point,
which is what the function exists to name. It is no longer a
candidate.
It won outright wherever the empty rule is optimal -
specificity always, and accuracy,
error and an unweighted cost once positives
are rare, which at 2% positives was 11 runs in 20 - and it won on the
tie-break wherever it merely tied. That last case was not an imbalance
problem at all: precision at rank 0 is the limit from
above, so it is 1 whenever the top-ranked instance is a
positive, and any classifier good enough to rank one first came back
with no threshold.
Choose an operating point gains the
accuracy trap beside the sensitivity and
specificity ones it already covered. On twenty positives in
four hundred, predicting nothing is 0.95 accurate and the
accuracy-optimal cutoff reaches 0.955 by finding two of the twenty;
mcc finds eleven.
prbe() reported a break-even point of 0
whenever the top-ranked instance was a negative. A precision-recall
curve is anchored at recall 0, where precision is
0 in that case, so precision and recall were trivially
equal at the origin and the crossing search counted it - though nothing
has been retrieved there. At a low proportion of positives the
top-ranked instance is nearly always a negative: of 150 random rankings
of a dataset with 20 positives in 1000, 148 had a negative on top and
all 148 got the spurious row. It sat in front of the real break-even
point, so taking the first row of the result returned 0
instead of the answer.
A curve that leaves the origin below the diagonal and never catches
up now gets the single NA row the help page has always
promised for a curve that never reaches equal precision and
recall.
prbe() gained a baselines column, the
proportion of positives. At chance the curve is flat at the prevalence,
so it meets the diagonal at that recall - a break-even point of
0.2 is chance on data that is 20% positive and five times
chance on data that is 4% positive.
Every table that reports an area now reports what that area is
worth by chance beside it. pauc() gained
baselines and sbaselines, one for each of the
two scales it standardizes on, and auc_ci(),
auc_boot() and auc_delong() gained
baselines. A partial area is where the chance level is
easiest to get wrong: over false positive rates [x1, x2] a
coin flip covers (x2^2 - x1^2) / 2, so a
standardized partial ROC area up to 0.2 has a
chance level of 0.1 and not the 0.5 a
standardized number invites. Both are NA for a
ylim other than c(0, 1), which is the case
cpaucs already declines.
Balanced and imbalanced data gains a section on the fact
that a baseline is an asymptote. The prevalence is what a
precision-recall area is worth by chance in the limit; an area measured
on a finite sample scatters around it, and with twenty positives a
classifier with no signal averages 1.15 times its baseline over the
whole curve and 1.86 times over the first tenth of recall, where it
clears twice the baseline about one run in five. The median sits below
chance, so the error is a skewed distribution rather than a shifted one.
Dividing an area by its baseline is not a test; auc_boot()
is, and it is well calibrated on the same case. ?auc and
?pauc say so too.
pauc() gained a corrected argument,
which adds a cpaucs column: the McClish correction of the
ROC partial area, which rescales the span between chance and perfect
onto 0.5 to 1 so that a partial area reads on
the same scale as a full one. It is what
pROC::auc(..., partial.auc.correct = TRUE) reports, to the
last digit.
Both this and the spaucs the package has always reported
are called “the standardized partial AUC”, and they are not the same
number: spaucs is the area over the area of the region, the
fraction of what was available that the curve covered. A value moved
between two tools looked like a disagreement when it was a choice of
convention, and only one of the two conventions could be had here.
The chance area is taken over the region actually asked for. The
formula usually quoted, and the one Coming from pROC or ROCR
published for readers to apply by hand, is the special case of a region
starting at a false positive rate of 0; it is wrong for a
region such as xlim = c(0.1, 0.3), which is exactly where
someone would reach for it.
cpaucs is NA on the precision-recall rows
and whenever part() was given a ylim, both
documented in ?pauc. Off by default, so nothing that reads
pauc() today changes.
The help page of pauc() opened by naming
auc() as the function it documents, a line copied from
auc() and never corrected.
Balanced and imbalanced data gains a section on why the baseline is reported beside the area rather than folded into it. Precision-Recall-Gain (Flach & Kull 2015) is the published way of folding it in, and on a classifier held fixed while only the prevalence moves, its area climbs from 0.639 to 0.941 - three tenths of its range, in the opposite direction to the precision-recall area, which falls. Normalizing by the baseline does not remove the dependence on prevalence, it reverses it.
auc() and average_precision() gained a
baselines column, and the summary print()
shows a Baseline beside each AUC. It is what the area would
be by chance: 0.5 on every ROC row, and the proportion of
positives on every precision-recall row.
A ROC AUC can be read on its own and a PRC AUC cannot, which is the package’s own argument and was the one number the package did not report. On one generator at three class balances the ROC AUC holds between 0.81 and 0.85 while the PRC AUC goes 0.801, 0.367, 0.129 - and the last of those, which reads as failure, is six times its baseline of 0.02. The plots have drawn the baseline all along; now the areas carry it too.
The baseline is taken per model and per test dataset, so a fold that
does not hold the classes in the proportions the whole dataset does is
read against its own. On a macro-average row it is averaged
over the classes with the same weights the AUCs were, uniform or by
prevalence.
Code that selects columns of these data frames by name or by
subset() is unaffected; code that relies on their column
count or position will see one more.
New metric_table(at = ) reports the metrics of
thresholds you name, rather than of every cutoff. A threshold is rarely
one of the observed scores, so it has no row of its own, but it always
names one of the cutoffs: score >= it calls a certain
number of instances positive, and that count is a rank the table already
holds. The row returned is that rank’s row, with at
carrying the threshold asked for and score the observed
cutoff realizing it. A threshold above every score gives the
rank = 0 row, and an NA score is never a
positive prediction, as everywhere else in the package.
This is what was missing to score a cutoff on data it was not chosen
on. classification_report(at = ) already took an arbitrary
threshold but reports only precision, recall and F-score;
metric_table() had every metric but only at the cutoffs the
data happens to contain. Choose an operating point now shows
the two ends together - best_cutoff() on nine folds,
metric_table(at = ) on the tenth - and what it costs not
to: on data with five percent positives, mcc reads 0.361
when each fold picks its own cutoff and 0.228 when the cutoff is chosen
without seeing it, held-out below in-sample in every fold.
best_cutoff() rejects at rather than
letting it narrow the search it does.
Document print(). The six print()
methods carried no help page at all: one of them pointed at a
print topic with @rdname, nothing defined that
topic, and so roxygen2 skipped it on every run and
?print.mdat failed. R CMD check did not notice
because the methods are registered rather than exported by name. The
topic now exists, and says which object each method handles and what it
shows.
print() returns its argument invisibly, as an S3
print method is expected to. Five of the six returned the
value of the last cat() call, so
x <- print(curves) gave NULL. What the
console shows is unchanged.
Regroup the website. A Compare and decide section now collects the four pages that answer “which model do I ship, and at what threshold” - comparing several models, uncertainty from one test set, choosing an operating point, and the classification report - which were previously split between How-to and Metrics on the grounds of what they returned rather than what they were for. Metrics is left as what it already effectively was, a catalog of what each number means, and Plots is unchanged. No page moved and no URL changed; the grouping is in the menus only.
Add a website page, Choose an operating point, which
best_cutoff() did not have: it was documented inside
Get the numbers out, a page about extracting data, and argued
about inside Balanced and imbalanced data, a page about reading
curves. The new page covers the criteria and the two groups they fall
into, weighting the two mistakes with cost_fp and
cost_fn - including why equal costs are a strong claim
rather than a neutral one when positives are rare - tied scores, the one
row per model per test dataset and the spread across them, and the fact
that a cutoff chosen on the data it is scored on is optimistic and that
nothing here corrects for that.
Add a website page, Coming from pROC or
ROCR, translating the two packages call by call now
that metric_table() and best_cutoff() make the
table honest. Every equivalence on it was checked against both packages,
including the three places the answers differ: the precision-recall
area, which ROCR reads off straight lines between raw
points; tied scores, which the other two collapse and
evalmod(basic_ties = "hold") matches; the threshold itself,
which pROC names by a midpoint and precrec by
an observed score; and the standardized partial AUC, where
spaucs is the fraction of the region the curve covered and
pROC rescales chance to 0.5 instead - with the
conversion between them. What has no equivalent is listed rather than
left to be discovered.
New best_cutoff() picks the cutoff that optimizes
one metric, in the manner of pROC::coords(x, "best"). It
returns a row of metric_table() together with the criterion
that chose it, so every other metric is there to be read at the same
cutoff, and one row per model per test dataset.
metric takes any of the metrics evalmod()
calculates, plus "youden" and "topleft" for
the two criteria the cutpoint literature names rather than the metric
table does. The direction is a property of the metric and so is not an
argument, and a tie is broken toward the cutoff that calls the fewest
instances positive - except between rows that share a
score, which are one threshold seen several times rather
than several cutoffs, and report the rank that threshold actually
produces.
The criteria are documented in two groups rather than as six
interchangeable options. Youden’s J and the closest point to the top
left corner are computed from sensitivity and specificity alone, both of
which are conditioned on the true class, so neither knows the prevalence
and on imbalanced data both can choose a cutoff at a precision no one
would deploy. fscore, mcc and a
cost weighted by cost_fp and
cost_fn are computed from precision as well. The default is
"youden" because that is what a caller arriving from
another package expects, and the reference page says in the same breath
when not to use it.
youden is also accepted as an alias for
informedness wherever evalmod(metrics = )
takes metric names.
New auc_delong() calculates the ROC AUC and its
variance analytically, by DeLong’s method, rather than by resampling.
auc_ci() reads a normal interval off it and
auc_diff() compares models through its covariance matrix,
so it is the exact counterpart of auc_boot(): same input,
same two functions reading the result, no boot_n, no seed,
nothing that moves between two runs, and no floor under the p-value.
The ROC AUC is a Mann-Whitney U statistic, so its variance follows
from the structural components of that statistic. Those components are
read off midranks rather than off all m * n comparisons,
and off the ranks precrec has already assigned, so ties and
NA scores are handled exactly as they are everywhere else
and the AUC reported is the one auc() reports.
There is no precision-recall counterpart, and the result carries ROC
rows only. The precision-recall AUC is not a U statistic and the
interpolated area is further from being one still, so
auc_boot() remains the answer there - and remains the safer
reading on a small or badly imbalanced test set, since DeLong’s variance
is an asymptotic one.
auc_diff() is now an S3 generic, dispatching on
whichever of the two objects it is given. Existing calls are
unaffected.
New metric_table() returns every basic evaluation
metric at every cutoff, one row per cutoff and one column per metric -
what pROC::coords() and the cutoff slots of a
ROCR::performance object give, and the shape most work
downstream of a curve needs: pick an operating point, hand a threshold
to production code, tabulate a report, compute a metric
precrec does not.
Nothing new is calculated. evalmod(mode = "basic") has
always kept the score and the label at every cutoff, and
as.data.frame() has always returned them - as two of the
fourteen values of its type column, alongside the metrics
rather than indexing them. The long form is right for plotting, which is
what it was built for, and wrong for the question “what happens if I cut
here”. metric_table() is that same data on its side,
through the same converter, so the two cannot drift apart.
It takes the doors evalmod() takes - an
mmdata() object, or scores and
labels - and also a mode = "basic" object that
has already been calculated, since reusing one is the point.
rank counts the instances called positive and
normalized_rank is that divided by the number of instances,
the x axis of the basic metric plots. Rows are per test dataset and
nothing is averaged across them, because a cutoff belongs to the dataset
it was read off.
The two columns auc_diff() gained in 0.22.1 are
renamed. statistic is now z_values and
p_values_norm is now p_values_wald, and the
documentation calls what they come from a Wald test throughout. Neither
column’s contents changed.
p_values_norm named the p-value for the distribution it
is read off rather than for the test that produced it, which left it
inconsistent with p_values beside it - named for the
percentile method - and readable as “normalized”, which it is not.
statistic said nothing about which distribution it is
referred to, and was singular where every other column in the row is
plural.
The reference page and the Uncertainty from one test set
article now give both formulas outright,
z_values = diffs / sd(d) and
p_values_wald = 2 * pnorm(-abs(z_values)) against the
percentile count, and answer under its own heading why two p-values are
reported at all: they fail in opposite ways - the percentile one has a
floor at 2 / (n + 1), the Wald one has an assumption of
normality - and it is whether they agree that tells you which is
failing.
0.22.1 is a day old and was not submitted to CRAN, so the renames are made now rather than carried.
auc_diff() reports a test statistic and a second
p-value, and takes a new alternative argument for the tail
they are read from. statistic divides the observed
difference in AUC by the standard deviation of the resampled
differences, so the bootstrap spread stands in for a standard error, and
p_values_norm reads that statistic off the normal
distribution.
The percentile p_values is floored at
2 / (n + 1) on purpose, which is 0.002 at the default
boot_n = 1000; a difference that is not in any doubt
bottoms out there, and reporting p < 0.001 from it needs
ten thousand resamples or more. The normal approximation reads an
unbounded p-value off the resamples that are already there.
It buys that resolution with an assumption. Nothing is shuffled
between the models, so the null is never enforced, and the resampled
differences are taken to be roughly normal - weakest where they are
skewed, which is few positives or either model near the ceiling of the
precision-recall AUC. p_values keeps its meaning, both are
reported side by side, and where the two disagree sharply it is the
normal one to distrust.
Requested in #21, which came out of #13.
evalmod(mode = "basic") takes a new
basic_ties argument for what the basic metrics report at
the cutoffs inside a run of tied scores. Such a cutoff separates
instances that share a score, so no threshold produces it. The default
"split" spreads the true and false positives of the run
evenly over its cutoffs, which is what precrec has always
done and is the interpolation the ROC and precision-recall curves need.
"hold" gives every cutoff in the run the counts it has once
the whole run is taken, so tied instances share one value of every
metric and each metric becomes a step function that changes only where
the score does.
A classifier that is exactly right but scores in a single bit
reported sensitivity climbing across the positives rather than reaching
1 at once, which is the even spread and not the classifier.
basic_ties = "hold" is the reading that case wants.
The two settings agree whenever the scores are all distinct,
"split" is the default because it is what every published
result was computed with, and the curves are not affected either
way.
reduce_points now works for
mode = "basic". The basic metrics carry one point per
cutoff, so plotting a large dataset drew a mark per instance: at a
million rows autoplot() spent 6.7 seconds against 0.5 for
the calculation behind it. fortify(),
as.data.frame() and autoplot() accept
reduce_points = TRUE for these objects now and keep
x_bins points per metric, which brings that plot to 1.9
seconds.
The default stays FALSE, so no existing output moves
unless the reduction is asked for, and the points kept are the
calculated ones - the reduction selects among them and never
interpolates. Nothing is dropped from an object that already holds
x_bins points or fewer.
The argument was accepted and silently discarded before, so a call that passed it was not an error; it simply had no effect.
evalmod(mode = "basic") records x_bins
on the object it returns. It places no supporting points there - the
basic metrics have no interpolation to place them on - but it is the
number of points kept when reduce_points thins them, so the
resolution of the drawn output is set the same way for both
modes.
Speed up the C++ hot paths. evalmod() runs about
1.5x faster on a million rows, and evalmod(mode = "basic")
about 1.2x. The results are unchanged, bit for bit.
Two things were costing the time. First, reading a vector through
Rcpp’s operator[] goes through a proxy whose bounds check
is a call to warning(), and in a translation unit the size
of precrec_plx.cpp the compiler leaves that call out of
line rather than inlining it away - so every element access in the
innermost loops carried a call, a compare and a barrier to optimizing
the loop around it. The four functions that scan whole vectors take a
raw pointer once instead, which is worth 1.8x in
calc_basic_metrics(), 1.7x in
create_prc_curve(), 1.5x in create_roc_curve()
and 1.3x in create_confusion_matrices().
Second, ranking the scores was 84% of get_score_ranks()
and all of it was the sort. A comparison sort over (index, score) pairs
is replaced by a radix sort on an order-preserving key, which is 2.3x
faster on distinct scores and 2.4x on heavily tied ones. Scores that
arrive already sorted are the one shape it loses on, and a monotonicity
scan in front of it hands those to a direct fill rather than to the
passes; the scan stops as soon as the scores are neither ascending nor
descending, which on unsorted input is within the first few elements.
Even so, ranking an already sorted vector is about 0.8x - the comparison
sort was unusually good at that one case, and the scan recovers most of
the difference but not all of it.
Ties still rank the way they did. The old comparator broke ties on the input index to keep the permutation from depending on the standard library’s choice of introsort; a radix sort is stable pass by pass, so it produces that same permutation from the algorithm instead.
Fix a buffer overrun in create_roc() and
create_prc(). Both built the interpolated points from a
grid snapped to each gap’s own start, so neighboring gaps disagreed
about where the grid lines fell and together emitted more points than
the x_bins the output buffers are sized for. The write ran
past the end of the vectors, which aborted the session from
x_bins = 5000 up -
evalmod(..., x_bins = 10000) on 200 points was enough to do
it. Both now walk one grid shared across the curve, so each line belongs
to exactly one gap and the count cannot exceed x_bins.
The curves move slightly as a result: the spurious extra points are gone, so a curve carries a few points more or fewer than it did, and the ones it carries sit on the grid. AUC, partial AUC and average precision are unchanged - they agree with 0.21.0 to within 2.2e-16, which is the rounding of the arithmetic rather than a change in what is computed.
Cap x_bins at 1e6. Every stage sized by
it - the interpolation buffers, the point reduction and the averaging
grid - allocates a vector of that length per curve, so
x_bins = 1e9 asked for tens of gigabytes and failed on the
allocation rather than at the argument. A million supporting points is
finer than a plot resolves and finer than the data behind it, so the
ceiling is well clear of real use.
Skip the interpolation of a gap that spans no grid line at all,
which is almost every gap once the input is large. That was costing a
division per input point to add at most x_bins points to
the whole curve, so it grew with the data while what it produced did
not. create_roc() is about 1.8x faster at 1e6 points and
create_prc() about 1.3x; evalmod() itself
gains about 4%, the curves being a small part of its work.
Add auc_boot(), which resamples a single test set,
so that auc_ci() can put an interval around an AUC without
several test sets to compare. Every interval in precrec
until now was built from the variation between test sets, and
auc_ci() on one dataset raised an error; that error now
names this function.
The resampling is stratified - positives drawn from the positives, negatives from the negatives - so every resample keeps the class balance of the original. An unstratified bootstrap of imbalanced data varies the balance from resample to resample, which moves the precision-recall baseline underneath the quantity being estimated.
auc_ci() on the result gives a percentile interval, read
off the resampled values rather than assumed from a distribution over
them, so it cannot leave [0, 1] and needs none of the
clipping the multi-dataset interval does. dtype is refused
there for the same reason.
Add auc_diff(), which compares two models on the
same resamples. The difference is calculated within a resample,
so the interval describes the difference itself. Two separate intervals
from auc_ci() cannot be read that way: they can overlap
while the difference is clearly on one side of zero, because they say
nothing about how the two models move together.
It reports a percentile interval and a p-value, floored at
2 / (n + 1) so that a resample count cannot manufacture
significance it does not support.
Verified rather than asserted: the bootstrap standard error matches the analytic variance of DeLong for the ROC AUC to within 2% at n = 100 to 1000, and the 95% interval covered a known true AUC in 142 of 150 simulations. Both checks are in the test suite, the first as an independent implementation of the DeLong formula.
Add a website page comparing precrec with
ROCR, pROC, PRROC,
yardstick, scikit-learn and
imbalanced-learn: which to reach for, what
precrec does not do that they do, and the one place the
numbers disagree rather than the interfaces. The step estimator of the
precision-recall area, which most of them report, reads more than a
tenth high at two percent positives against the interpolated area
auc() returns, and the page computes that rather than
claiming it.
Add a website page on using precrec from
tidymodels. No adapter is needed:
collect_predictions() returns a fold column, a truth column
and one .pred_<class> column per class, which is the
shape mmdata() already takes through nfold_df,
and factor labels have always been accepted.
The page leads with the one difference that is silent when it bites.
yardstick treats the first factor level as the event;
precrec treats the last as the positive class. Pairing
yardstick’s default score column with
precrec’s default inverts the AUC without a warning, and
the page shows a case that comes out 1 one way and 0 the other.
tidymodels stays out of the package entirely,
Suggests included, so the chunks that need it are shown
rather than run and the website builds without it.
bench/run_tidymodels_parity.R checks the page against a
real fit instead, and found two of its recipes wrong before it
shipped.
Add classification_report(), the per-class table of
precision, recall and F-score that scikit-learn’s
classification_report prints, with the
macro avg and weighted avg rows under it. It
reproduces scikit-learn’s documented examples number for
number.
The third summary row follows the rule scikit-learn
documents. A binary problem cut at a threshold puts every observation in
exactly one of the two classes, so the row is accuracy. A
multi-class problem is evaluated one-vs-rest and each class is
thresholded on its own, so an observation can fall into no class or into
several; there is then no single-label accuracy, and the row is
micro avg, computed from the true positives, false
positives and false negatives pooled over the classes. Where the
predictions do happen to be single-label the two coincide, which is the
identity scikit-learn gives for printing one and not the
other.
at names the operating point and has no default.
scikit-learn reports on y_pred, so its caller
has already chosen one; precrec holds scores and evaluates
every cutoff, and no threshold is meaningful on every score scale, so
the choice stays with the caller. It takes one number for all classes or
one per class.
zero_division sets what an empty denominator reports,
0 as in scikit-learn or NA as in
the per-cutoff metrics of evalmod().
Add a website page for it, and trim the metric pages back towards the length the site was built at. Each release since had appended a section to them, taking the article average from the 88 lines the site was built at to
99. It is now 93 over 22 articles, one more article than before.
Nothing a page needed was dropped: the cuts are restated prose, a
duplicated auc() chunk on the AUC page, and an unused
beta = 2 example on the agreement page.
Settle on “metric” as the word for the quantities
evalmod(metrics = ),
metric_curve(x_metric = , y_metric = ) and
prob_metrics() name. The documentation had called them
“measures” throughout while the arguments that select them had always
been called metrics, so a reader had to work out that the
two words meant one thing. Nothing about what precrec
computes has changed.
The website articles that describe them moved from
measures-*.html to metrics-*.html, and
plots-basic-measures.html to
plots-basic-metrics.html. The old addresses redirect to the
new ones.
print() on a basic-mode object now heads its summary
column Metric rather than Meas., and its
banner reads
=== Basic performance evaluation metrics ===.
Correct the count in README.md, which said ten
opt-in metrics when there have been seventeen since 0.18.0.
Add the jaccard,
positive_likelihood_ratio and
negative_likelihood_ratio basic measures, opt-in through
evalmod(metrics = ) like the other added measures. They are
the scikit-learn classification metrics
precrec did not already have under a name of its own.
jaccard is the Jaccard index,
TP / (TP + FP + FN), also called the critical success index
or threat score. It is the confusion matrix with its true negative
corner left out, which is the same omission precision and
sensitivity make, and it is why it holds still on an
imbalanced dataset where accuracy is mostly counting true
negatives.
The two likelihood ratios are sensitivity / fpr and
fnr / specificity. precrec already carried
their quotient - the diagnostic odds ratio odds is
LR+ / LR- - but neither one alone, and a classifier can be
worth using on the strength of one of them. Both divide by a rate that
is 0 over part of every ranking, and are NA
there, as odds and lift already are where
their own denominator vanishes.
Add the D2 scores d2_brier and
d2_logloss to prob_metrics() and
prob_metrics_ci(), through a new metrics
argument.
A D2 score divides a loss by the loss of the null model - the one
that predicts the observed prevalence for every case and ignores the
scores - and subtracts the result from 1, so it reads like
R-squared: 1 is perfect, 0 is no better than
knowing the prevalence, and a negative value is a model that does worse
than that. Negative values are not clipped, and a dataset holding a
single class has a null loss of 0 and so a D2 score of
NA.
The argument defaults to NULL, which is the three
metrics prob_metrics() has always returned, so an existing
call still gets three rows per model and dataset.
metrics = "all" asks for all five.
Add the roc_dist and sedi basic
measures, opt-in through evalmod(metrics = ) like the other
added measures.
roc_dist is the distance from the point
(1 - specificity, sensitivity) to the perfect corner of ROC
space. It is the one measure in the table that is better when it is
smaller, and the one whose maximum is sqrt(2) rather than
1; minimizing it is one of the standard ways of choosing an
operating point off a ROC curve.
sedi is the symmetric extremal dependence index, a skill
score from forecast verification built to stay informative when the
positive class is rare. The four logarithms in its definition are
undefined at both ends of every dataset, so the rates are clamped away
from 0 and 1 first.
Add average_precision(), which returns the step
estimator of the area under a precision-recall curve - the precision at
each cutoff weighted by the recall it gains over the cutoff before
it.
It is a different estimator from the one auc() reports,
not a different way of adding up the same one: it joins the raw
precision-recall points with horizontal steps, where auc()
measures the area under the curve evalmod() has
interpolated the way Davis and Goadrich showed is correct. The step
estimator reads high wherever the two disagree, and auc()
remains the number to prefer. average_precision() is here
because several other packages report it under this name, and because
the size of the gap between the two is worth being able to see.
It is read off the raw per-cutoff points rather than off the curve,
so x_bins does not change it, and unlike
prbe() it survives
evalmod(raw_curves = FALSE).
Add the macro_weight argument to auc().
The macro-average of the per-class AUCs of a multiclass evaluation has
always weighted every class equally;
macro_weight = "prevalence" weights each class by the
number of observations it has instead, and names the added rows
macro-average-weighted to keep the two apart. The default
is unchanged, and the two agree on a balanced dataset.
For a ROC evaluation the two are the measures other packages call
roc_aunu and roc_aunp.
Rewrite the package website. The single long
Introduction vignette is now a short
Get started page plus three sets of short articles,
reachable from new How-to, Measures and
Plots menus: one page per task, one page per family of
measures, and one page per plot. The vignette shipped with the package
is the Get started page; the rest are on the website only,
so nothing was added to the size of the package.
Update README.md to match, and to describe the
measures added in 0.15.0 and 0.16.0.
Shrink the logo. man/figures/logo.png was 3392 x
3392 pixels and 430 KB, and ships inside the package; it is now 240 x
240 and 13 KB, which is the size the README and the website
actually display. The favicons are regenerated from it.
label column of
as.data.frame(evalmod(mode = "basic")), and the label panel
plot() and autoplot() draw from it, could come
out in a different order on macOS than on Linux for the same data. Tied
scores now keep the order they were given in, on every platform. No
evaluation measure changes; only the order of the tied instances
themselves.Add metric_curve(), which takes the name of a
measure for the x axis and the name of a measure for the y axis and
draws one against the other, the way ROCR::performance()
does. Every measure evalmod() can calculate is available on
both axes, under its own name or under the ROCR identifier.
It returns an ssxycurves, msxycurves,
smxycurves or mmxycurves object, chosen the
way evalmod() chooses between its own, and the object works
with print(), as.data.frame(),
fortify(), plot() and
autoplot().
Only two pairs are joined by a line: false positive rate against
sensitivity, which is the ROC curve, and sensitivity against precision,
which is the precision-recall curve. Those two have a defined
interpolation, and for them metric_curve() hands the work
to the same code evalmod(mode = "rocprc") uses, so the two
cannot disagree. Every other pair is drawn as points, because joining
raw per-cutoff points with straight lines is the error this package was
written to avoid; pass type = "l" to join them anyway.
metric_curve() draws one curve per test dataset and does
not average over them. An average needs a rule for interpolating between
the points of each curve, which is what an unregistered pair does not
have.
Add the evaluation measures ROCR provides that
precrec did not: fpr, fnr,
false_discovery_rate, false_omission_rate,
predicted_positive_rate,
predicted_negative_rate, lift,
odds, mi, chisq and
cost. Each also answers to the identifier ROCR
uses for it - fall, miss, pcfall,
pcmiss, rpp, rnp,
mutual_information - and to its standard abbreviation where
it has one, so a call written against ROCR keeps working.
evalmod() gains cost_fp and
cost_fn for the two weights the cost measure
takes; with the default weights of 1 it is the error
rate.
They are not calculated unless asked for. evalmod()
gains a metrics argument that names the measures to add, or
takes "all"; the default NULL is the fourteen
measures the function has always returned, so an existing call gets the
same object with the same fourteen plot() and
autoplot() panels. A measure that was not calculated cannot
be plotted, and the error says which argument asks for it.
The odds ratio and the chi-square statistic are NA at
the top and the bottom of every dataset, where the 2x2 table has an
empty cell and neither is defined. ROCR reports an infinity
or a NaN there. NA is what
precision and npv already do with their own
undefined end, and it keeps an infinity off a shared axis. The mutual
information is 0 at those two points rather than
NA, because a cutoff that predicts one class for everything
carries no information about the labels - that value is defined, and it
is zero.
Add prbe(), which finds the points of a
precision-recall curve at which precision and recall are equal. It takes
the object evalmod() returns and gives back a data frame
with one row per break-even point, in the manner of
auc().
ROCR::performance(pred, "prbe") interpolates linearly
between adjacent raw precision-recall points to find the crossing, which
is not correct and is the reason this package exists;
prbe() reads the crossing off the curve
evalmod() has already interpolated properly.
Add the sar measure, the mean of accuracy, the AUC
of the ROC curve, and one minus the root mean squared error. Like the
other added measures it is opt-in through
evalmod(metrics = ). The RMSE reads the values of the
scores rather than their ranks, so sar warns and returns
NA when the scores are not probabilities between 0 and 1;
every other measure asked for in the same call is still
returned.
Add the root mean squared error to prob_metrics()
and prob_metrics_ci(), as the "rmse" metric.
It is the square root of the Brier score, so each model and dataset now
takes up three rows rather than two.
Declare stats in Imports. It was used
but not listed.
Fix the y axis of plot() for informedness and
markedness. Both run from -1 to 1, and both were drawn on a 0 to 1 axis,
which cut off the negative half of the curve. autoplot()
was never affected. The axis range of a measure now comes from one table
rather than from a list of names that had the internal short names
missing from it.
Support datasets with more than two classes, by one-vs-rest
decomposition. Pass a matrix with one score column per class together
with the class labels and mmdata() builds one binary
problem per class, carried on the model axis, so evalmod(),
plot(), autoplot(),
as.data.frame() and the averaging all treat the classes as
they would several models on one test set. mmdata() and
evalmod() gain a multiclass argument
("none" or "ovr"), detected from the input
when it is left unset; binary input is read exactly as before.
auc() gains a macro argument and reports the
macro-average of the per-class AUCs alongside them. The new
C3N150 sample dataset is a 3-class example.
Each one-vs-rest decomposition has its own class balance, so the baseline of a precision-recall curve differs from class to class. The plots now leave that baseline out whenever the datasets do not share a prevalence, rather than drawing one line that fits none of them.
Handle a dataset in which every label belongs to the same class
instead of stopping on it. evalmod(mode = "basic") now
warns and calculates what it can - accuracy and error rate are defined,
specificity without negatives and sensitivity without positives come
back as NA - where it used to stop. ROC and
precision-recall curves remain undefined for such a dataset, so
evalmod() still stops by default; the new
on_single_class = "na" asks it to warn and return
NA instead, so that one degenerate fold of an n-fold run no
longer aborts the whole evaluation. auc_ci() and
prob_metrics_ci() leave those NAs out of the
interval and report how many datasets it was built from.
Add five confusion-matrix measures to
evalmod(mode = "basic"): balanced accuracy, negative
predictive value, informedness (Youden’s J), markedness, and Cohen’s
kappa. They are calculated in the same pass as the existing measures and
appear as new curvetype values in
as.data.frame(), plot(),
autoplot(), fortify() and
print(). Code that plots the default set of measures now
gets fourteen panels instead of nine; pass curvetype to
pick a subset. evalmod(mode = "basic") does about a third
more work for them and holds about 40% more memory at peak;
evalmod() in its default mode is unaffected, because the
curves are drawn from three measures and the rest are no longer built
for it.
Add a beta argument to evalmod(), which
generalizes fscore to the F-beta score. The default
beta = 1 is the F1 score the function has always returned,
so existing results are unchanged.
Add prob_metrics() and
prob_metrics_ci(), which calculate the Brier score and the
log loss of prediction scores together with their confidence intervals
over multiple test datasets. Both measures read the values of the scores
rather than their ranks, so the scores must be probabilities; anything
outside the range 0 to 1 is rejected with a
precrec_error_invalid_scores condition.
Replace the vdiffr plot tests with
machine-independent snapshots, and drop vdiffr from
Suggests. The tests now record what a plot is made of – its
panels, their titles and axis labels, and the x/y/group data behind
every layer – as text, instead of comparing a rendered SVG. Rendering
brought in the local font metrics, so a baseline belonged to one
machine, could not be committed and had to be skipped on CI. The new
baselines live in tests/testthat/_snaps/ under version
control, the comparison runs everywhere including CI, and a failure
names what changed.
Stop autoplot() and fortify() warning
“Arguments in ... must be used” on every call. The
fortify methods accept raw_curves and
reduce_points for a common interface, but a single test
dataset has no average to contrast a raw curve with and the basic
measures have no point reduction, so those methods never read the
argument – which the ggplot2 fortify generic reported as a
possible misspelling. The behavior is unchanged; the arguments are now
consumed explicitly and documented as having no effect on those
objects.
Fix the panel titles of autoplot() in
mode = "basic". The measure name arrives from the plot data
as a factor, and the title lookup read it by its level code instead of
its name, so the label panel was titled “MCC” and the
mcc, npv and balanced_accuracy
panels were left untitled.
Require R >= 4.1 (was R >= 3.2.1)
Migrate the unit tests to testthat edition 3, and run test files in parallel
Remove the deprecated context() calls from the unit
tests
Convert the roxygen2 documentation to markdown. Links to
functions now render with parentheses, such as evalmod(),
and links to other packages are qualified, such as
gridExtra::arrangeGrob(). The rendered help pages are
otherwise unchanged.
Replace assertthat with cli and
rlang for argument validation. assertthat is
no longer a dependency. Errors raised by argument checks now carry
condition classes, so they can be caught by class rather than by
message: precrec_error_invalid_<argument>,
precrec_error_invalid_arg and precrec_error.
The wording of these messages has changed, but the same inputs are
accepted and rejected as before.
Suggest the nearest valid value when an argument is given one
that is not allowed. evalmod(mode = "ROCPRC") now points at
rocprc in the error, and a curvetype with a
typo in it is matched against the measure names. A value that resembles
nothing in the set is reported as before, without a guess.
Fix the error raised when an argument that must be a whole number
is given an infinite one. evalmod(x_bins = Inf) failed with
R’s own “missing value where TRUE/FALSE needed” rather than a
precrec condition, because Inf %% 1 is
NaN.
Use checkmate for the argument type checks behind
the existing helpers. The messages and the condition classes are
unchanged; checkmate and its only dependency,
backports, are the new Imports.
Replace the apply family with a small internal
.map_* family. All 44 lapply,
vapply and Filter calls in the package now go
through helpers named after their purrr equivalents, and most of them
through a typed one that states what the call returns. No dependency was
added and no behavior changed: the plot snapshots, the correctness suite
and the benchmarks are the same before and after.
Add the package website to URL in
DESCRIPTION, and rebuild the pkgdown site with the
Bootstrap 5 template.
Add inst/WORDLIST and a spell-check test, plus a
committed .lintr configuration.
Fix the ranking of NA scores when
na_worst = TRUE. The sentinel value used for
NA was DBL_MIN, the smallest positive
double, so NAs outranked every negative score instead of
being ranked last. Results change only when the scores contain both
NAs and negative values: evalmod() now returns
the same curves and AUCs for such input as it does when a constant is
added to every score. na_worst = FALSE was never
affected.
Calculate the standard errors of averaged curves and averaged
points with Welford’s algorithm instead of E[x^2] - E[x]^2.
The previous formula lost precision through catastrophic cancellation
and clamped the resulting negative variances to zero. Confidence bands
from evalmod(calc_avg = TRUE) change at around the 1e-8
level.
Add as.data.table() methods for the objects
evalmod() returns, so the table precrec builds
internally can be had without a conversion. They accept the same
arguments as as.data.frame() and hold the same content.
data.table was already a dependency.
Build the internal tables with data.table. The
public contract is unchanged: as.data.frame(),
fortify(), auc(), pauc() and
auc_ci() still return plain data frames, with the same
columns, types and row order as before. Tables reached through the
returned frame can no longer be modified by reference.
Speed up the pure-R conversion path used when
use_rcpp = FALSE. It grew the result with
rbind() once per curve, copying everything collected so far
on every pass. Over 20 test datasets it is now around 38 times faster
and allocates around 43 times less memory. auc_ci() and the
basic measure summary had the same pattern and got the same treatment.
The default Rcpp path was never affected.
Speed up as.data.frame() and autoplot()
for large curve objects, and cut the memory they need. The C++ converter
filled a C++ buffer and then copied it into the vectors it returned; it
now fills those vectors directly. Converting a curve object built from
200,000 observations is around 27% faster, and one pass of
evalmod() plus as.data.frame() over 1,000,000
observations peaks at 584 MB instead of 897 MB in
mode = "basic", and 424 MB instead of 470 MB for ROC and
precision-recall curves.
Speed up mmdata() by around 25%. Scores are sorted
through a comparison the sort can inline, rather than through a function
pointer, which it could not. The resulting ranks are unchanged.
Speed up the averaging of basic evaluation measures across
datasets, which evalmod(calc_avg = TRUE, mode = "basic")
performs. Collecting the distinct x values used a std::set
and a std::map, each allocating a node per value and
following a pointer per lookup, once for every point of every dataset; a
sorted vector and a binary search replace them. It is around 1.6 times
faster at 100,000 observations and above.
Restructure unit tests for svg comparisons with vdiff
Reformat signatures of S3 methods
Update unit tests to avoid is.atomic(NULL) issue
Update argument names of S3 functions to keep them consistent
Use patchwork to combine multiple plots instead of using grid and gridExtra
Test ggplot results with vdiffr
Fix a bug with as.data.frame when multiple datasets given
Add format_nfold function to convert a dataframe with n-fold data to a list
Add ‘aucroc’ mode for fast AUC (ROC)
Change how to treat ‘show_cb’ and ‘raw_curves’ options
Improve as.data.frame with Rcpp
Create GitHub pages with pkgdown
The first release version of precrec
The package offers five functions