wconf is a package that allows users to create weighted confusion matrices and accuracy scores
Used to improve the model selection process, the package includes several weighting schemes which can be parameterized, as well as the option for custom weight configurations. Furthermore, users can decide whether they wish to positively or negatively affect the accuracy score as a result of applying weights to the confusion matrix. “wconf” integrates with the “caret” package, but it can also work standalone when provided data in matrix form.
Confusion matrices are used to visualize the performance of classification models in tabular format. A confusion matrix takes the form of an “n x n” matrix depicting:
the reference category, in columns;
the predicted category, in rows;
the number of observation corresponding to each combination of “reference - predicted” category couples, as cells of the matrix.
Visually, the simplest binary classification confusion matrix takes on the form:
\[ A = \begin{bmatrix}TP & FP \\FN & TN\\ \end{bmatrix} \] where:
\(TP\) - True Positives - the number of observations that were “positive” and were correctly predicted as being “positive”
\(TN\) - True Negatives - the number of originally “negative” observations that were correctly predicted by the model as being “negative”.
\(FP\) - False Positives - also called “Type 1 Error” - represents observations that are in fact “negative”, but were incorrectly classified by the model as being “positive”.
\(FN\) - False Negatives - also called “Type 2 Error” - represents observations that are in fact “positive”, but were incorrectly classified by the model as being “negative”.
The traditional accuracy metric is compiled by adding the true positives and true negatives, and dividing them by the total number of observations.
\[ A = \frac{TP + TN} {N} \]
A weighted confusion matrix consists in attributing weights to all classification categories based on their distance from the correctly predicted category. This is important for multi-category classification problems (where there are three or more categories), where distance from the correctly predicted category matters.
The weighted confusion matrix, for the simple binary classification, takes the form:
\[ A = \begin{bmatrix}w1*TP & w2*FP \\w2*FN & w1*TN\\ \end{bmatrix} \]
In the case of the weighted confusion matrix, a weighted accuracy score can be calculated by summing up all of the elements of the matrix and dividing the resulting amount by the number of observations.
\[ A = \frac{w1*TP + w2*FP + w2*FN + w1*TN} {N} \]
For more details on the method, see the paper:
Monahov, A. (2023). wconf: The weighted confusion matrix and accuracy scores package for R
This function compiles a weight matrix according to one of several weighting schemas and allows users to visualize the impact of the weight matrix on each element of the confusion matrix.
In R, simply call the function:
weightmatrix(n, weight.type = "arithmetic", weight.penalty = FALSE, standard.deviation = 2, geometric.multiplier = 2, interval.high=1, interval.low = -1, custom.weights = NA, plot.weights = FALSE) {
The function takes as input:
n – the number of classes contained in the confusion matrix.
weight.type – the weighting schema to be used. Can be one of: “arithmetic” - a decreasing arithmetic progression weighting scheme, “geometric” - a decreasing geometric progression weighting scheme, “normal” - weights drawn from the right tail of a normal distribution, “interval” - weights contained on a user-defined interval, “custom” - custom weight vector defined by the user.
weight.penalty – determines whether the weights associated with non-diagonal elements generated by the “normal”, “arithmetic” and “geometric” weight types are positive or negative values. By default, the value is set to FALSE, which means that generated weights will be positive values.
standard.deviation – standard deviation of the normal distribution, if the normal distribution weighting schema is used.
geometric.multiplier – the multiplier used to construct the geometric progression series, if the geometric progression weighting scheme is used.
interval.high – the upper bound of the weight interval, if the interval weighting scheme is used.
interval.low – the lower bound of the weight interval, if the interval weighting scheme is used.
custom.weights – the vector of custom weights to be applied, is the custom weighting scheme was selected. The vector should be equal to “n”, but can be larger, with excess values being ignored.
plot.weights – optional setting to enable plotting of weight vector, corresponding to the first column of the weight matrix
The function outputs a matrix:
w | the nxn weight matrix. |
This function calculates the weighted confusion matrix by multiplying, element-by-element, a weight matrix with a supplied confusion matrix object.
In R, simply call the function:
wconfusionmatrix(m, weight.type = "arithmetic", weight.penalty = FALSE, standard.deviation = 2, geometric.multiplier = 2, interval.high=1, interval.low = -1, custom.weights = NA, print.weighted.accuracy = FALSE) {
The function takes as input:
m – the caret confusion matrix object or simple matrix.
weight.type – the weighting schema to be used. Can be one of: “arithmetic” - a decreasing arithmetic progression weighting scheme, “geometric” - a decreasing geometric progression weighting scheme, “normal” - weights drawn from the right tail of a normal distribution, “interval” - weights contained on a user-defined interval, “custom” - custom weight vector defined by the user.
weight.penalty – determines whether the weights associated with non-diagonal elements generated by the “normal”, “arithmetic” and “geometric” weight types are positive or negative values. By default, the value is set to FALSE, which means that generated weights will be positive values.
standard.deviation – standard deviation of the normal distribution, if the normal distribution weighting schema is used.
geometric.multiplier – the multiplier used to construct the geometric progression series, if the geometric progression weighting scheme is used.
interval.high – the upper bound of the weight interval, if the interval weighting scheme is used.
interval.low – the lower bound of the weight interval, if the interval weighting scheme is used.
custom.weights – the vector of custom weights to be applied, is the custom weighting scheme was selected. The vector should be equal to “n”, but can be larger, with excess values being ignored.
print.weighted.accuracy – optional setting to print the weighted accuracy metric, which represents the sum of all weighted confusion matrix cells divided by the total number of observations.
The function outputs a matrix:
w_m | the nxn weighted confusion matrix. |
This example provides a real-world usage example of the wconf package on the Iris dataset included in R.
We will attempt the more difficult task of predicting petal length from sepal width. In addition, for this task, we are only given categorical information about the length of the petals, specifically that they are:
“Short (length between: 1-3)”
“Medium (length between: 3-5 cm)”
“Long (length between: 5-7 cm)”.
Numeric data is available for the sepal width.
Using caret, we train a multinomial logistic regression model to fit the numeric sepal width onto our categorical petal length data. We run 10-fold cross-validation, repeated 3 times to avoid overfitting and find optimal regression coefficient values for various data configurations.
Finally, we extract the confusion matrix. We wish to weigh the confusion matrix to represent preference for observations fitted closer to the correct value. We would like to assign some degree of positive value to observations that are incorrectly classified, but are close to the correct category. Since our categories are equally spaced, we can use an arithmetic weighing scheme.
Let’s first visualize what this weighting schema would look like:
# View the weight matrix and plot for a 3-category classification problem, using the arithmetic sequence option.
weightmatrix(3, weight.type = "arithmetic", plot.weights = TRUE)
#> Error in weightmatrix(3, weight.type = "arithmetic", plot.weights = TRUE): could not find function "weightmatrix"
To obtain the weighted confusion matrix, we run the “wconfusionmatrix” command and provide it the confusion matrix object generated by caret, a weighting scheme and, optionally, parameterize it to suit our objectives. Using the “wconfusionmatrix” function will automatically determine the dimensions of the weighing matrix and the user need only specify the parameters associated with their weighting scheme of choice.
The following block of code produces the weighted confusion matrix, to out specifications.
# Load libraries and perform transformations
library(caret)
#> Warning: package 'caret' was built under R version 4.2.3
#> Loading required package: ggplot2
#> Warning: package 'ggplot2' was built under R version 4.2.3
#> Loading required package: lattice
#> Warning: package 'lattice' was built under R version 4.2.3
data(iris)
$Petal.Length.Cat = cut(iris$Petal.Length, breaks=c(1, 3, 5, 7), right = FALSE)
iris
# Train multinomial logistic regression model using caret
set.seed(1)
<- trainControl(method="repeatedcv", number=10, repeats=3)
control <- train(Petal.Length.Cat ~ Sepal.Width, data=iris, method="multinom", trace = FALSE, trControl=control)
model
# Extract original data, predicted values and place them in a table
= iris$Petal.Length.Cat
y = predict(model)
yhat = table(data=yhat, reference=y)
preds
# Construct the confusion matrix
= confusionMatrix(preds)
confmat
# Compute the weighted confusion matrix and display the weighted accuracy score
wconfusionmatrix(confmat, weight.type = "arithmetic", print.weighted.accuracy = TRUE)
#> Error in wconfusionmatrix(confmat, weight.type = "arithmetic", print.weighted.accuracy = TRUE): could not find function "wconfusionmatrix"