---
title: "Clustering: wheat varieties"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Clustering: wheat varieties}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set (collapse = TRUE, comment = "#>", fig.width = 6, fig.height = 4.2,
                       fig.align = "center")
optional = c ("cluster")
available = all (sapply (optional, requireNamespace, quietly = TRUE))
knitr::opts_chunk$set (eval = available)
```

```{r, echo = FALSE, eval = !available, results = "asis"}
cat ("**Note.** This vignette needs the following packages, some of which are missing:",
     paste (optional, collapse = ", "), "-- the code is shown but not run.\n")
```

One of the case studies of the *Analyse de données (L3 Informatique)* course, for which
`fdm2id` was written. Choosing the number of clusters when the criteria disagree, and judging a
partition by compactness, by stability, and against a known truth -- three rankings that need
not agree.

The other case studies are listed by `vignette (package = "fdm2id")`; they use the same
handful of functions on other data, and can be read in any order.

```{r, message = FALSE, warning = FALSE}
library (fdm2id)
```

# The data

Three varieties of wheat grain -- Kama, Rosa and Canadian -- with 70 observations of each,
studied by X-ray. Seven geometric descriptors were extracted: area, perimeter, compactness,
length, width, asymmetry coefficient and length of the groove. The eighth column is the
variety, which is *not* used to build the clusters -- only to judge them at the very end.

```{r}
data (wheat)
summary (wheat [, -8])
```

```{r, fig.height = 6}
plotdata (wheat [, -8])
```

# Question 1. Raw data, or centred and scaled?

```{r}
apply (wheat [, -8], 2, sd)
```

**Answer.** *The variables are expressed in different units, and their standard deviations are
nowhere near homogeneous -- `Area` weighs a hundred times more than `Compactness` in a
Euclidean distance. Better to work on the centred and scaled data.*

```{r}
wheat [, -8] = scale (wheat [, -8])
```

# Question 2. How many clusters do the methods see?

```{r}
# Variable: K-means starts from centres drawn at random. 'nstart = 100' keeps the best of a
# hundred starts, which makes the answer stable in practice, but only 'seed' makes it exact.
kmeans.getk (wheat [, -8], nstart = 100, graph = TRUE, seed = 0)
```

```{r, fig.height = 5}
single = HCA (wheat [, -8], method = "single")
plotclus (single, wheat, "tree")
```

```{r, fig.height = 5}
ward = HCA (wheat [, -8], method = "ward")
plotclus (ward, wheat, "tree")
```

**Answer.**

* *For $K$-means: the pseudo-$F$ is highest at two clusters, but its value at three is high as
  well.*
* *For single linkage: no cluster can be read off the dendrogram at all -- the chaining effect
  leaves a comb.*
* *For Ward: the largest jump splits the data in two, but a split in three also looks
  reasonable.*

Criteria disagree, and that disagreement is itself informative:

```{r}
sapply (c ("pseudo-F", "silhouette", "elbow"),
        function (criterion) kmeans.getk (wheat [, -8], criterion = criterion,
                                          nstart = 100, seed = 0))
```

# Question 3. With three clusters, which method is the most compact? The most stable?

```{r}
km = KMEANS (wheat [, -8], k = 3, nstart = 100, seed = 0)
ward = HCA (wheat [, -8], k = 3, method = "ward")
intern (km, wheat [, -8], eval = c ("intraclass", "interclass"))
intern (ward, wheat [, -8], eval = c ("intraclass", "interclass"))
```

```{r}
# Variable: stability resamples the dataset. Note that HCA itself is deterministic -- here the
# randomness is entirely in the resampling, not in the method being judged.
stability (KMEANS, wheat [, -8], type = "global", k = 3, nstart = 100, seed = 0)
stability (HCA, wheat [, -8], type = "global", method = "ward", k = 3, seed = 0)
```

**Answer.** *$K$-means produces the more compact clusters -- lower within-cluster inertia,
higher between-cluster inertia -- and the more stable ones, with a higher Jaccard index under
resampling.*

# Question 4. Which method comes closest to the three varieties?

Comparing a clustering with a known truth is not the same problem as evaluating a classifier:
the cluster numbers mean nothing, only the grouping does. `comp = "pairwise"` therefore counts
pairs of observations rather than labels.

```{r}
compare (km, wheat [, 8], comp = "pairwise")
compare (ward, wheat [, 8], comp = "pairwise")
```

**Answer.** *Ward's clusters are very slightly closer to the three varieties than $K$-means'.*

Note that this reverses the ranking of question 3: the more compact and more stable clustering
is not the one that recovers the varieties best. Compactness and stability are properties of
the partition on its own, and can be computed without ever knowing the varieties; agreement
measures the partition against something outside the data it was built from. Nothing
guarantees that the two rankings agree, and here they do not.

```{r}
table (km$cluster, wheat [, 8])
table (ward$cluster, wheat [, 8])
```

Both partitions line up with the three varieties, and both make their mistakes in the same
place: almost every misassigned grain involves Kama, the variety that sits geometrically
between the other two.