---
title: "Screening: T-Rex+Screening and the biobank screening"
output:
  rmarkdown::html_vignette:
    toc: true
    css: vignette_styles.css
    includes:
      before_body: vignette_nav.html
bibliography: references.bib
csl: ieee.csl
link-citations: true
vignette: >
  %\VignetteIndexEntry{Screening: T-Rex+Screening and the biobank screening}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", fig.align = "center",
                      fig.width = 6, fig.height = 4, fig.retina = 2)
options(width = 80)
library(TRexSelector)
old_threads <- trex_set_num_threads(min(2L, trex_get_max_threads()))
```

# Screening before selecting

A genomics biobank holds hundreds of thousands of variables and hundreds of phenotypes. Running the full
T-Rex calibration on every phenotype is costly, and for most phenotypes the answer is that nothing is
selected. T-Rex+Screening [@machkour2023screen] is a single-pass version of the random experiments without
the L-loop and the T-loop: $K$ experiments with $p$ dummies each, stopped at the first dummy, and an
estimate of the false discovery rate (FDR) read off the vote. The estimate is one over the number of
selected variables, and it is 1 when nothing is selected. The screening reports this estimated FDR instead
of controlling a target, so it is a fast first pass. The biobank screening wraps it in a decision rule per
phenotype: when the estimated FDR of the screening falls into an acceptance window, the screening result
is kept, otherwise the full T-Rex selector runs as a fallback.

# T-Rex+Screening

The screening selector takes no target FDR.

```{r data}
set.seed(2024)
n <- 100
p <- 300
true_support <- c(3, 50, 120, 200, 280)
beta <- numeric(p)
beta[true_support] <- 2
X <- matrix(rnorm(n * p), n, p)
y <- as.numeric(X %*% beta + rnorm(n))
```

```{r screen}
scr <- TRexScreeningSelector$new(X, y, seed = 3, verbose = FALSE,
                                 control = trex_control(K = 20))
scr$select()
scr$selected_indices
scr$estimated_FDR
```

`beta_mat` holds the coefficients of the original variables at the stop of each experiment (one column
per experiment) and `dummy_betas` those of the dummies, the material the estimate is read off. The
dummies are fresh per experiment under the default L-loop strategy `"STANDARD"`.

```{r mats}
dim(scr$beta_mat)
dim(scr$dummy_betas)
```

## A bootstrap confidence interval

`use_bootstrap_CI = TRUE` selects by the coefficients instead of the vote. The non-zero dummy coefficients
of the experiments are resampled `R_boot` times, at least twice, for a confidence interval of their mean,
and a variable is selected when its averaged coefficient lies outside that interval. `confidence_level` is
the smallest level at which the selection is no larger than the ordinary one, searched on a grid of step
`ci_grid_step` in $(0, 1]$. The estimated FDR is again one over the number of selected variables. The
confidence-based screening is meant for the plain screening. With a dependency-aware variant it mostly
selects nothing, because the filter lowers the ordinary count that the confidence level has to reach.

```{r boot}
boot <- TRexScreeningSelector$new(X, y, seed = 3, verbose = FALSE,
                                  screen_control = trex_screen_control(use_bootstrap_CI = TRUE,
                                                                       R_boot = 200),
                                  control = trex_control(K = 20))
boot$select()
boot$used_bootstrap
boot$confidence_level
boot$estimated_FDR
boot$selected_indices
```

## Dependent variables

`trex_method` makes the screening dependency-aware. The dummies stay the same independent dummies, and
the variants lower the relative occurrence of a variable that resembles its correlated neighbours.
`"TREX_DA_AR1"` compares a variable with the neighbours inside a window, which holds the lags whose
correlation `abs(cor_coef)^lag` reaches `rho_thr_DA`. `"TREX_DA_NN"` keeps the neighbours of that
window whose sample correlation reaches `rho_thr_DA`, and a variable without such a neighbour keeps its
relative occurrence. With a threshold at or above the correlation of the data no neighbour is left, and
`"TREX_DA_NN"` selects as `"TREX"`, so the threshold stays below the correlation that the filter is meant
to guard against. At the default the variant selected as `"TREX_DA_AR1"` in simulations.
`"TREX_DA_EQUI"` compares a variable with all other variables and
`"TREX_DA_BLOCK_EQUI"` with the variables of its block out of `n_blocks`, and both are skipped when
`abs(cor_coef)` does not exceed `rho_thr_DA`. Setting `cor_coef` to `-2`, the default, estimates the
coefficient from the data, and `estimated_correlation` reports the coefficient that was used.

```{r da}
ar <- TRexScreeningSelector$new(X, y, seed = 3, verbose = FALSE,
                                screen_control = trex_screen_control(trex_method = "TREX_DA_AR1"),
                                control = trex_control(K = 20))
ar$select()
round(ar$estimated_correlation, 3)
```

## A response without signal

The screening stops at the first dummy, so about one null variable is expected to enter before it. On a
response of pure noise it selected at least one variable in a quarter to a half of the runs in
simulations, with fewer than one selected variable on average. A small selection is therefore no evidence
of a signal. The estimated FDR rests on that expected number of false discoveries and is no bound on the
FDP of a run. With two or more variables selected on noise it is 0.5 or less, while every one of them is
false. The biobank screening below accepts a screening result only when the estimate lies in its window
and runs the T-Rex selector otherwise, which selected nothing on noise in about 99 % of the runs.

# The biobank screening

`TRexBiobankScreeningSelector` screens one or many phenotypes against one genotype matrix. `Y` is a
vector for one phenotype or a matrix with one column per phenotype. The control is
`trex_biobank_control()`: the acceptance window of the estimated FDR is `lower_bound_FDR` to
`upper_bound_FDR`, and `target_FDR_trex` is the target of the fallback selector. The phenotypes draw
independent dummies, because the phenotype in column `k` runs with `seed + k - 1`, and a
single-phenotype run with that seed reproduces it.

```{r biobank}
Y <- cbind(y, as.numeric(X %*% (0.5 * beta) + rnorm(n)), rnorm(n))
colnames(Y) <- c("strong", "weak", "null")
bio <- TRexBiobankScreeningSelector$new(X, Y, seed = 3,
                                        biobank_control = trex_biobank_control(0.05, 0.15,
                                                                               target_FDR_trex = 0.1),
                                        control = trex_control(K = 20))
res <- bio$select()
length(res)
```

Each record names the phenotype, the method that produced the result and whether the fallback ran.
For a screening result `estimated_FDR` is the estimate of the accepted variant. For a fallback result it
is the target level `target_FDR_trex`, at which the T-Rex selector controls the FDR. An estimate can equal the
target, so `used_fallback_trex` tells the two cases apart.

```{r records}
summary_tab <- do.call(rbind, lapply(res, function(r)
  data.frame(phenotype = r$phenotype_index, selected = length(r$selected_indices),
             estimated_FDR = round(r$estimated_FDR, 3), method = r$method_used,
             fallback = r$used_fallback_trex)))
summary_tab
res[[1]]$selected_indices
```

The record also keeps the ordinary and the bootstrap screening results
(`selected_indices_screen_ordinary`, `estimated_FDR_screen_bootstrap`, ...), so the decision can be
audited. For a single phenotype `select()` returns one record instead of a list.

# Large designs

Both screening selectors accept a memory-mapped design (`mem_matrix()`), which the memory-mapping page
describes, and the biobank screening a memory-mapped response matrix `Y`, which it only reads. The
combination of the screening and a memory-mapped design is what solved sparse regression problems
with five million variables on a laptop [@scheidt2023bigtrex].

# Examples

The [TRexSelector_Examples](https://github.com/trexselector/TRexSelector_Examples/tree/main)
repository holds runnable demos and Monte Carlo studies of T-Rex+Screening and the biobank screening,
in memory and memory-mapped, with their results and a README per demo, in
[R](https://github.com/trexselector/TRexSelector_Examples/tree/main/R/trex_selector_methods/ad/trex_screening),
[Python](https://github.com/trexselector/TRexSelector_Examples/tree/main/Python/trex_selector_methods/ad/trex_screening)
and
[C++](https://github.com/trexselector/TRexSelector_Examples/tree/main/cpp/trex_selector_methods/ad/trex_screening).

```{r teardown, include = FALSE}
trex_set_num_threads(old_threads)
```

# Bibliography
