Package {rMIDAS2}

rMIDAS2: Multiple Imputation with 'MIDAS2' Denoising Autoencoders

Description

Fits 'MIDAS' denoising autoencoder models for multiple imputation of missing data, generates multiply-imputed datasets, computes imputation means, and runs Rubin's rules regression analysis. Wraps the 'MIDAS2' 'Python' engine via a local 'FastAPI' server over 'HTTP', so no 'reticulate' dependency is needed at runtime. Methods are described in Lall and Robinson (2022) doi:10.1017/pan.2020.49 and Lall and Robinson (2023) doi:10.18637/jss.v107.i09.

Author(s)

Maintainer: Thomas Robinson t.robinson7@lse.ac.uk

Authors:

• Thomas Robinson t.robinson7@lse.ac.uk

• Ranjit Lall ranjit.lall@politics.ox.ac.uk

See Also

Useful links:

• https://github.com/MIDASverse/MIDAS2

• Report bugs at https://github.com/MIDASverse/MIDAS2/issues

Build a base request pointing at the running server

Description

Build a base request pointing at the running server

Usage

base_req(path)

Arguments

Value

An httr2 request object.

Record the column types of the user's data

Description

The server returns imputed categorical values as level strings. Capturing the input types lets midas_transform() hand back a data frame with the same column classes (and factor levels) the user supplied.

Usage

capture_col_meta(data)

Arguments

Value

A named list of per-column type descriptors.

Check that arrow is available for the parquet transport

Description

Check that arrow is available for the parquet transport

Usage

check_arrow_available(categorical)

Arguments

Value

Invisibly TRUE, or throws an error.

Check whether the installed backend is up-to-date with PyPI

Description

Compares the locally installed version of midasverse-midas-api against the latest release on PyPI. Runs silently on success; emits a message when an update is available. Failures (e.g. no network) are silently ignored.

Usage

check_backend_version(python, package = "midasverse-midas-api")

Arguments

Value

No return value, called for side effects.

Remove the saved virtualenv path

Description

Remove the saved virtualenv path

Usage

clear_venv_path()

Value

No return value, called for side effects.

Combine results using Rubin's rules

Description

Runs a GLM across all stored imputations and combines the results using Rubin's combination rules for multiple imputation inference.

Usage

combine(
  model_id,
  y,
  ind_vars = NULL,
  dof_adjust = TRUE,
  incl_constant = TRUE,
  ...
)

Arguments

Value

A data frame with columns term, estimate, std.error, statistic, df, and p.value.

Examples

## Not run: 
df <- data.frame(Y = rnorm(200), X1 = rnorm(200), X2 = rnorm(200))
df$X1[sample(200, 40)] <- NA
fit <- midas_fit(df, epochs = 10L)
midas_transform(fit, m = 10)
results <- combine(fit, y = "Y")
results

## End(Not run)

Path to the package config directory

Description

Path to the package config directory

Usage

config_dir()

Value

Character path to the config directory.

Ensure the server is running

Description

Starts the server if it is not already running. Called internally by every client function so users never have to manage the server manually.

Usage

ensure_server(...)

Arguments

Value

Invisibly returns the base URL of the running server.

Examples

## Not run: 
ensure_server()

## End(Not run)

Extract model ID from a string or fitted model object

Description

Accepts either a bare character model ID or a list with a ⁠$model_id⁠ element (as returned by midas_fit() or midas()).

Usage

extract_model_id(x)

Arguments

Value

Character model ID.

Find a free TCP port

Description

Samples random ports in the dynamic range and uses serverSocket() to verify availability.

Usage

find_free_port()

Value

Integer port number.

GET and return parsed body

Description

GET and return parsed body

Usage

get_json(path, timeout = 60, simplify = TRUE)

Arguments

Value

Parsed JSON response as a list.

Does a data frame contain categorical columns?

Description

Factor, character and logical columns are all treated as categorical: the MIDAS engine models them with a softmax (multi-level) or binary head.

Usage

has_categorical(data)

Arguments

Value

TRUE if any column is a factor, character or logical.

Compute mean imputation

Description

Calculates the element-wise mean across all stored imputations for a model.

Usage

imp_mean(model_id, ...)

Arguments

Value

A data frame with the mean imputed values.

Examples

## Not run: 
df <- data.frame(X1 = rnorm(200), X2 = rnorm(200))
df$X1[sample(200, 40)] <- NA
fit <- midas_fit(df, epochs = 10L)
midas_transform(fit, m = 10)
mean_df <- imp_mean(fit)

## End(Not run)

Install the MIDAS2 Python backend

Description

Creates an isolated Python environment and installs the midasverse-midas-api package (which pulls in midasverse-midas as a dependency).

Usage

install_backend(
  method = c("pip", "conda", "uv"),
  envname = "midas2_env",
  package = "midasverse-midas-api"
)

Arguments

Details

This is the only function in the package that uses reticulate, and only for environment creation. It is never used at runtime.

Value

No return value, called for side effects.

Examples

## Not run: 
install_backend()
install_backend(method = "conda")

## End(Not run)

Load the saved virtualenv path (or NULL)

Description

Load the saved virtualenv path (or NULL)

Usage

load_venv_path()

Value

Character path or NULL.

Multiple imputation (all-in-one)

Description

Convenience function that fits a MIDAS model and generates imputations in a single call. Equivalent to calling midas_fit() followed by midas_transform().

Usage

midas(
  data,
  m = 5L,
  hidden_layers = c(256L, 128L, 64L),
  dropout_prob = 0.5,
  epochs = 75L,
  batch_size = 64L,
  lr = 0.001,
  corrupt_rate = 0.8,
  num_adj = 1,
  cat_adj = 1,
  bin_adj = 1,
  pos_adj = 1,
  omit_first = FALSE,
  seed = 89L,
  ...
)

Arguments

Value

A list with model_id and imputations (a list of data frames). Columns keep the classes of data, and cells that were observed in data are returned exactly (see the "Observed values" section of midas_transform()).

Examples

## Not run: 
df <- data.frame(X1 = rnorm(200), X2 = rnorm(200))
df$X1[sample(200, 40)] <- NA
result <- midas(df, m = 5, epochs = 10)
head(result$imputations[[1]])

## End(Not run)

Registry of column metadata, keyed by model id

Description

midas_transform() may be called with a bare model id, so the types captured at fit time are stashed here rather than only on the fit object.

Usage

midas_col_meta_registry

Fit a MIDAS model

Description

Sends data to the server and fits a MIDAS denoising autoencoder.

Usage

midas_fit(
  data,
  hidden_layers = c(256L, 128L, 64L),
  dropout_prob = 0.5,
  epochs = 75L,
  batch_size = 64L,
  lr = 0.001,
  corrupt_rate = 0.8,
  num_adj = 1,
  cat_adj = 1,
  bin_adj = 1,
  pos_adj = 1,
  omit_first = FALSE,
  seed = 89L,
  ...
)

Arguments

Value

A list with model_id, n_rows, n_cols, col_types and col_meta (the column types of data, used by midas_transform() to return imputations with the classes you supplied).

Column types

Numeric columns are sent as numbers. Factor, character and logical columns are categorical: they are sent via Parquet, which preserves their type so that the engine fits a softmax head over the levels of a multi-level factor and a binary head over a two-level one. Character and logical columns are converted to factors on the way out. The Parquet transport requires the arrow package; without it, data containing categorical columns raise an error rather than being silently mangled.

Choosing dropout_prob

The default of 0.5 suits wide data with many columns. On narrow frames – fewer than roughly 15 columns – it removes most of the signal available to reconstruct each cell, and a smaller value (around 0.2) can substantially improve reconstruction accuracy. corrupt_rate is generally best left at its default. If imputations on a narrow data set look poor, lower dropout_prob before changing anything else.

Examples

## Not run: 
df <- data.frame(X1 = rnorm(200), X2 = rnorm(200), X3 = rnorm(200))
df$X2[sample(200, 40)] <- NA
fit <- midas_fit(df, epochs = 10L)
fit$model_id

# categorical columns are supported directly
df$grp <- factor(sample(c("a", "b", "c"), 200, TRUE))
df$grp[sample(200, 30)] <- NA
fit2 <- midas_fit(df, epochs = 10L)
fit2$col_types

## End(Not run)

Generate multiple imputations

Description

Generates m imputed datasets from a fitted MIDAS model.

Usage

midas_transform(model_id, m = 5L, data = NULL, ...)

Arguments

Value

A list of m data frames, each with imputed values. Columns are returned with the classes supplied to midas_fit(), including factor levels and ordering.

Observed values

The imputation engine ingests data as float32, and returns observed cells from that downcast copy rather than re-estimating them. An observed numeric value therefore comes back as the binary32 round-trip of what you supplied – a relative change of roughly 6e-8, which is invisible on standardised covariates but is a change of about 1e-4 on data of order 1000. Pass the original frame as data to have observed cells restored exactly. midas() does this automatically, since it is given the data.

Examples

## Not run: 
df <- data.frame(X1 = rnorm(200), X2 = rnorm(200))
df$X1[sample(200, 40)] <- NA
fit <- midas_fit(df, epochs = 10L)
imps <- midas_transform(fit, m = 10)
head(imps[[1]])

# restore observed cells exactly
imps <- midas_transform(fit, m = 10, data = df)
identical(imps[[1]]$X2[!is.na(df$X2)], df$X2[!is.na(df$X2)])

## End(Not run)

Overimputation diagnostic

Description

Masks a fraction of observed values, re-imputes them, and computes RMSE to assess imputation quality.

Usage

overimpute(model_id, mask_frac = 0.1, m = 5L, seed = NULL, ...)

Arguments

Value

A list with rmse (named numeric vector) and mean_rmse.

Examples

## Not run: 
df <- data.frame(X1 = rnorm(200), X2 = rnorm(200))
df$X1[sample(200, 40)] <- NA
fit <- midas_fit(df, epochs = 10L)
diag <- overimpute(fit, mask_frac = 0.1)
diag$mean_rmse

## End(Not run)

Parse a JSON table response into a data.frame

Description

Builds the frame column by column from the unsimplified JSON body. The previous implementation went through do.call(rbind, ...), which produces a character matrix as soon as one column is non-numeric and so returned every column – including numeric ones – as character.

Usage

parse_table(res, meta = NULL)

Arguments

Value

A data frame with per-column types preserved.

POST JSON and return parsed body

Description

POST JSON and return parsed body

Usage

post_json(path, body, timeout = 600, simplify = TRUE)

Arguments

Value

Parsed JSON response as a list.

Send a data frame to a parquet endpoint as multipart form data

Description

Send a data frame to a parquet endpoint as multipart form data

Usage

post_parquet(path, data, params, timeout = 600)

Arguments

Value

Parsed JSON response.

Coerce categorical columns to factor for the parquet transport

Description

arrow writes a factor as a dictionary-typed column, which pandas reads back as a category dtype; that is what makes the MIDAS engine give the column a softmax head. Bare character and logical columns are written as plain strings/booleans, which the engine rejects, so they are converted to factors here.

Usage

prepare_categorical(data)

Arguments

Value

The data frame with character and logical columns coerced to factor.

Restore the column types the user supplied

Description

Restore the column types the user supplied

Usage

restore_col_types(df, meta)

Arguments

Value

df with columns coerced back to their original classes.

Restore the observed cells of the user's data exactly

Description

The engine ingests data as float32 (midas2/dataset.py), and writes observed cells straight back from that downcast copy, so an observed numeric value returns as the binary32 round-trip of what was supplied – a relative change of about 6e-8, which on data of order 1000 is a visible 1e-4. Nothing is being re-estimated: the intent is that observed cells are untouched, and this restores them bit-exactly from the original frame.

Usage

restore_observed(imputed, data)

Arguments

Details

Categorical cells already round-trip exactly (they travel as level strings), so they are unaffected; they are restored here too so that the guarantee is uniform across column types.

Value

imputed, with every cell that was observed in data replaced by its original value.

Save the virtualenv path to persistent config

Description

Save the virtualenv path to persistent config

Usage

save_venv_path(path)

Arguments

Value

No return value, called for side effects.

Start the MIDAS2 API server

Description

Launches ⁠python -m midas2_api⁠ as a background process and waits for the ⁠/health⁠ endpoint to respond.

Usage

start_server(python = "python3", port = NULL, venv = NULL, max_wait = 120L)

Arguments

Value

Invisibly returns the port number.

Examples

## Not run: 
start_server()
start_server(venv = "~/.virtualenvs/midas2_env")

## End(Not run)

Stop the MIDAS2 API server

Description

Kills the background Python process and clears the internal state.

Usage

stop_server()

Value

No return value, called for side effects.

Examples

## Not run: 
stop_server()

## End(Not run)

Convert an all-numeric data frame to a nested list suitable for JSON

Description

The ⁠/fit⁠ and ⁠/complete⁠ endpoints accept numbers only. Categorical data must go through the parquet transport instead (see midas_fit()), which preserves column types; this helper therefore validates that it has been handed numeric data rather than silently coercing.

Usage

to_nested_list(x)

Arguments

Value

A nested list of rows.

Assemble the shared training form fields

Description

Assemble the shared training form fields

Usage

train_params(
  hidden_layers,
  dropout_prob,
  epochs,
  batch_size,
  lr,
  corrupt_rate,
  num_adj,
  cat_adj,
  bin_adj,
  pos_adj,
  omit_first,
  seed
)

Uninstall the MIDAS2 Python backend

Description

Stops the running server (if any), removes the Python environment created by install_backend(), and clears the saved configuration.

Usage

uninstall_backend(method = c("pip", "conda", "uv"), envname = "midas2_env")

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
uninstall_backend()
uninstall_backend(method = "conda")

## End(Not run)

Update the MIDAS2 Python backend

Description

Upgrades the midasverse-midas-api package (and its dependencies) in the existing Python environment. Stops the running server first so that the new version is loaded on next use.

Usage

update_backend(
  method = c("pip", "conda", "uv"),
  envname = "midas2_env",
  package = "midasverse-midas-api"
)

Arguments

Value

No return value, called for side effects.

Examples

## Not run: 
update_backend()

## End(Not run)