Migrating from rMIDAS to rMIDAS2

Why rMIDAS2?

rMIDAS2 is the successor to rMIDAS. It re-implements the MIDAS multiple imputation algorithm with several improvements:

rMIDAS rMIDAS2
Backend TensorFlow (Python, via reticulate) PyTorch (Python, via local HTTP API)
Runtime R dependency on reticulate Yes No
Preprocessing Manual (convert()) Automatic
Python versions 3.6–3.10 3.9+
TensorFlow required Yes (< 2.12) No

The API is deliberately simpler: most pipelines that required four function calls in rMIDAS need just one or two in rMIDAS2.

Installation

# Remove rMIDAS (optional -- it can coexist)
# remove.packages("rMIDAS")

# Install rMIDAS2
install.packages("rMIDAS2")

# One-time Python backend setup
library(rMIDAS2)
install_backend()

Side-by-side comparison

1. Setup

rMIDAS required configuring a reticulate Python environment with TensorFlow:

# --- rMIDAS ---
library(rMIDAS)
# Python environment configured automatically on first load,
# or manually via set_python_env()

rMIDAS2 uses a standalone Python server – no reticulate needed at runtime:

# --- rMIDAS2 ---
library(rMIDAS2)
install_backend()        # one-time setup
# The server starts automatically when you call any imputation function

2. Data preparation

rMIDAS required explicit preprocessing with convert(), where you had to specify which columns were binary and which were categorical:

# --- rMIDAS ---
data(adult)
adult_conv <- convert(adult,
                      bin_cols = c("income"),
                      cat_cols = c("workclass", "marital_status"),
                      minmax_scale = TRUE)

rMIDAS2 detects column types automatically – just pass your data frame directly:

# --- rMIDAS2 ---
# No convert() step needed. Pass raw data to midas() or midas_fit().

Numeric columns are modelled on the real line, factors with more than two levels get a softmax head, and two-level factors a binary head. Character and logical columns are treated as factors. Because the levels have to reach the Python engine intact, data containing categorical columns are sent as Parquet, which requires the arrow package:

install.packages("arrow")

Without arrow, categorical data raise an error naming the package to install; numeric-only data are unaffected.

Note that this worked incorrectly before version 0.2.0: passing a data frame containing a factor raised an HTTP 422 error from the server. If you are on 0.1.1 or earlier, upgrade rather than one-hot encoding by hand – encoding categorical variables as separate dummy columns causes them to be modelled as independent binary variables rather than as a single multinomial draw.

3. Training

rMIDAS used train():

# --- rMIDAS ---
mid <- train(adult_conv,
             training_epochs = 20L,
             layer_structure = c(256, 256, 256),
             input_drop      = 0.8,
             learn_rate      = 0.0004,
             seed            = 89L)

rMIDAS2 uses midas_fit() (or the all-in-one midas()):

# --- rMIDAS2 ---
fit <- midas_fit(adult,
                 epochs        = 20L,
                 hidden_layers = c(256L, 128L, 64L),
                 corrupt_rate  = 0.8,
                 lr            = 0.001,
                 seed          = 89L)

Parameter name changes:

rMIDAS (train()) rMIDAS2 (midas_fit()) Notes
training_epochs epochs
layer_structure hidden_layers Default changed from 256-256-256 to 256-128-64
input_drop corrupt_rate
learn_rate lr Default changed from 0.0004 to 0.001
dropout_level dropout_prob
train_batch batch_size Default changed from 16 to 64
cont_adj num_adj
softmax_adj cat_adj
binary_adj bin_adj

4. Generating imputations

rMIDAS used complete():

# --- rMIDAS ---
imps <- complete(mid, m = 10)
# Returns a list of 10 data.frames
head(imps[[1]])

rMIDAS2 uses midas_transform():

# --- rMIDAS2 ---
imps <- midas_transform(fit, m = 10)
# Returns a list of 10 data.frames
head(imps[[1]])

Or skip midas_fit() + midas_transform() entirely and use the all-in-one midas():

# --- rMIDAS2 (all-in-one) ---
result <- midas(adult, m = 10, epochs = 20)
head(result$imputations[[1]])

5. Rubin’s rules regression

The combine() interface has changed:

rMIDAS took a formula and a list of completed data frames:

# --- rMIDAS ---
combine("income ~ age + hours_per_week", imps)

rMIDAS2 takes a model ID and an outcome variable name. Independent variables default to all other columns:

# --- rMIDAS2 ---
combine(fit, y = "income")

# Specify predictors explicitly:
combine(fit, y = "income", ind_vars = c("age", "hours_per_week"))

The output format is the same: a data frame with columns term, estimate, std.error, statistic, df, and p.value.

6. Overimputation diagnostic

rMIDAS required re-specifying the data and column types:

# --- rMIDAS ---
overimpute(adult,
           binary_columns  = c("income"),
           softmax_columns = c("workclass", "marital_status"),
           training_epochs = 20L,
           spikein = 0.3)

rMIDAS2 runs overimputation on an already-fitted model:

# --- rMIDAS2 ---
diag <- overimpute(fit, mask_frac = 0.1)
diag$mean_rmse
diag$rmse     # per-column RMSE

7. Mean imputation (new in rMIDAS2)

rMIDAS2 adds imp_mean(), which computes the element-wise mean across all imputations – useful as a quick single point estimate:

# --- rMIDAS2 only ---
mean_df <- imp_mean(fit)
head(mean_df)

8. Cleanup

rMIDAS2 runs a background Python server that should be stopped when you are done:

# --- rMIDAS2 ---
stop_server()

Complete migration example

Below is a full rMIDAS pipeline and its rMIDAS2 equivalent.

rMIDAS (old)

library(rMIDAS)

data(adult)
adult <- adult[1:1000, ]

# 1. Preprocess
adult_conv <- convert(adult,
                      bin_cols  = c("income"),
                      cat_cols  = c("workclass", "marital_status"),
                      minmax_scale = TRUE)

# 2. Train
mid <- train(adult_conv, training_epochs = 20L, seed = 89L)

# 3. Generate imputations
imps <- complete(mid, m = 5)

# 4. Analyse
combine("income ~ age + hours_per_week", imps)

rMIDAS2 (new)

library(rMIDAS2)

data(adult)
adult <- adult[1:1000, ]

# 1. Fit and impute (no preprocessing needed)
result <- midas(adult, m = 5, epochs = 20, seed = 89L)

# 2. Analyse
combine(result, y = "income", ind_vars = c("age", "hours_per_week"))

# 3. Clean up
stop_server()

Quick-reference cheat sheet

Task rMIDAS rMIDAS2
Install Python env Automatic / set_python_env() install_backend()
Preprocess data convert(data, bin_cols, cat_cols) Not needed
Train model train(data, training_epochs, ...) midas_fit(data, epochs, ...)
Generate imputations complete(model, m) midas_transform(model, m)
Train + impute (one step) Not available midas(data, m, epochs, ...)
Mean imputation Not available imp_mean(model)
Rubin’s rules combine(formula, df_list) combine(model, y, ind_vars)
Overimputation overimpute(data, ...) overimpute(model, mask_frac)
Shutdown Not needed stop_server()