memshare: Shared memory multithreading in R via C++17, with zero-copy access through ALTREP data structures.

Overview

memshare enables multicore computation in R without redundant memory copies. Large vectors, matrices, or lists are stored once in shared memory and exposed to R processes as ALTREP views. This allows workers in a PSOCK cluster to operate on the same physical data while avoiding serialization overhead.

Key features:

• Shared memory allocation via C++17 back-end (shm_open on Unix, MapViewOfFile on Windows).
• ALTREP wrappers so R sees shared objects as native vectors or matrices.
• High-level parallels to parallel::parApply and parallel::parLapply:
  • memApply() — apply a function row/column-wise over a matrix in shared memory.
  • memLapply() — apply a function element-wise over a list in shared memory.
• Automatic cleanup of shared objects when all views are released or when the package is unloaded.
• Tested on Linux, macOS, and Windows.

Installation

From CRAN:

install.packages("memshare")

From GitHub (development version):

remotes::install_github("mthrun/memshare")

System requirements: R ≥ 4.0 and a C++17 compiler.

Quick start

Example 1: Parallel correlations with a matrix

library(memshare)
library(parallel)

set.seed(1)
n = 10000
p = 2000

X = matrix(rnorm(n * p), n, p)
y = rnorm(n)

res = memApply(
  X = X, MARGIN = 2,
  FUN = function(v, y) cor(v, y),
  VARS = list(y = y)
)
str(res)

Example 2: List operations

library(memshare)
library(parallel)

list_length = 1000
matrix_dim = 100

ListV = lapply(
     1:list_length,
     function(i) matrix(rnorm(matrix_dim * matrix_dim),
     nrow = matrix_dim, ncol = matrix_dim))

y = rnorm(matrix_dim)

namespace = "ID123"
res = memshare::memLapply(ListV, function(el, y) {
   el %*% y
}, NAMESPACE=namespace, VARS=list(y=y), MAX.CORES = 1)

Each element el of ListV is multiplied by y in parallel. The list resides once in shared memory.

Concepts

• Pages: memory regions owned by the current R session that loaded the package.

• Views: ALTREP wrappers exposing shared memory variables (read/write capable).

• Namespaces: string identifiers defining a shared memory context across sessions.

When the package is detached, all handles and associated shared memory pages are released, unless another R process still holds references.

Manual

The full manual for users or developers is available here: Package documentation

References