Package {Entropic.Scree}

Entropic Scree Measurement Mapping and Rank Estimation

Description

Maps relationships among measurements using a double-centered, joint-entropy-normalized mutual information matrix. Provides informational-axis loadings, primary and extended spectral-boundary diagnostics, and spectral-volume and informational-gravity summaries for discrete and discretized tabular data. The selected Primary Informational Rank is a candidate boundary in the information spectrum; a returned candidate does not establish a resolved elbow or correspondence with generating dimension. For the methods paper, see doi:10.5281/zenodo.22028086.

Usage

Entropic.Scree(
  data,
  low_entropy_thresh = 0.05,
  num_bins = NULL,
  bin_multiplier = 1,
  num_cores = parallel::detectCores() - 2,
  interactive_mode = TRUE,
  purge_constants = TRUE,
  check_collinearity = TRUE,
  triple_tap_window = 20,
  fwer_alpha = 0.01,
  extract_eigenvectors = TRUE,
  extract_bipolar_modules = TRUE,
  bipolar_top_n = NULL,
  return_processed_data = FALSE,
  categorical_missing = c("error", "category"),
  numeric_missing = c("error", "median_indicator")
)

Arguments

Details

Rows of measurement_loadings identify retained measurements and columns Axis_1, ..., Axis_K identify the selected primary informational axes, where K = K_roots. Every loading is retained, including zeros. These are raw signed eigenvector coefficients (equal-axis coordinates), not eigenvalue-scaled coordinates, measurement-to-score correlations, or native observation-level factor scores. For eigenvalue-scaled coordinates on a chosen positive-axis set, multiply each eigenvector column by the square root of its corresponding raw_eigenvalues value. Do not use floored working weights to treat nonpositive raw directions as positive geometry.

The full eigenvectors matrix includes all spectral directions for later inspection or rank updates; its presence does not mean all directions were selected. Eigenvector signs are arbitrary, and repeated eigenvalues can admit different bases for the same subspace. Opposite poles express informational contrasts, not positive versus negative correlation between measurements.

The Extended Signal Tail lies after K_roots and ends inclusively at K_extended; it is empty when the boundaries coincide. Spectral-volume allocations are distinct from coordinate geometry and are not measured proportions of causal signal or measurement error. Factor-Specific Informational Gravity refers to an informational axis, not an identified latent generating factor.

Missing-data preparation. Both missing-data options are off by default. Numeric medians use observed finite values only. Each generated companion is 0 for an originally observed value and 1 for an originally missing value. Identical row-by-row missingness patterns share one companion with all associated parent names, avoiding duplicate indicator weighting. Names and missing-category labels never overwrite or merge existing values. The original input object is not modified. Imputation and explicit missing states change the represented data; results are not missingness-conditioned MI.

Original measurements undergo the usual constant, duplicate, numeric QR, and entropy checks first. Indicators do not enter the numeric QR check. An indicator is mandatory when any associated numeric parent survives, even below the entropy threshold or duplicating an encoded original measurement. Mandatory indicators precede orphaned indicators in filtering. When no parent survives, a companion faces normal binary constant, duplicate and entropy filtering against retained originals and mandatory companions. Constant required indicators are invalid. Complete columns generate no indicator. Protection depends on retaining at least one associated numeric parent, not on the indicator's marginal entropy or uniqueness. Numeric filling uses the ordinary sample median of observed values.

Metric definitions and paper notation. On common prepared records, marginal entropy is H(X)=-\sum p(x)\ln p(x) in nats, and I(X;Y)=H(X)+H(Y)-H(X,Y). Joint-entropy normalization gives \mathcal{M}_{ij}=I(X_i;X_j)/H(X_i,X_j). Double-centering produces \mathcal{M}_c=\mathbf{H}\mathcal{M}\mathbf{H}, with \mathbf{H}=\mathbf{I}-\mathbf{1}\mathbf{1}^{T}/m; its trace is the sum of the raw eigenvalues \lambda_i. The detector's mean reference is \mu=\operatorname{Tr}(\mathcal{M}_c)/m; its baseline count is b=\#\{i:\eta_i>\mu\}, not a comparison with the mean working weight.

Working weights are \eta_i=\max(\varepsilon,\lambda_i), with \varepsilon=10^{-9}, S_{\varepsilon}=\sum_{i=1}^{m}\eta_i, and p_i=\eta_i/S_{\varepsilon} over all m directions. R_{eff}=\exp(-\sum_i p_i\ln p_i). With inclusive boundaries 1\le k=K_{roots}\le e=K_{extended}\le m, define P_{sig}=\sum_{i=1}^{K_{extended}}\eta_i/S_{\varepsilon}. Redundant Signal Volume is m-R_{eff}, Unique Signal Volume is P_{sig}R_{eff}, and Idiosyncratic Informational Volume is (1-P_{sig})R_{eff}. Their sum is m. Total Shared Signal Volume is TSV=(m-R_{eff})+P_{sig}R_{eff}. Average Informational Gravity is \mathrm{AIG}=TSV/K_{roots}. Factor-Specific Informational Gravity is FSIG_i=TSV\eta_i/\sum_{j=1}^{K_{roots}}\eta_j, and the Structural Topology Profile is T_i=FSIG_i/FSIG_1=\eta_i/\eta_1 over primary axes. Printed allocation percentages use m as denominator; the spectral fraction P_{sig} is different from TSV/m.

Raw positive mass is m_+=\sum_i\max(0,\lambda_i) and raw negative mass is m_-=\sum_{\lambda_i<0}|\lambda_i|. The Negative Spectral Mass Ratio is \mathrm{NSMR}=m_-/m_+ for m_+>0, before flooring; NSMR_percent is exactly 100\,\mathrm{NSMR}. Raw mass m_+ is distinct from working-weight sum S_{\varepsilon}. Internally, m_plus denotes only raw positive mass and S_epsilon denotes the working sum.

R identifiers. raw_eigenvalues corresponds to \lambda_i, eigenvalues to \eta_i, and similarity_matrix to \mathcal{M}_c. K_log_gap and K_triple_tap correspond to the paper's automatic recommendations K_{log} and K_{tap}; K_roots and K_extended are the selected boundaries. total_signal_volume corresponds to TSV, FSIG_final to FSIG_i, and structural_topology_profile to T_i. measurement_loadings contains v_{ik} on the selected primary axes, with every retained measurement included. Axis_1, Axis_2, etc. name the corresponding informational-axis pole lists. Auxiliary allocations use FSIG_auxiliary_bulk and FSIG_auxiliary_mean_reference; neither is a unit-eigenvalue Kaiser rule.

The auxiliary bulk endpoint is the last axis before a detected macro gap; without such a gap it is b. Its allocation cutoff is max(e, endpoint). The mean-reference auxiliary cutoff is max(e, b). Each auxiliary applies the same TSV formula at its own cutoff and distributes that volume across the current k primary proportions. These are implementation diagnostics, not separately defined primary metrics in the paper. All cutoffs and their sources are returned and all auxiliary vectors refresh on a rank update.

Numerical geometry and interpretation. A raw axis is numerically positive when its eigenvalue exceeds m\epsilon_{machine}\max(1,\max_i|\lambda_i|). An entirely degenerate spectrum is rejected before floor-based ranks or gravity can be presented. The tolerance is a numerical guard, not a spectral boundary estimator. Selected coefficients on null or negative raw directions may still be exported algebraically, with a warning and explicit status; they do not constitute a wholly positive-spectrum coordinate map. A working spectrum entirely at the floor also produces a warning. The working-weight and allocation formulas are not altered by this diagnostic. Local t thresholds are nominal heuristic thresholds, not established family-wise error control.

Data return. processed_data contains retained original-unit values, opted-in median fills or explicit categorical missing labels, and retained 0/1 companions. Internal discretization and category codes are solely MI-estimation inputs and do not trigger export. Integer versus double storage alone is not a change in numeric observations. The status records changed, added and dropped columns, or explains that the original input can be reused.

Value

A list containing the following components:

Output structure

Axis-specific pole lists use Axis_1, Axis_2, and so on, matching the columns of measurement_loadings. Each pole list contains Positive_Anchor and Negative_Anchor coefficients. NSMR is a ratio; NSMR_percent is 100 times that ratio.

Within auxiliary_diagnostics, macro_boundary_index identifies the first axis after a detected macro gap (NA when no gap is detected). macro_endpoint is the inclusive endpoint, and macro_endpoint_source identifies whether it comes from a detected gap or the mean-reference count. bulk_allocation_cutoff is the inclusive auxiliary allocation endpoint. Update.Entropic.Scree() returns a result without modifying its input object in place.

Examples

# Generate a small random dataset
dummy_data <- data.table::data.table(
  V1 = rnorm(50),
  V2 = rnorm(50),
  V3 = rnorm(50)
)

# Run without interactive rank-selection prompts
res <- Entropic.Scree(dummy_data, interactive_mode = FALSE, num_cores = 1)
res$measurement_loadings
# All measurements on one selected axis, with no loading cutoff:
res$measurement_loadings[, 1, drop = FALSE]
# Pole summaries use the same informational-axis names:
res$bipolar_modules$Axis_1
res$NSMR_percent

Update Entropic Scree Spectral Boundaries

Description

Changes the selected Primary Informational Rank and/or Extended Signal Tail boundary of an existing result. Recalculates spectral-volume and informational-gravity allocations and the primary-axis loading table without recomputing mutual information or the eigendecomposition.

Usage

Update.Entropic.Scree(
  scree_obj,
  new_K_roots = NULL,
  new_K_extended = NULL,
  bipolar_top_n = NULL,
  interactive_mode = TRUE
)

Arguments

Details

The returned primary-axis loading matrix uses the same saved full eigenvectors. All retained measurements are included on each selected axis. If the input result has no saved eigenvectors, loading outputs remain NULL; this function does not silently repeat the eigendecomposition. Pole lists use Axis_1, Axis_2, etc., matching the primary loading-matrix columns. See Entropic.Scree for definitions and interpretation. Both explicitly named auxiliary allocations refresh using the current ranks. Their explicit inclusive cutoffs and mean raw-eigenvalue reference are recorded in auxiliary_diagnostics. Macro metadata are calculated from the saved spectrum without an eigensolve. Geometry status is unavailable when the input has no raw eigenvalues. Preparation receipts and any saved original-unit analysis data remain unchanged.

Value

The updated result list, returned invisibly, including the selected boundaries, primary measurement_loadings, complete pole lists by default, spectral-volume allocations, informational gravity, and Structural Topology Profile. The saved full eigenvectors and eigenvalues are unchanged.

Output structure

Axis-specific pole lists use Axis_1, Axis_2, and so on, matching the columns of measurement_loadings. Each pole list contains Positive_Anchor and Negative_Anchor coefficients. NSMR is a ratio; NSMR_percent is 100 times that ratio.

Within auxiliary_diagnostics, macro_boundary_index identifies the first axis after a detected macro gap (NA when no gap is detected). macro_endpoint is the inclusive endpoint, and macro_endpoint_source identifies whether it comes from a detected gap or the mean-reference count. bulk_allocation_cutoff is the inclusive auxiliary allocation endpoint. Update.Entropic.Scree() returns a result without modifying its input object in place.