Monte Carlo conditional-likelihood inference for endogenous-covariate diversification models on phylogenetic trees. R package, version 1.1.0, submitted to CRAN.
devtools::install_github("franciscorichter/emphasis")| What | Where |
|---|---|
| Repository | https://github.com/franciscorichter/emphasis: source,
CI, docs site, install_github() |
| Docs site | https://franciscorichter.github.io/emphasis/: overview, the four covariates and their biological meaning (covariates), usage, function reference, what is established, estimability, real phylogenies, open issues, glossary, and one page per experiment |
| Simulator | https://franciscorichter.github.io/emphasis/simulator.html:
forward simulation of every model and link in the browser; its agreement
with simulate_tree() is checked by
dev/simulator/agreement.R |
| Issues and bug reports | https://github.com/franciscorichter/emphasis/issues |
| Citation | Richter et al. (2020), A general class of species diversification models for phylogenetic trees, Statistica Neerlandica |
Toolchain: R ≥ 3.5, a C++ compiler, and the packages in
DESCRIPTION (Rcpp, RcppParallel
for TBB parallelism, nloptr, BH headers,
ape, DDD, progress;
mgcv for the optional survival GAM).
devtools::load_all() # compiles src/ and loads the package
devtools::document() # regenerates NAMESPACE and man/ from roxygen
devtools::test() # unit tests (tests/testthat)
devtools::check() # full R CMD checkR CMD build . # source tarballCI runs R CMD check on every push to main
and on pull requests through
.github/workflows/R-CMD-check.yaml (r-lib/actions): macOS
and Windows on R release, Linux on R release, devel and oldrel. The
suite and what it checks are under Validation.
R/
simulate.R simulate_tree(): forward / batch / conditional (augmentation); model + link resolution;
.expand_pars/.contract_pars between compact and full parameter layouts
inference.R estimate_rates() dispatcher (mcem / cem / gam), control defaults, compare_models()
emphasis.R .mcem_dynamic_fresh(): thinning-based MCEM
bdi.R conditional (BDI) augmentation sampler + .mcem_bdi() (see "Augmentation samplers")
covariate_path.R lineage_table(), covariate_path()
de.R cross-entropy method (CEM) and importance-sampling utilities
gam.R auto_bounds(), train_GAM(), predict_survival(), likelihood-surface GAM
survival.R survival_mc(), survival_surface(), and estimate_rates(cond = "surface")
pipeline.R emphasis_pipeline(): the bounds → GAM → CEM → MCEM chain, kept for compatibility
diagnostics.R diagnose_mcem/cem/gam + print methods
src/
augment_tree.cpp data augmentation: insert extinct / unsampled lineages (thinning proposal)
E_step.cpp parallel augmentation via TBB, IS weights
M_step.cpp NLopt Subplex maximisation of the weighted log-likelihood
div_tree.cpp forward simulation
loglik.cpp, mcem.cpp, rcpp_*.cpp, rsbplx.cpp, rinit.cpp
inst/include/ header-only model code (model.hpp, augment_tree.hpp, ed_covariate.hpp, general_tree.hpp, …)
man/ generated by roxygen; do not edit by hand
tests/testthat/ unit tests
site/ the docs site's sources and builder: build.py, template.html, style.css, src/*.md,
reference.R + reference/ (Rd fragments), public.py (the public filter)
*.html, covariates/, experiments/, assets/
the docs site as built (committed; GitHub Pages serves them); simulator.html is
hand-written; D.html is a redirect the build writes; self-hosted MathJax;
excluded from the package build
assets/simulator/emphasis-sim.js the simulator page's engine: a port of general_tree.hpp and ed_covariate.hpp
dev/ excluded from the package build
audit/ the as-implemented map of the estimator, its hypotheses, findings and reproduction scripts
simulator/ the check that the simulator page draws from the distribution simulate_tree() does
Exported functions: simulate_tree,
estimate_rates, compare_models,
covariate_path, lineage_table,
auto_bounds, diagnose_mcem,
survival_mc, survival_surface; and, kept for
compatibility, emphasis_pipeline, train_GAM,
predict_survival, diagnose_cem,
diagnose_gam.
The package ships no data. Inputs are an ape::phylo
object, a simulate_tree() result, or a numeric vector of
branching times. A tree that is not ultrametric (one with branching
times after the present) is refused with a message. The examples below
use simulated trees.
Species diversification is modelled as a stochastic birth-death process whose speciation and extinction rates depend on endogenous covariates, quantities generated by the diversification process itself:
\[ \lambda(s,t) = f\!\bigl(\beta_0,\; \beta_N N(t) + \beta_D D_s(t) + \beta_{ED}\,\mathrm{ED}_s(t) + \beta_K K_s(t)\bigr), \quad \mu(s,t) = f\!\bigl(\gamma_0,\; \gamma_N N(t) + \gamma_D D_s(t) + \gamma_{ED}\,\mathrm{ED}_s(t) + \gamma_K K_s(t)\bigr) \]
Two link functions \(f\) are available:
| Link | Formula | Properties |
|---|---|---|
| Exponential (default) | \(f(\beta_0, \eta) = \exp(\beta_0 + \eta)\) | Positive; unbounded; monotonic; \(\beta_0\), \(\gamma_0\) are log rates |
| Linear | \(f(\beta_0, \eta) = \max(0,\; \beta_0 + \eta)\) | Non-smooth at zero; monotonic; a rate driven below zero is clipped at zero |
The link is selected with link = "exponential" (the
default) or "linear". The linear link is supported.
Four lineage-level covariates are read on the complete tree, the first from its branching times and the others from its topology:
| Symbol | Name | Definition |
|---|---|---|
| \(N(t)\) | Diversity (tempo) | Number of lineages alive at time \(t\), a readout of the branching times alone |
| \(D_s(t)\) | Age-imbalance | \(D_s = I_s - M\): a lineage’s isolation time \(I_s\) (time since it last speciated) relative to the clade mean \(M\); centred, so \(\sum_s D_s = 0\) |
| \(\mathrm{ED}_s(t)\) | Evolutionary distinctiveness | Fair proportion on the complete tree: \(\sum_{b \in \mathrm{anc}(s,t)} \ell_b / n_b\) over the branches from the crown to \(s\), each divided by the alive lineages below it. Its last term is the pendant age \(I_s\); its sum over lineages is Faith’s phylogenetic diversity |
| \(K_s(t)\) | Ancestral splits | The number of speciation events on the path from the crown to \(s\) at time \(t\), hidden splits counted |
\(D\). Because \(\sum_s D_s = 0\), the age-imbalance is a within-clade signal, uncorrelated with diversity: \(N\) measures when the clade diversifies, \(D\) the age structure of which lineages do. The mean pendant age \(M = P/N\) is used only internally to centre \(D\) and is not a user covariate. A lineage’s pendant age restarts when it splits. Between events every pendant age and their mean grow at unit rate, so \(D_s\) is constant from one event to the next; the simulator, the likelihood’s compensator and its event terms all read that value, over the same alive multiset of pendant starts. See the biological meaning of D for what \(\beta_D\) tests (age-dependent speciation).
\(\mathrm{ED}\).
The per-lineage form of phylodiversity dependence. It is defined on the
complete tree, so hidden lineages count among a branch’s descendants and
contribute branches, and the augmentation supplies them; it needs the
tree’s topology (a phylo, not a branching-time vector). The
same routine computes it in the simulator and in the likelihood
(inst/include/ed_covariate.hpp). Its cost is quadratic in
the augmented tree: the per-segment tables over every alive
lineage are built once per tree and cached across the parameter values
an M-step tries. Because the augmentation builds the complete clade the
current parameters imply, an \(\mathrm{ED}\) fit is started from the
estimate of the nested model (ed from cr,
ned from dd) with the \(\mathrm{ED}\) coefficients at zero; from a
point far above the net rate the augmented trees grow by orders of
magnitude and the E-step’s clock runs out.
\(\mathrm{ED}^{c}\). \(\mathrm{ED}^{c}_s(t) = \mathrm{ED}_s(t) -
\overline{\mathrm{ED}}(t)\), the fair proportion minus its mean
over the lineages alive at \(t\).
Fair-proportion \(\mathrm{ED}\) is not
centred: its clade mean is Faith’s PD over \(N\), a clade-level quantity the \(N\) term already tracks, so in an
ned fit \(\beta_{ED}\)
carries a clade-level component and a within-clade contrast at once. The
centred covariate carries only the contrast; it sums to zero over the
alive lineages and is constant between events. At \(\beta_{ED} = \gamma_{ED} = 0\)
nedc is dd.
\(K\). Constant
between events, one more on both daughters at a split, hidden splits
counted, so like \(D\) and \(\mathrm{ED}\) it is defined on the complete
tree and needs the topology and the augmentation. Its compensator is the
segment length. It is the observable trace of what inherited-rate models
(BAMM, ClaDS) carry as a latent variable; over 6,433 lineages on 30
complete dd clades its rank correlation with each of \(N\), \(D\)
and \(\mathrm{ED}\) is at most 0.29 in
magnitude. The thinning proposal reads it per lineage; the conditional
proposal refuses it with a reason, since its mean field does not carry
\(K\).
Setting coefficients to zero recovers the model classes:
| Shortcut | Covariates | Parameters | Hypothesis |
|---|---|---|---|
"cr" |
none | \((\beta_0, \gamma_0)\) | Constant rates (null) |
"dd" |
\(N\) | \((\beta_0, \beta_N, \gamma_0, \gamma_N)\) | Diversity dependence (ecological limits) |
"d" |
\(D_s\) | \((\beta_0, \beta_D, \gamma_0, \gamma_D)\) | Age-dependent speciation |
"nd" |
\(N + D_s\) | \((\beta_0, \beta_N, \beta_D, \gamma_0, \gamma_N, \gamma_D)\) | Both |
"ed" |
\(\mathrm{ED}_s\) | \((\beta_0, \beta_{ED}, \gamma_0, \gamma_{ED})\) | Distinctiveness-dependent diversification |
"ned" |
\(N + \mathrm{ED}_s\) | \((\beta_0, \beta_N, \beta_{ED}, \gamma_0, \gamma_N, \gamma_{ED})\) | Distinctiveness over and above diversity |
"edc" |
\(\mathrm{ED}^{c}_s\) | \((\beta_0, \beta_{ED}, \gamma_0, \gamma_{ED})\) | The same, on the centred covariate |
"nedc" |
\(N + \mathrm{ED}^{c}_s\) | \((\beta_0, \beta_N, \beta_{ED}, \gamma_0, \gamma_N, \gamma_{ED})\) | Within-clade distinctiveness over and above diversity |
"k" |
\(K_s\) | \((\beta_0, \beta_K, \gamma_0, \gamma_K)\) | Inherited diversification: a speciation-rich ancestry speciates more (or less) |
"nk" |
\(N + K_s\) | \((\beta_0, \beta_N, \beta_K, \gamma_0, \gamma_N, \gamma_K)\) | The same over and above diversity |
Models can also be given as a formula: model = ~ N + D,
model = ~ N + ED, model = ~ N + EDc,
model = ~ N + K. ("ep" is an alias for
"d".) Internally a model is a five-slot vector
c(use_N, use_M, use_D, use_ED, use_K), where slot 4 is 1
for the raw covariate and 3 for the centred one, and the full parameter
vector has twelve slots, with the two \(\mathrm{ED}\) coefficients and then the two
\(K\) coefficients last, so that
eight-slot parameter vectors and three-slot model vectors keep their
meaning.
Reconstructed phylogenies record only extant lineages; extinct side branches are unobserved. The marginal likelihood integrates over all possible extinction histories \(z\):
\[ L_\theta(y_{\text{obs}}) = \int f_\theta(y_{\text{obs}}, z)\,dz \]
It is estimated by importance sampling: the observed tree is augmented with latent extinct lineages, and each augmented tree is weighted by \(f_\theta / q\).
Two Monte Carlo averages appear in an MCEM fit, and only one of them is unbiased.
The EM objective is an expectation of a log, \(Q(\theta)=E[\log f(y,z\mid\theta)]\), and
the M-step averages \(\log f\) directly
(src/M_step.cpp). Averaging logs to estimate a mean of logs
is unbiased.
The log-likelihood is not. \(\hat\ell=\log\big(\tfrac1N\sum_i w_i\big)\)
is a log of a mean, so by Jensen’s inequality \(E[\hat\ell]\le\log p(y\mid\theta)\): biased
down, never up, by an amount that grows as the effective sample falls.
AIC inherits it doubled, and it does not
cancel between models: the model that is harder to sample is penalised
for being harder to sample.
Every fit therefore records details$final_IS$ESS and
$gap, and compare_models() reports both and
warns when a ranking is not safe against them: when the
handicap of a lower-ESS model reaches the AIC gap it is losing by, or
when its draw is heavy enough that the handicap cannot be quantified. It
also warns when the model it chooses rests on fewer than 10 effective
draws. print() on a fit shows the effective sample and,
when it matters, how far below the truth the estimate sits.
gap is an indicator of size and not a bound. Against
log-normal weights at \(N=200\) it
overstates the true bias where that bias is negligible (ratio 4.9 at
\(n_{\text{eff}}=188\), absolute size
\(3\times10^{-5}\)) and understates it
where it is not (ratio 0.46 at \(n_{\text{eff}}=17\), 0.16 at \(n_{\text{eff}}=6\), where the true bias is
half a nat). The gap_heavy flag marks \(n_{\text{eff}}<N/10\), where the ratio
first falls below one half, and a heavy draw is treated as
unquantified.
The chosen model is not the lowest AIC. Among nested
models the richer one has the lower AIC whenever its two extra
parameters raise the log-likelihood by more than 2, which under the null
happens on about 13% of trees (the likelihood-ratio statistic is \(\chi^2_2\)), and more often with several
richer models offered beside the null. compare_models()
therefore reports chosen: the model with the fewest
parameters among those within margin = 2 AIC of the minimum
and, among those, the one with the lower AIC, whatever order the fits
are passed in; within_margin marks the rows that qualify,
and the full ranking is kept.
Three routes to a model.
estimate_rates() fits one named model.
estimate_rates(model = "all") fits the nested ladder
cr, dd, nd, ned,
nk (the last two need a phylo), each started
from the estimate of the model it nests and boxed by
auto_bounds() unless control$bounds names a
box per model, and returns the fits with their
compare_models() table and its chosen model;
control$models restricts the ladder.
covariate_path() selects and estimates at once, along one
dgLARS path (Augugliaro, Mineo & Wit 2013) over the covariates, on
the segment-level Poisson form of the log-linear point process
(lineage_table() is that form, one row per lineage and
segment), choosing a point by BIC or AIC for speciation and extinction
separately. Called on a phylo, it treats the tree as fully
observed; the augmented route, with the hidden lineages entering at
their importance weights, runs on the draws of an E-step.
What a fit records about itself. A fit carries
notes, each stating the number it rests on: a lineage-level
covariate on fewer than 50 tips; an effective sample below 10 at the
last E-step; turnover above 0.5 at the intercepts; under
model = "all" with both nd and
ned in the ladder, that the two read the same pendant edge;
clock when max_time cut a step short; and
mean_field when the surrogate proposal’s mean-field
iteration was relaxed below the full step (which the default does at
every sweep when \(\rho < 1\)), ran
under a damping the user set, or did not converge within
its sweep budget; the note names the setting and whether it was the
default. print() shows them.
Two proposals \(q\) are available,
selected with simulate_tree(method = ) for conditional
simulation and control = list(sampling = ) for MCEM:
| Sampler | Where | Scope | Property |
|---|---|---|---|
"bdi", the surrogate (default) |
R/bdi.R |
cr, dd and \(\mathrm{ED}\) models on the linear and
exponential links; \(D\) models on
request; any \(\rho \in (0,1]\) |
Draws from the exact conditional distribution under constant rates (ESS = sample size); under diversity dependence, \(D\) or \(\mathrm{ED}\) it is a mean-field proposal built on a self-consistent backward-forward iteration, and the weights carry the approximation |
"thinning"
(sampling = "dynamic_fresh") |
src/augment_tree.cpp |
Every model and link | Poisson-thinning proposal, C++ with TBB parallelism |
.bdi_supported(model_bin, link, rho) gates both entry
points, and .bdi_unsupported_reason() supplies the phrase
the fallback message carries. What runs on thinning:
control$sampling names "bdi". Under
the surrogate the average lineage has \(D =
0\), so the mean field is the dd one and \(\beta_D\) enters only the attachment of
hidden births, which is exact: each candidate parent is weighted by the
model’s own speciation rate at that instant. The pre-registered
condition for making it the \(D\)
default is not met.The surrogate’s mean field is a Picard iteration, and where its map
expands it diverges; control$damping under-relaxes the
sweep. By default the step is "auto" at complete sampling
(\(\rho = 1\)), which halves the
relaxation whenever the residual grows, and a fixed ⅛ when \(\rho < 1\). At the shipped sweep budget
(20 sweeps at \(\rho = 1\), 200 below)
a fixed ⅛ converges more diverging cases below \(\rho = 1\) (142 against 131 of 192 units at
\(\rho = 0.2\)) but none of 192 at
\(\rho = 1\), against 131 for
"auto", at 1.4 to 3.3 times the E-step time and with no
gain in effective sample (E10).
A mean field that converges only under relaxation leaves an effective
sample of about 1 either way. Every E-step’s mean field is in the trace
(mf_converged, mf_delta,
mf_sweeps, mf_omega, mf_budget),
and fit$notes$mean_field records the relaxation used.
The thinning proposal reads ED. For a model with
\(\mathrm{ED}\) the sampler draws the
missing births at the model’s total speciation rate over the lineages
alive on the segment, \(\Lambda(t) = \sum_s
\lambda_s(t)\) with each lineage at its own \(\mathrm{ED}_s(t)\), and attaches a birth to
lineage \(p\) with probability
proportional to \(\lambda_p(t)\); the
segment’s extinction rate reads the mean \(\mathrm{ED}\). The density mirrors it
through the ED table (Model::ed_proposal_seg_t,
Model::sampling_prob_ed), and at \(\beta_{ED} = \gamma_{ED} = 0\) it is the
mean-field density term for term. The mean-field thinning proposal,
\(N\lambda(N, M)\) with the ED term at
zero and uniform attachment, is reachable through
control = list(proposal = "meanfield"); the ED-aware one is
the default because it completes where the mean-field one returns zero
weight (E3).
The estimator is Monte Carlo EM. At each iteration the E-step draws hidden lineages from the proposal and weights them, and the M-step maximises the weighted complete-data log-likelihood over a box with Rowan’s Subplex (NLopt), the first simplex sized at a tenth of the box in every coordinate; the per-lineage tables are built once per E-step and cached across the values the M-step tries.
The box comes from the observed tree alone, by
auto_bounds(): a centre at the net rate the tip count and
crown age imply, a bisection along each axis toward the point where
forward simulation stops producing clades within a tenth and ten times
the observed size, each slope axis scaled by its covariate’s size on the
tree, and a second pass that tightens the box where the augmentation
fails at its corners. Its \(\lambda\)
ceiling has turnover headroom (the implied net rate divided by \(1 - 0.95\)) and the search probes the
constant-net-rate ray, so a high-turnover optimum is inside the box.
Two stopping rules: rel_change (default) stops once the
relative step, in units independent of the tree’s time scale, falls
below 1e-2 on three consecutive iterations; mc_error
measures the step against its own Monte Carlo standard error from five
batches, grows the draws by half whenever a step is inside the noise,
and stops once the draws reach their cap and the step has stayed inside
the noise for three iterations, returning the mean of the last three
iterates. rel_change compares the step with a fixed
tolerance rather than with the noise of the iterate, so on
dd fits at 200 draws it can run to its cap;
mc_error lands about 2% from a 400-iteration reference (E9).
Both are subject to max_iter and max_time, and
a fit reports which of the four ended it.
control$max_time bounds the E-step and the M-step: both
samplers read the clock between draws and inside a draw, the M-step’s
optimiser reads the same deadline before every evaluation of its
objective and keeps its best point only if it improves on the current
iterate, and the fit ends with stop_reason = "time_budget"
and a clock note. After the loop one more E-step runs at
the returned point, with max_time again, so that the
reported log-likelihood, effective sample and gap describe the estimate
and a fit returns within about twice max_time.
estimate_rates(method = "cem" | "gam") and
emphasis_pipeline() are further estimators (a cross-entropy
search, a likelihood surface fitted on a grid); nothing above depends on
them. survival_mc(), survival_surface(),
train_GAM() and predict_survival() are the
survival probabilities of the next section.
Reconstructed trees are conditioned on survival to the present. The conditioned log-likelihood is:
\[ \ell_{\text{cond}}(\theta) = \ell(\theta) - \log P_\theta(\text{survival}) \]
Survival is crown survival: both crown lineages leave a sampled
descendant at the present. The likelihood is not conditioned by default
(cond = NULL), and neither is the ladder. cond
takes one of:
"surface": \(\log P_\theta\) estimated by direct
simulation on local quadratic surfaces around the iterate, for any model
and link, under MCEM. The recommendation for the lineage-level
models.cr and dd, with
the exact value: under cr on the exponential link one crown
lineage leaves a sampled descendant with probability \(p = \rho d / (\rho\lambda + (\lambda(1-\rho) -
\mu)e^{-dT})\), \(d = \lambda -
\mu\), and \(P_\theta = p^2\);
the example in ?estimate_rates codes it. Under
dd it is the solution of the master equation on the clade
size. Neither function is in the package.survival_surface(),
held fixed through the fit.train_GAM(), which
simulates clades at Latin-hypercube points of the box and counts a clade
that reaches the lineage cap as surviving
(auto_bounds(train_surv_gam = TRUE) returns one), or an
nnet object. In E43,
GAMs trained on 2000, 5000 and 20 000 clades for each of the eight
lineage-level models had a 90th-percentile error of \(\log P(S)\) of 0.44 to 2.56, against the
0.05 its rule required, and the error did not fall with the training
size; the emulator is not accurate enough to condition on. Its accuracy
under cr and dd over the whole box is measured
in E36.\(P_\theta\) does not depend on the observed tree, so one function per model, crown age and \(\rho\) serves every fit of a cell.
The direct estimate. survival_mc()
draws clades forward to the crown age, clade \(i\) from seed seed + i, so
that two calls with the same seed draw the same clades at different
parameters (common random numbers). A clade that passes the lineage cap
counts as surviving, as in train_GAM(). Under \(\rho < 1\) a clade that reaches the
present with \(n_1\) and \(n_2\) extant descendants of its two crown
lineages contributes \((1-(1-\rho)^{n_1})(1-(1-\rho)^{n_2})\), the
probability that sampling each extant tip with probability \(\rho\) keeps one on both sides. The
estimate is \(\log((k + 1/2)/(n + 1))\)
over \(n\) clades of total contribution
\(k\), with the delta-method standard
error. Against the exact cr value it agrees within its
error at \(\rho = 1\) and \(\rho = 0.7\)
(tests/testthat/test-survival-surface.R). At a standard
error of 0.01 one estimate took E43 a median of 2 to 29 s of one core
under the lineage-level models, and the M-step evaluates
cond at every value it tries, which is why
"surface" does not call it there.
cond = "surface" runs four steps
(R/survival.R, .fit_surface_cond()):
init_pars, to the stopping
rule.survival_surface()): direct estimates at
\(2p + 1\) points, \(p\) the coefficients of a full quadratic in
the \(d\) parameters (31 points for
\(d = 4\), 57 for \(d = 6\)), the centre and a Latin hypercube
of a box of half-width 0.03 of the fit’s box per coordinate; the same
clades at every point, as many as give a standard error of 0.02 at the
centre; a quadratic fitted by weighted least squares, the tangent plane
outside the box, capped at \(\log P =
0\).cond, in segments of 25 iterations that
carry the draw count forward. The surface is rebuilt at the iterate
whenever the iterate leaves the inner half of its box; the run ends when
the stopping rule is met inside the inner half, or on
max_iter iterations or max_time.init_pars, the nested
start (the nested model’s estimate, lineage-level coefficients at zero),
and the run with the higher conditioned log-likelihood is kept.The reported loglik is the kept run’s unconditioned
log-likelihood minus a direct \(\log
P(S)\) at its estimate, at a standard error of 0.01;
loglik_var adds that error’s square. max_time
bounds the four steps together, and the direct estimate at the estimate
gets max_time again, as the final E-step does. The fit’s
conditioning records the surfaces built, the clades
simulated and the seconds spent, the direct \(\log P(S)\) at the estimate with its
standard error, the unconditioned log-likelihood, which start was kept,
whether step 4 ran, the last surface, and each run’s estimate, stop
reason, iterations and trace. In the pilot that chose the method for E46
(100 tips, cr and dd, where the exact
conditioned maximum is known), steps 1 to 3 reached a median deficit of
0.0011 nats (worst 0.13) against 0.0074 for MCEM with the exact factor
and 0.26 for the unconditioned fit; without step 4 a second, lower
maximum on the extinction edge was kept on 4 of 10 cr trees
at turnover 0.10. The surfaces cost under 0.3% of a
cr/dd fit, 0.4-1.8% under ned and
3-9% under nk.
Its settings are entries of control, read only under
cond = "surface":
| Entry | Default | What it sets |
|---|---|---|
surface_radius |
0.03 |
half-width of a surface’s box, as a share of the fit’s box per coordinate |
surface_se |
0.02 |
standard error of \(\log P\) at a surface’s centre that the clades per point are set for |
surface_final_se |
0.01 |
standard error of the direct \(\log P\) at the estimate |
surface_segment |
25 |
MCEM iterations per segment |
surface_max_lin |
max(1000, 20 * n_tips) |
lineage cap of a simulated clade |
surface_edge |
1e-3 |
share of the box span within which \(\gamma_0\) triggers step 4 |
surface_restart |
TRUE |
whether step 4 runs |
surface_max_clades |
40000 |
cap on the clades per surface point |
surface_final_max_clades |
1e5 |
cap on the clades of the direct estimate at the estimate |
surface_seed |
NULL |
seed of the clades; NULL draws one from R’s generator,
so set.seed() reproduces them |
emphasis_pipeline() conditions on the GAM that its
bounds stage trains; leaving that stage out, with
control$lower_bound, control$upper_bound and
optionally control$survival_gam, conditions on another
survival probability or on none.
When the phylogeny samples only a fraction \(\rho\) of the living species, set
rho in the control list (or
simulate_tree(rho = ...)). The augmentation then inserts
both extinct lineages and unsampled extant lineages, and the likelihood
gains the binomial sampling factor \(n_{\text{obs}}\log\rho +
n_{\text{unsamp}}\log(1-\rho)\). With rho = 1
(default) the tree is completely sampled.
Both proposals implement this. In the surrogate \(p(t)\) becomes the probability of leaving a
sampled descendant, which solves the same ODE with the terminal
condition \(p(t_p) = \rho\) and has the
closed form \(\rho d / (dE +
\rho\lambda(1-E))\) under constant rates; a missing lineage still
alive at the present is an unsampled extant tip, emitted with the
5e10 sentinel so that \(N(t)\) counts it. The value of
rho used is recorded in the fit. A rho outside
(0, 1] is an error, in R and in the compiled code alike.
library(emphasis); library(ape)
# a clade under diversity dependence: lambda = exp(b0 + bN N), mu = exp(g0)
sim <- simulate_tree(pars = c(log(0.6), -0.01, log(0.05), 0), max_t = 12,
model = "dd", link = "exponential")
phy <- sim$tes
# 1. one model
box <- auto_bounds(phy, model = "dd", link = "exponential", train_surv_gam = FALSE)
fit <- estimate_rates(phy, model = "dd", link = "exponential",
control = list(lower_bound = box$lower_bound,
upper_bound = box$upper_bound,
sample_size = 200))
fit # estimates, log-likelihood, effective sample, gap, stop reason, notes
# 2. the ladder: cr, dd, nd, ned, nk from nested starts, ranked by AIC
fits <- estimate_rates(phy, model = "all", link = "exponential",
control = list(sample_size = 200))
fits$comparison # AIC, ESS, gap, swing, within_margin; attrs "chosen", "ranking_at_risk"
fits$chosen # the fewest parameters within 2 AIC of the minimum
fits$fits$nd$pars
# 3. the path: which covariates the data ask for first (exponential link only)
cp <- covariate_path(phy, covariates = c("N", "D", "ED", "K"), criterion = "BIC")
cp$speciation$entry_order; cp$speciation$active
# the numbers the likelihood is built on
head(lineage_table(phy))
# the surrogate on a D model (the default there is thinning), thinning anywhere,
# and incomplete sampling
estimate_rates(phy, model = "nd", link = "exponential",
control = list(sampling = "bdi", sample_size = 200,
lower_bound = lb, upper_bound = ub))
estimate_rates(phy, model = "dd", control = list(sampling = "dynamic_fresh", ...))
estimate_rates(phy, model = "dd", control = list(rho = 0.8, ...))
# conditioned on survival through local surfaces of a direct estimate (any model)
fit_c <- estimate_rates(phy, model = "nd", cond = "surface", init_pars = nested_start,
control = list(lower_bound = lb, upper_bound = ub, sample_size = 200))
fit_c$conditioning[c("surfaces", "clades", "seconds", "logP", "logP_se", "start")]# Forward simulation: lambda(N) = 0.5 exp(-0.02 N), mu = 0.1
sim <- simulate_tree(
pars = c(log(0.5), -0.02, log(0.1), 0),
max_t = 10,
model = "dd",
link = "exponential" # the default; or "linear"
)
sim$status # "done", "extinct", or "too_large"
plot(sim$tes) # reconstructed (extant-only) tree
# Batch simulation (parallel)
pars_mat <- cbind(
beta_0 = log(runif(100, 0.3, 1.0)),
beta_N = runif(100, -0.05, -0.005),
gamma_0 = log(runif(100, 0.01, 0.3)),
gamma_N = rep(0, 100)
)
batch <- simulate_tree(
pars = pars_mat, max_t = 10,
model = "dd", link = "exponential",
max_tries = 1, num_threads = 4
)
# Conditional simulation: augment an observed tree with latent lineages
aug <- simulate_tree(sim$tes, pars = c(log(0.5), -0.02, log(0.1), 0), model = "dd",
n_trees = 50, method = "bdi") # or method = "thinning"
length(aug$trees); aug$log_qmax_tries is the number of forward draws in all, the
first included: the default 1 is a single draw,
unconditioned on survival, and an extinct clade is redrawn until one
survives or max_tries draws have been made.
# fits named one by one, compared with the Monte Carlo guard
fit_cr <- estimate_rates(phy, model = "cr", link = "exponential", control = ctrl_cr)
fit_dd <- estimate_rates(phy, model = "dd", link = "exponential", control = ctrl_dd)
compare_models(cr = fit_cr, dd = fit_dd) # AIC, ESS, gap, swing, within_margin; attr "chosen"| Function | Purpose |
|---|---|
simulate_tree() |
Forward simulation under every model and link; given a tree, hidden
lineages from either proposal (method = "bdi" /
"thinning") |
estimate_rates() |
Monte Carlo EM for one model, or the nested ladder with
model = "all"; the trace of every iteration in
details$mcem |
compare_models() |
AIC table over fits with effective sample, gap, swing and the
ranking-at-risk flag; chosen is the fewest parameters
within 2 AIC of the minimum, ties broken by the lower AIC |
covariate_path() |
The dgLARS path over the covariates, for speciation and extinction; entry order and the set BIC or AIC chooses |
lineage_table() |
One row per lineage and segment: the covariates and the event mark the likelihood reads |
auto_bounds() |
The parameter box from the observed tree, and the optional survival GAM |
survival_mc() |
\(\log P_\theta(S)\) by direct simulation, for any model, link and \(\rho\) |
survival_surface() |
A local quadratic surface of \(\log
P_\theta(S)\); predict() gives \(\log P\), \(P\) or its gradient |
diagnose_mcem() |
The trace of a fit: iterates, log-likelihood, effective sample, weights |
emphasis_pipeline(), train_GAM(),
predict_survival(), diagnose_cem(),
diagnose_gam() |
Further estimators and the survival emulator, exported for compatibility |
The full reference, with every argument, is on the docs site.
Each experiment is pre-registered with its rule before it runs, and has one page on the docs site (experiments). The conclusions that hold are collected on What is established and Estimability.
cr it meets the pre-set rule in
all 24 cells (25-800 tips; turnover 0.10, 0.25, 0.40; sampling fraction
1 and 0.8). Under dd with complete sampling it meets it in
11 of 14 cells: all nine at 25-100 tips, and two of five at 200-400
tips, where the three failing cells are those whose fits ended on the
eight-hour clock (E45).bd_ML also puts
extinction at zero on 37 of 40 trees (E32).cr (mean ratio 1.000 over 450 trees) and
dd (within 0.02) (V2).182055b the mc_error stopping rule
could not fire for d, nd, k and
nk fitted with the thinning proposal, so every such fit ran
to its iteration cap; fits under the default rel_change
rule were not affected.The rest of the earlier results are being remeasured by a validation series, V1-V12, each replacing named earlier experiments: proposal densities (V2), stopping rule and box (V3), exact agreement including 170 real trees (V4), recovery of the eight lineage-level models by tree size under the conditioned likelihood (V5, from E46), sampling fraction (V6), model choice (V7), covariate path (V8), misspecification (V9), extinction (V10), real phylogenies (V11) and cost (V12). Until V5 reports, no lineage-level coefficient has a measured reliable tree size.
| Repository | Holds |
|---|---|
emphasis-paper (private) |
the pre-registrations (00-design.md), drivers
(sim/), results (results/) and reports: the
validation series and its status
(report/validation-series.pdf), the audit of E1-E48
(report/experiment-history.pdf), the map from each E
experiment to the V experiment replacing it
(report/e-to-v-map.md), the results the papers must state
(report/paper-results-2026-10-31.md) |
emphasis-methods, emphasis-biology
(private) |
the two manuscripts, the method and the bird and mammal application, each with a supplement |
| experiments wiki (private) | one page per experiment, rendered into the docs site by
site/build.py |
The two manuscripts are being revised so that they state only results that hold and point each pending one to its V experiment; both still lack an abstract, and their titles are open.
The test suite has 42 files in tests/testthat/. At the
last full run 42,458 expectations passed with NOT_CRAN=true
and 2,225 in CRAN mode, where the Monte Carlo gates are skipped; a full
run takes a few minutes. CI runs R CMD check on macOS,
Windows and Linux (R release, devel and oldrel).
These files hold the likelihood and the proposals to a computation written independently of the C++ code, or to the simulator:
| File | What it checks |
|---|---|
test-d-bookkeeping.R |
the \(D\) compensator on a fully
observed tree against pendant ages read off the edge list, under both
links, and the d/nd complete-data
log-likelihood of simulated trees against one computed in R |
test-ed-complete.R |
the raw-\(\mathrm{ED}\)
complete-data log-likelihood of simulated ed trees against
one computed in R from the definition of fair proportion |
test-d-compensator.R |
the linear-link \(D\) compensator against quadrature over the instantaneous \(D\), and the pendant PD the thinning sampler reads against a replay of the tree |
test-ed.R |
\(\mathrm{ED}\) against a
fair-proportion computation on ape trees, its sum against
Faith’s PD, and ned at \(\beta_{ED} = 0\) against
dd |
test-k.R |
the simulator’s \(K\) against
ape’s node depth, and the \(K\) likelihood against the Poisson form of
the lineage table |
test-lineage-table.R |
the full model’s log-likelihood rebuilt from
lineage_table() against eval_logf, and the
point-process path against dglars |
test-de.R |
the importance-sampling estimate on a fixed tree against a direct calculation, and the effective-sample identities |
test-proposal-density.R |
\(E_q[f/q]\) against the brute-force marginal likelihood on a 2-tip tree, and the drawn lifetimes against the density \(\log q\) charges (KS test) |
test-parent-support.R |
the parent a hidden birth attaches to, and \(E_q[f/q]\) against the brute-force marginal likelihood on 3- and 4-tip trees |
test-bdi-cr.R, test-bdi-d.R |
the surrogate’s survival probability against Nee’s formula and its zero-variance weights under constant rates; \(E_q[f/q]\) under a \(D\) model against quadrature |
test-thinning-envelope.R |
that the thinning envelope dominates the proposal rate, against a closed form for the probability of no missing lineage |
test-survival-surface.R |
the direct survival estimate against the exact cr value
at \(\rho = 1\), \(0.7\) and \(0.4\); the surface against a direct
estimate at its centre and its gradient against its value; end-to-end
cond = "surface" fits under cr and
d |
test-jensen.R |
the gap against the true bias of a log of a Monte Carlo
mean, and the warnings of compare_models() |
Others hold the package’s behaviour: test-ultrametric.R
(refusal of a non-ultrametric tree), test-time-budget.R
(the clock inside the E-step and the M-step),
test-compare-chosen.R (the parsimonious choice and its
tie-break), test-bdi-damping.R (relaxation of the mean
field), test-mstep-start.R (the first simplex sized from
the box), test-rho.R (incomplete sampling),
test-reproducibility.R (seeding of the C++ samplers) and
the rest of tests/testthat/.
dev/audit/ holds an audit of the estimator:
01-map.md documents what the code computes (the weights,
fhat, the M-step objective, the MCEM/CEM/GAM stages, the
index, time and log conventions) and raises hypotheses, each tested
against a closed form, a property or a controlled comparison, with the
reproduction scripts in dev/audit/checks/;
02-findings.md carries the confirmed findings ranked by
severity and the refuted hypotheses with the reason each is not a
defect. The measurements of the estimator on simulated and real trees
are on the docs site’s experiments
pages.
https://franciscorichter.github.io/emphasis/ is
generated from its sources. One command rebuilds every page from three
inputs: the overview, usage and covariate texts in
site/src/*.md and site/src/covariates/*.md,
the function reference fragments in site/reference/ (made
from man/*.Rd by site/reference.R, run on a
machine with R), and a checkout of the experiments wiki, passed on the
command line:
python3 site/build.py --wiki <path to the experiments wiki checkout> # pandoc 3 requiredIt writes the root pages (index.html,
usage.html, reference.html,
established.html, estimability.html,
real-phylogenies.html, open-issues.html,
glossary.html), covariates/index.html with one
page per covariate (N, D, ED,
K), the redirect D.html,
experiments/index.html and one
experiments/E<n>.html per wiki page, copies the
wiki’s figures to assets/wiki/, numbers figures and
equations, builds each long page’s table of contents, and exits non-zero
on a broken internal link or a missing image. The built pages are
committed, and GitHub Pages serves the root of main
(.nojekyll), so a push to main is a deploy.
The style is one stylesheet, site/style.css.
Francisco Richter (author and maintainer), franciscorichter@gmail.com
Thijs Janzen and Hanno Hildenbrandt contributed to an earlier C++
implementation, parts of which remain in src/ and
inst/include/.
MIT. See LICENSE for details.