Compute approximate leave-one-out cross-validation (LOO-CV) using Pareto smoothed importance sampling (PSIS) for brma model objects and store the result in the object.
Usage
# S3 method for class 'brma'
add_loo(object, unit = "estimate", r_eff = NULL, parallel = FALSE, ...)Arguments
- object
a brma model object.
- unit
output/deletion unit.
"estimate"computes one contribution per effect-size estimate."cluster"computes one contribution per cluster and is available only for multilevel models fitted through the specialized clustered interface.- r_eff
optional vector of relative effective sample sizes. If not provided, it is computed from the log-likelihood values.
- parallel
Logical. If
TRUE,loo::relative_eff()andloo::loo()useRoBMA.get_option("max_cores"). Log-likelihood construction is unchanged. IfFALSE, those computations use one core.- ...
additional arguments (currently ignored).
Details
With unit = "estimate", LOO-CV is computed with one contribution per
effect-size estimate. For binomial and Poisson models, each pair of counts
(ai/ci or x1i/x2i) that defines a single effect size estimate is treated as
one contribution.
Estimate-unit GLMM log-likelihoods integrate the nuisance base rate or
log-rate over its prior: point priors and untruncated beta (base rate) and
normal (log-rate) priors adaptively, other priors on a refined prior-CDF
grid; a uniform(0, 1) base-rate prior is a Beta(1, 1) prior and is integrated
adaptively. The grid is accepted only when its log-likelihood changes by less
than 1e-6 between refinements. Any nuisance prior without an adaptive
route (other than an untruncated beta base rate or an untruncated normal
log-rate; truncated ones take the grid too) can fail to reach this,
especially with few events or very large arms, and then stops with an error of
class RoBMA_glmm_grid_nonconvergence (also of class
RoBMA_glmm_numerical_error); gamma, lognormal, and inverse-gamma
log-rate priors and lognormal base-rate priors are examples. A beta base-rate
prior or a normal log-rate prior avoids it. When the likelihood is computed
automatically after fitting (the autocompute.loo and
autocompute.waic options), such a failure gives a warning and the fit
is returned without LOO/WAIC; add_loo() then gives the same error.
With unit = "cluster", LOO-CV is computed with one joint contribution
per cluster. For normal models without selection, weighted or not, this uses
the closed-form Gaussian integral over the held-out cluster effect, with the
likelihood weights applied as powers of the row densities.
Cluster-unit binomial and Poisson GLMM log-likelihoods are unavailable until
certified nested adaptive quadrature is implemented; use
unit = "estimate" for GLMMs.
Cluster-unit contributions are also unavailable for brma.mv() models,
whose random-formula deletion semantics still need a dedicated design; use
unit = "estimate" there.
Selection-model deletion conditions the Gaussian law on retained outcomes and on the context declared by the fitted selection model. It preserves the original publication event and selection standard errors. Product weights on retained outcomes cancel; best selection continues to depend on the best p-value among both retained and deleted outcomes. This finite-vector deletion score differs from prediction for an entirely new publication event.
For Gaussian multilevel and known-V brma.mv() models,
estimate-unit LOO integrates Gaussian local effects and uses
\(p(y_i \mid y_{-i}, \theta)\) within each fitted dependency block. Thus
deleting one estimate retains the other estimates in its cluster or random-
effect block; it does not represent prediction for a wholly new group.
Gaussian brma(..., cluster = ...) models with non-unit likelihood
weights also integrate the cluster effect. Their score integrates the
entire powered row density against the cluster-effect distribution given
the retained weighted observations, including the power-dependent normalizing
constant. It is not the power of an unweighted marginal density. Earlier
stored weighted estimate-unit LOO results must be recomputed with
add_loo(); the fitted posterior does not need refitting for this
post-fit calculation. Selection and GLMM models retain their supported routes.
Numerically unresolved weighted Gaussian retained moments stop with
RoBMA_weighted_gaussian_unavailable, a
RoBMA_numerical_unavailable condition.
For brma.mv() models with known random-effect group covariance
R, the known R matrix and fitted random-effect covariance enter
the marginal \(ZGZ'\) covariance through BayesTools' compiled random-effect
metadata. The same calculation applies to every supported random structure.
Estimate-unit deletion retains the fitted grouping and known-V
dependency structure; it does not represent deletion of an entire new random
group. Models with no local effects, sampled local effects, marginalized
local effects, or a mixture of sampled and marginalized effects remain
comparable when their data hash, deletion unit, retained context, and
likelihood target agree. The sampled/marginalized labels are retained as
provenance rather than comparison keys.
The PSIS object is essential for model comparison via
loo_compare and is automatically saved in the loo result.
RoBMA stores target metadata so comparisons can reject mismatched data,
unit, or retained-context targets.
Each finite log-likelihood column observed to be constant across posterior
draws uses exact uniform importance ratios. Such columns bypass relative-ESS
and generalized-Pareto fitting, including when other columns vary. Their
Pareto-k diagnostic is recorded as the sentinel zero, meaning constant
ratios with no tail instability rather than an estimated Pareto shape.
Finite varying columns are centered by a column-specific constant before
relative-ESS and PSIS evaluation. The offset is restored exactly afterward,
preserving the LOO target while avoiding probability-scale underflow.
Important for model comparison: When comparing models via
loo_compare, the selection is based on expected
predictive performance for the documented deletion target, not in-sample
fit. For dependent estimates, deleting one estimate may retain other
observations from its fitted dependency block; this is not new-group
prediction.