Sensitivity analysis across alternative priors
Source:R/model_sensitivity.R
sensitivity_analysis.RdRefits the model under each of several prior configurations and compares
(1) the posterior of every parameter with a reference configuration and
(2) out-of-sample predictive performance by PSIS-LOO or K-fold
cross-validation (Vehtari et al., 2017), with the standard error of each
difference. This is the global complement to prior_sensitivity(), which
is local and needs no refitting.
Usage
sensitivity_analysis(
data,
formula,
prior_configurations,
family = gaussian(),
chains = 4,
iter = 2000,
seed = 1234,
reference = 1L,
metric = c("elpd_loo", "p_loo", "looic"),
shift_threshold = 0.5,
k_threshold = NULL,
cv = c("loo", "kfold", "none"),
K = 10L,
parallel = FALSE,
keep_fits = TRUE,
...
)Arguments
- data
A data frame containing all variables in
formula.- formula
A model formula, e.g.
y ~ x1 + x2.- prior_configurations
A named list of prior configurations (each anything accepted by
prior_configinfit_model_with_prior(), withNULLmeaning rstanarm's defaults), or the bmbeR 1.x format: an unnamed list oflist(label = , prior_config = ).- family
A family object, family function or family name:
gaussian,binomial,poisson,Gamma,inverse.gaussianorneg_binomial_2.- chains
Number of Markov chains (at least 4 are recommended).
- iter
Iterations per chain, including warm-up (half by default).
- seed
Random seed passed to Stan, for reproducibility (a single non-negative whole number).
- reference
Name or index of the reference configuration.
- metric
Estimate reported per configuration:
"elpd_loo","p_loo"or"looic"(withcv = "kfold", the K-fold equivalents).- shift_threshold
Posterior-mean shift (in reference SDs) that is flagged.
- k_threshold
Passed to
loo::loo(): if notNULL, observations with Pareto \(\hat{k}\) above it are refitted exactly (slow).- cv
Cross-validation method:
"loo"(PSIS-LOO, default),"kfold"(refits each modelKtimes; slower but does not depend on Pareto \(\hat{k}\)) or"none"(compare posteriors only). rstanarm supports neither LOO nor K-fold for models fitted withweights, so weighted models needcv = "none".- K
Number of folds when
cv = "kfold".- parallel
Fit the configurations in parallel with
future.apply::future_lapply()? Choose the backend withfuture::plan(), e.g.future::plan("multisession", workers = 4). Each worker runs its chains according tocores(oroptions(mc.cores)in the worker), so keep workers x cores within the number of CPU cores. Data-masked arguments such asweights = wmust refer to columns ofdata.- keep_fits
Keep the fitted models in the result?
- ...
Further arguments passed to
fit_model_with_prior()(includingon_nonconvergenceandconvergence_args) and from there torstanarm::stan_glm().
Value
An object of class bmb_sensitivity with elements summary
(posterior summaries by configuration and parameter), comparison (a
data frame of paired ELPD differences, as in loo::loo_compare(), with
one row per configuration and the column n_high_k; NULL when
cv = "none"), metric
(named vector of the chosen estimate), converged (named logical), loo
(list of loo or kfold objects), cv, errors (named character vector of
failed configurations) and, if keep_fits = TRUE, fits. Use
as.data.frame() for the posterior summaries.
Details
A shift of shift_sd means the posterior mean under a configuration
differs from the reference by that many reference posterior SDs;
sd_ratio compares posterior SDs. Shifts at or above shift_threshold
are flagged (a heuristic; choose it for your context).
LOO compares predictive performance, not prior plausibility: a prior can change the estimate of a parameter substantially while barely changing predictions, so read both parts of the output.
PSIS-LOO is unreliable when observations have high Pareto \(\hat{k}\);
the number of such observations is reported for each configuration. Use
k_threshold to refit those observations, or cv = "kfold" for K-fold
cross-validation (Vehtari et al., 2017).
All configurations are validated before any model is fitted. If fitting
one configuration fails, the error is recorded in errors and the others
are still compared; the reference configuration must succeed.
References
Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4
Examples
# \donttest{
data(kidiq, package = "rstanarm")
sens <- sensitivity_analysis(
kidiq, kid_score ~ mom_iq,
prior_configurations = list(
default = NULL,
tight = prior_config(slope = prior_spec("normal", 0, 0.1))
),
chains = 2, iter = 1000, refresh = 0
)
#> Fitting configuration 1/2: 'default'
#> Fitting configuration 2/2: 'tight'
sens
#> <bmb_sensitivity> 2 prior configurations (reference: 'default')
#>
#> Posterior means by configuration (shift in reference posterior SDs):
#> default tight
#> (Intercept) 25.9 41.7 (+2.57!)
#> mom_iq 0.609 0.451 (-2.57!)
#> sigma 18.3 18.4 (+0.21)
#> ! = shift of at least 0.5 posterior SDs
#>
#> Predictive comparison (PSIS-LOO; differences relative to the best):
#> elpd_diff se_diff elpd_loo se_elpd_loo p_loo n_high_k
#> default 0.0 0.0 -1878.6 14.5 2.9 0
#> tight -3.4 2.6 -1881.9 14.5 2.6 0
#> |elpd_diff| smaller than ~2 se_diff is not a meaningful difference.
plot(sens)
# }