Skip to contents

Refits the model under each of several prior configurations and compares (1) the posterior of every parameter with a reference configuration and (2) out-of-sample predictive performance by PSIS-LOO or K-fold cross-validation (Vehtari et al., 2017), with the standard error of each difference. This is the global complement to prior_sensitivity(), which is local and needs no refitting.

Usage

sensitivity_analysis(
  data,
  formula,
  prior_configurations,
  family = gaussian(),
  chains = 4,
  iter = 2000,
  seed = 1234,
  reference = 1L,
  metric = c("elpd_loo", "p_loo", "looic"),
  shift_threshold = 0.5,
  k_threshold = NULL,
  cv = c("loo", "kfold", "none"),
  K = 10L,
  parallel = FALSE,
  keep_fits = TRUE,
  ...
)

Arguments

data

A data frame containing all variables in formula.

formula

A model formula, e.g. y ~ x1 + x2.

prior_configurations

A named list of prior configurations (each anything accepted by prior_config in fit_model_with_prior(), with NULL meaning rstanarm's defaults), or the bmbeR 1.x format: an unnamed list of list(label = , prior_config = ).

family

A family object, family function or family name: gaussian, binomial, poisson, Gamma, inverse.gaussian or neg_binomial_2.

chains

Number of Markov chains (at least 4 are recommended).

iter

Iterations per chain, including warm-up (half by default).

seed

Random seed passed to Stan, for reproducibility (a single non-negative whole number).

reference

Name or index of the reference configuration.

metric

Estimate reported per configuration: "elpd_loo", "p_loo" or "looic" (with cv = "kfold", the K-fold equivalents).

shift_threshold

Posterior-mean shift (in reference SDs) that is flagged.

k_threshold

Passed to loo::loo(): if not NULL, observations with Pareto \(\hat{k}\) above it are refitted exactly (slow).

cv

Cross-validation method: "loo" (PSIS-LOO, default), "kfold" (refits each model K times; slower but does not depend on Pareto \(\hat{k}\)) or "none" (compare posteriors only). rstanarm supports neither LOO nor K-fold for models fitted with weights, so weighted models need cv = "none".

K

Number of folds when cv = "kfold".

parallel

Fit the configurations in parallel with future.apply::future_lapply()? Choose the backend with future::plan(), e.g. future::plan("multisession", workers = 4). Each worker runs its chains according to cores (or options(mc.cores) in the worker), so keep workers x cores within the number of CPU cores. Data-masked arguments such as weights = w must refer to columns of data.

keep_fits

Keep the fitted models in the result?

...

Further arguments passed to fit_model_with_prior() (including on_nonconvergence and convergence_args) and from there to rstanarm::stan_glm().

Value

An object of class bmb_sensitivity with elements summary (posterior summaries by configuration and parameter), comparison (a data frame of paired ELPD differences, as in loo::loo_compare(), with one row per configuration and the column n_high_k; NULL when cv = "none"), metric (named vector of the chosen estimate), converged (named logical), loo (list of loo or kfold objects), cv, errors (named character vector of failed configurations) and, if keep_fits = TRUE, fits. Use as.data.frame() for the posterior summaries.

Details

A shift of shift_sd means the posterior mean under a configuration differs from the reference by that many reference posterior SDs; sd_ratio compares posterior SDs. Shifts at or above shift_threshold are flagged (a heuristic; choose it for your context).

LOO compares predictive performance, not prior plausibility: a prior can change the estimate of a parameter substantially while barely changing predictions, so read both parts of the output.

PSIS-LOO is unreliable when observations have high Pareto \(\hat{k}\); the number of such observations is reported for each configuration. Use k_threshold to refit those observations, or cv = "kfold" for K-fold cross-validation (Vehtari et al., 2017).

All configurations are validated before any model is fitted. If fitting one configuration fails, the error is recorded in errors and the others are still compared; the reference configuration must succeed.

References

Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4

Examples

# \donttest{
data(kidiq, package = "rstanarm")
sens <- sensitivity_analysis(
  kidiq, kid_score ~ mom_iq,
  prior_configurations = list(
    default = NULL,
    tight   = prior_config(slope = prior_spec("normal", 0, 0.1))
  ),
  chains = 2, iter = 1000, refresh = 0
)
#> Fitting configuration 1/2: 'default'
#> Fitting configuration 2/2: 'tight'
sens
#> <bmb_sensitivity> 2 prior configurations (reference: 'default')
#> 
#> Posterior means by configuration (shift in reference posterior SDs):
#>             default tight         
#> (Intercept) 25.9    41.7 (+2.57!) 
#> mom_iq      0.609   0.451 (-2.57!)
#> sigma       18.3    18.4 (+0.21)  
#>   ! = shift of at least 0.5 posterior SDs
#> 
#> Predictive comparison (PSIS-LOO; differences relative to the best):
#>         elpd_diff se_diff elpd_loo se_elpd_loo p_loo n_high_k
#> default       0.0     0.0  -1878.6        14.5   2.9        0
#> tight        -3.4     2.6  -1881.9        14.5   2.6        0
#>   |elpd_diff| smaller than ~2 se_diff is not a meaningful difference.
plot(sens)

# }