Compares the held-out expected log predictive density (ELPD) of several
models evaluated with evaluate_model_performance() on the same test
observations. Differences are paired by observation, and their standard
errors are computed from the per-observation differences, as in
loo::loo_compare(). It stops if the models were not scored on the same
test observations.
Value
A data frame sorted from best to worst with columns model,
elpd, elpd_diff (relative to the best model) and se_diff.
Examples
# \donttest{
data(wells, package = "rstanarm")
wells$dist100 <- wells$dist / 100
set.seed(1)
test <- sample(nrow(wells), 1000)
fit1 <- rstanarm::stan_glm(switch ~ dist100, family = binomial(),
data = wells[-test, ], refresh = 0)
fit2 <- rstanarm::stan_glm(switch ~ dist100 + arsenic, family = binomial(),
data = wells[-test, ], refresh = 0)
compare_performance(
distance = evaluate_model_performance(fit1, wells[test, ]),
distance_arsenic = evaluate_model_performance(fit2, wells[test, ])
)
#> model elpd elpd_diff se_diff
#> 1 distance_arsenic -644.7463 0.00000 0.000000
#> 2 distance -671.6709 -26.92464 6.313672
# }