Evaluate predictive performance on test data
Source:R/model_evaluation.R
evaluate_model_performance.RdScores a fitted model's predictions for new data using the full posterior predictive distribution. Proper scoring rules (Gneiting & Raftery, 2007) reward accuracy and calibrated uncertainty together; they are reported alongside the familiar point-prediction metrics.
Usage
evaluate_model_performance(
fit,
data_test,
formula = NULL,
threshold = 0.5,
analysis_type = c("auto", "regression", "classification"),
prob = 0.9,
ndraws = NULL,
seed = 1234,
offset = NULL
)Arguments
- fit
A
stanregobject.- data_test
A data frame of test data containing the model's variables. Rows with missing values in those variables are dropped.
- formula
Optional; retained for compatibility with bmbeR 1.x. It must be the model's formula.
- threshold
Probability threshold for classification.
- analysis_type
"auto"(classification for binary outcomes of a binomial model, regression otherwise),"regression"or"classification".- prob
Probability mass of the predictive intervals used for
Coverage.- ndraws
Number of posterior draws to use (default: all).
- seed
Seed for posterior predictive simulation.
- offset
Offset for the test data, on the link scale, when the model was fitted with the
offsetargument ofrstanarm::stan_glm().
Value
An object of class bmb_performance: a named list of metrics
(see Details) with attributes type, n, prob, threshold and
elpd_pointwise (the per-observation ELPD used by
compare_performance()). Access metrics with $, e.g. perf$RMSE, or
convert with as.data.frame().
Details
Always reported:
ELPD: expected log predictive density on the test data, the sum over observations of \(\log \frac{1}{S}\sum_s p(y_i \mid \theta_s)\) (higher is better), with its standard errorELPD_SEand per-observation meanELPD_mean. This is the out-of-sample analogue ofelpd_loo(Vehtari et al., 2017).
Regression (continuous or count outcomes):
RMSE,MAEof the posterior mean prediction (on the scale of the model's response, e.g.log(y)forlog(y) ~ x).CRPS: continuous ranked probability score of the posterior predictive distribution (lower is better), estimated from draws.CoverageandInterval_Width: share of test outcomes inside the centralprobposterior predictive interval, and its mean width. For a well-calibrated model coverage should be close toprob.
Classification (binary outcomes of a binomial model):
Accuracy,Precision,Recall,F1_Scoreafter classifying the posterior mean probability atthreshold.Brier: mean squared error of the predicted probabilities (lower is better; Brier, 1950).AUC: area under the ROC curve.
In bmbeR 1.x, classification thresholded predict() output, which for
rstanarm is on the link (log-odds) scale; probabilities are now used.
Offsets. Offsets written in the formula (e.g.
y ~ x + offset(log(exposure))) are computed from data_test. If the
model was fitted with the offset argument instead, supply the test-data
offset with offset.
R's random number stream is left unchanged.
References
Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. JASA, 102(477), 359–378. doi:10.1198/016214506000001437
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3. doi:10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2
Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4
Examples
# \donttest{
data(wells, package = "rstanarm")
wells$dist100 <- wells$dist / 100
set.seed(1)
test <- sample(nrow(wells), 500)
fit <- rstanarm::stan_glm(switch ~ dist100 + arsenic, family = binomial(),
data = wells[-test, ], chains = 2, iter = 1000,
refresh = 0)
evaluate_model_performance(fit, wells[test, ])
#> <bmb_performance> classification, 500 test observations
#> ELPD (log score, higher = better) -325
#> SE of ELPD 6.34
#> ELPD per observation -0.651
#> Accuracy (threshold 0.5) 0.616
#> Precision 0.628
#> Recall 0.846
#> F1 score 0.721
#> Brier score (lower = better) 0.229
#> AUC 0.634
# }