Plot the interval calibration of a Bayesian cross-validation
Source:R/plotting-diagnostics.R
plot_calibration.Rdcv_bayes() scores every fold's posterior predictive intervals
at each of coverage_levels: the share of held-out observations the
interval of that nominal level contained. This draws observed coverage
against nominal with the diagonal, one point per level for the
fold-weighted pooled value (from predictive_coverage) and one faint
point per fold, so you can see systematic over-confidence (points below
the line) or intervals wider than they need to be (above it). Three levels is a thin
curve; pass coverage_levels = seq(0.1, 0.9, by = 0.1) to
cv_bayes() for a full one. Each level is drawn at the nominal value
cv_bayes() records in coverage_levels, so whatever was
computed is drawn where it belongs (0.975 at 0.975, not rounded).
Arguments
- cv
The list returned by
cv_bayes(), or acompare_models_cv()result that ran the Bayesian backend (its$bayes_cvis used).- ...
Not used.
See also
cv_bayes(), which makes the result.
Other cross-validation:
compare_models_cv(),
cv_bayes(),
cv_gwr(),
cv_rf(),
cv_spatial(),
plot_cv_metrics()
Examples
if (FALSE) { # \dontrun{
# Needs brms and a Stan toolchain, and takes minutes: run it, do not check it.
library(sf)
set.seed(1)
n <- 80
dat <- st_as_sf(
data.frame(x = 5e5 + runif(n, 0, 1000), y = 5e6 + runif(n, 0, 1000),
a = rnorm(n)),
coords = c("x", "y"), crs = 32632)
dat$z <- 2 * dat$a + rnorm(n)
# Nine coverage levels give a curve rather than three points; chains and
# iterations are kept small to keep this to a few minutes, so the intervals
# will be rough.
cv <- cv_bayes(dat, "z", "a", k = 3, coverage_levels = seq(0.1, 0.9, 0.1),
fit_args = list(chains = 2, iter = 1000))
plot_calibration(cv)
} # }