You supply fit_fn(train_sf), which fits on one fold's training rows and
returns a spatial_fit; cv_spatial() builds or takes the folds, calls
fit_fn once per fold, predicts the held-out rows and returns pooled and
per-fold scores. cv_rf() is a wrapper over it.
Usage
cv_spatial(
data_sf,
response_var,
predictor_vars,
fit_fn,
folds = NULL,
k = 5,
seed = 123,
boundary = NULL,
pointize = "auto",
predict_args = list(),
fold_info_fn = NULL,
p = NULL,
block_size = NULL,
auto_range = FALSE,
parallel = FALSE,
metrics = NULL,
.caller = "cv_spatial"
)Arguments
- data_sf
An sf layer with the response and the predictors.
- response_var
Name of the response column.
- predictor_vars
Character vector of predictor column names.
- fit_fn
A function of one argument, the training slice of
data_sf, returning aspatial_fitbuilt withnew_spatial_fit(). It is called once per fold on the training rows only, so anything done inside it (scaling, tuning, an inner variable sweep) is already nested and leak-free. Thesubclassit stamps must have apredict.<subclass>()method registered, since that is how each fold is scored. The fit modelsresponse_var(itsresponse_varis the call's): a fold whose fit models another column fails, with the two names infold_status, since its predictions are scored againstresponse_var.- folds
Optional fold definitions, in one of three shapes: a
make_folds()result; a bare list oflist(train =, test =)pairs of..row_idvalues; or a vector of fold labels, one per row, which becomes leave-that-label-out splits. The label vector is how folds built by another package are used here:blockCV::cv_spatial()returns one as$folds_ids, while its$folds_listholds two unnamed vectors per fold and is refused by name. Train and test must be disjoint; a fold that trains on its own test rows is not a cross-validation split and is refused with an error. IDs naming no row of the prepared data are dropped with a logged count (expected when rows were removed for missing values; a sign the folds came from other data when they were not). WhenNULL, the folds aremake_folds(method = "block_kfold"), built withk,seed,boundary,block_sizeandauto_range. Withfolds = NULLthe score estimates the error at the distance the block grid built here puts between the held-out points and their training data, which overstates the error of a map where the samples cover the area it predicts over. The excess is how much the model's error grows from the map's distances to the data to the block grid's, and is small for a model whose error changes little with the distance from the data.fold_separation()withprediction_pointscompares the held-out distances with those of the locations the model will predict at, andmake_folds(method = "knndm", prediction_points = )builds folds matched to them.- k
Number of folds to build when
foldsisNULL; not read whenfoldsis supplied. Default 5.- seed
Seed for building the folds when
foldsisNULL, and for the one seed drawn per fold beforefit_fnruns, so a learner that draws random numbers gives the same scores on every run, sequential or parallel. With a seed, the caller's random stream is restored afterwards. Default 123;NULLdraws the per-fold seeds from the session's RNG state, and advances it.- boundary
Optional polygon layer (sf or sfc). Data without a CRS are aligned to it, and the blocks built when
foldsisNULLare clipped to it (seemake_folds()and spatialkit-geometry).- pointize
How a feature that is not a point is reduced to one before fitting: a mode of
coerce_to_points(). Default"auto".- predict_args
A list of extra arguments for
predict()on each fold's fit.- fold_info_fn
Optional
function(fit, test_sf, y, yhat)returning a named list (or a named vector) of per-fold extras (a bandwidth, a tuning value, anything read off the fitted object), added as columns offold_metrics. It sees the fit and the held-out layer, whichmetricsdoes not; it is applied per fold only, and its values are not pooled. An element..per_rowthat is a data frame with one row per held-out observation is spliced intopredictionsinstead (NAin the rows of a fold that returned none; one of the wrong length is dropped and logged); its columns must be named, once, and not reuse a columnpredictionsalready has (..row_id,fold,y,yhat,y_train_mean). Every other element must be named, hold one value, and not reuse a columnfold_metricsalready has; anything else is an error. Afold_info_fnthat throws on a fold is logged, its columns areNAfor that fold, andfold_status$messagesays so; the fold is kept.- p
Number of predictors for the per-fold adjusted \(R^2\), or
NULLto skip it. It applies only to models with one global parameter count; passNULLfor models with spatially varying coefficients, such as GWR.- block_size
Optional minimum block edge length, in the units of the CRS the folds are built in, for the blocks built when
foldsisNULL; seemake_folds(). It is a size you choose and is compared with the autocorrelation range only when one can be estimated, in which case a smaller size is warned about. Beside suppliedfoldsit has nothing to size and raises a warning of classspatialkit_fold_args_ignored.- auto_range
Logical, default
FALSE.TRUEwithfolds = NULLestimates the autocorrelation range and uses it as the minimum block size.TRUEbeside suppliedfoldsraises the same warning asblock_size.- parallel
Logical or positive integer.
TRUEdetects the number of cores and fits the folds in parallel withparallel::mclapply()(macOS and Linux; sequential on Windows); an integer above 1 uses that many cores. DefaultFALSE(sequential). A learner that runs OpenMP code (a function of GWmodel, or an xgboost built with GNU libgomp) can hang the forked workers once it has run in the session; keep such afit_fnsequential.- metrics
Optional scoring function of your own; see the section "Your own metrics". Default
NULL: the built-in metrics only.- .caller
For package wrappers; do not set it.
Value
A list with:
overallOne row of metrics pooled over the held-out rows of every fold that produced predictions.
fold_metricsOne row per fold that produced predictions:
fold,n_train,n_test,n_pred, the metrics, and the columns offold_info_fnandmetrics.predictionsOne row per held-out row scored:
..row_id,fold,y,yhatandy_train_mean, the mean response of that fold's training rows.foldsThe splits as run, each with its
fold_id.n_folds_attempted,n_folds_succeededThe folds supplied (or built) and the folds scored. A
fit_fnthat fails on every fold otherwise looks like a run that scoredNA, so compare the two before readingoverall.fold_statusA data frame with one row per fold supplied:
fold,statusandmessage.statusis one of"ok":messageis empty unlessfold_info_fnfailed on the fold;"error": the fit or itspredict()threw, andmessageis the error text;"skipped": nothing to score (too few matched rows, a prediction of the wrong length, or no finite pair of observed and predicted values);"dropped": an empty test set, or fewer than two training rows once incomplete rows were removed, so the fold never reached the fitter;"worker_error": a parallel worker died, for example killed for lack of memory.
Every fold missing from
fold_metricshas its reason here, after the console output of a long run is gone.orphan_rowsThe
..row_ids of rows in the data that no fold names: they enter no training set and are never scored. It is empty unless the folds were built on a different or subsetted layer.n_unknown_idsThe number of distinct row IDs the folds name that the data do not have (expected when rows were removed for missing values). Each such row is named by every fold, once as a test row and once in each other fold's training set, and this counts the row, not the mentions.
n_droppedThe number of rows
prep_model_data()removed for missing or non-finite values or a bad geometry before any fold was fitted.
A failure costs that fold only: with parallel = TRUE each fold runs in
its own worker, and a sequential run catches the error per fold. An
error that stops a sequential run (a metrics or fold_info_fn return
value of the wrong shape) stops a parallel one too, naming the fold. When some folds, but not all, end as "error", "skipped" or
"worker_error", the function warns, naming them and how many rows
overall covers: it is pooled over the folds that produced predictions,
and the fold that fails is often the hardest to predict, so it may
flatter the model. A "dropped" fold has its own warning.
The fold column of fold_metrics, predictions and fold_status
carries the fold's index in the folds object that was supplied, so it
lines up with make_folds()$assignment$fold even when some folds were
unusable and dropped. Splits that already carry a fold_id, as the
folds of a cv_*() result do, keep it: handing one run's folds to
another labels each fold as the first run and fold_separation() do, a
dropped fold's gap included.
R2 is the out-of-sample \(R^2\): the total sum of squares is
taken about the mean of the training rows (in overall, each held-out
row about its own fold's training mean), the null prediction available
when the fold is predicted, and not about the held-out rows' own mean,
a null model that would know the test data. On spatial blocks the two
can differ widely; R2 is below 0 when the model predicts worse than
the training mean. model_metrics(newdata = ) uses the same baseline.
overall$Adj_R2 is always NA: the pooled out-of-sample predictions
come from k separately fitted models and have no single parameter
count to adjust for. The per-fold fold_metrics$Adj_R2 carries the
adjusted value when p is supplied, and is NA otherwise. A fit_fn
that returns a gwr_fit of a response at a large offset gives
predictions the columns y_centre, yhat_centred and
y_train_mean_centred (see cv_gwr()).
Name collision with blockCV
The blockCV package exports a function of the same name that does
the opposite job: blockCV::cv_spatial() builds spatial folds, where this
function runs a cross-validation over folds it is given. With both
packages attached, whichever was attached last masks the other;
spatialkit::cv_spatial() always resolves to this one. The two work
together: blockCV::cv_spatial() returns its fold assignment as
$folds_ids, a vector of fold labels, and that vector is accepted as the
folds argument here and in every other cv_*() function.
Your own metrics
The built-in columns (RMSE, MAE, MAPE, SMAPE, R2, Adj_R2) are
the Gaussian regression set; see "Which metrics survive a non-Gaussian
response" in model_metrics() for which of them hold up on a count or a
bounded response. metrics scores what they cannot: a
function(y, yhat) that returns a named numeric vector (a named list of
scalars, or a one-row data frame, also serve), for example a Poisson
deviance, a log score on a probability, or a loss with your own weights.
It is applied twice, as the built-in metrics are: once per fold, to that
fold's held-out rows, so each name becomes a column of fold_metrics; and
once to the pooled out-of-sample predictions of every fold, so each name
becomes a column of overall. Only the pairs the built-in metrics use
reach the function (both y and yhat finite), so its columns describe
the same rows as RMSE, and n_pred counts them.
The contract: every element named, names unique and not one of the
built-in column names, one number per name. The built-in names include
the per-fold extras of the backend or of your fold_info_fn (bandwidth
for cv_gwr(); CRPS, coverage_*, gp_k, gp_n_basis and n_draws
for cv_bayes()), and mean_CRPS, which compare_models_cv() writes.
Anything else is an error, since a scoring function that returns the wrong
shape is a mistake to report, not a fold to skip. A function that throws
on a fold is logged and its columns are NA for that fold (and for
overall, if it throws on the pooled predictions); a fold is never
dropped for it. When no fold produced a prediction the empty
fold_metrics frame still carries the function's columns, typed, provided
the function can be called on zero-length input.
fold_info_fn is the per-fold half of the same mechanism, with access to
the fitted object and the held-out layer; metrics sees only the two
vectors but is also pooled. compare_models_cv() hands one metrics to
every backend, so the columns are comparable across the rows of its
overall.
Warnings a learner raises in every fold
A warning from a fold's fit or from its predict() reaches the caller as
it is raised, so a learner that warns in every fold warns once per fold.
Two groups of the package's own warnings are collected over the folds
instead and raised once per call, under this function's name and with the
class they had:
the two in which
predict()on a GWR counts locations ("spatialkit_gwr_not_estimable"and"spatialkit_gwr_extrapolation"), with the totals over the folds that had any, ascv_gwr()raises them (seepredict.gwr_fit());the four
fit_bayesian_spatial_model()raises about a fit, naming the folds, ascv_bayes()raises them (see the section "Warnings from the folds" there).
With parallel above one core a fold's warnings are raised after the folds
have run, once per distinct text; one of the package's own classes keeps
its class and its fields.
Percentage errors on responses with zeros
MAPE divides by the observed value and SMAPE by
\(|y| + |\hat{y}|\), so neither is defined where its denominator is zero.
Neither returns Inf or NaN. Both are averaged over the rows
whose denominator is non-zero, and are NA when no row qualifies.
Non-zero is judged at the scale of the data: a denominator no larger
than 100 machine epsilons times the largest one counts as zero, so the
rule does not depend on the units of the response.
The n_MAPE and n_SMAPE columns record how many rows that was;
the n column (n_pred in a cross-validation result) counts
finite observation/prediction pairs. Read a percentage error next to that
count: when n_MAPE is smaller, MAPE is an average over a
subset of the data, whatever its value.
Any response taking exact zeros is affected: counts, rainfall, abundance,
claim amounts. On a zero-inflated response with 62 zeros out of 120,
MAPE averages the 58 non-zero rows and n_MAPE is 58.
SMAPE drops the rows where observation and prediction are both
near zero, which on a well-fitted zero-inflated model are the rows it got
right, so it averages the harder rows only and reads worse than the fit
deserves. n_SMAPE shows how many rows it kept; it does not correct
the average.
RMSE, MAE and \(R^2\) use every finite row; prefer them
whenever the response can be zero. For a Bayesian GP fit,
cv_bayes() also reports CRPS and interval coverage, which
are proper scoring rules and are defined at zero.
See also
new_spatial_fit() for the constructor a fit_fn must use;
cv_gwr(), cv_bayes() and cv_rf() for the built-in backends.
Other cross-validation:
compare_models_cv(),
cv_bayes(),
cv_gwr(),
cv_rf(),
plot_calibration(),
plot_cv_metrics()
Examples
library(sf)
set.seed(1)
n <- 80
site <- st_as_sf(
data.frame(x = 5e5 + runif(n, 0, 1000), y = 5e6 + runif(n, 0, 1000),
elev = rnorm(n)),
coords = c("x", "y"), crs = 32632
)
site$price <- 10 + 0.01 * (st_coordinates(site)[, 1] - 5e5) +
2 * site$elev + rnorm(n)
# 1. A fit_fn returning a spatial_fit of your own subclass.
lm_fit <- function(train_sf) {
new_spatial_fit(
subclass = "lm_fit",
engine = lm(price ~ elev, st_drop_geometry(train_sf)),
formula = price ~ elev,
response_var = "price",
predictor_vars = "elev",
data_sf = train_sf
)
}
# 2. The predict() method cv_spatial() scores each fold with. Without it
# every fold fails and `overall` comes back all-NA.
predict.lm_fit <- function(object, newdata = NULL, ...) {
if (is.null(newdata)) newdata <- object$data_sf
as.numeric(stats::predict(object$engine, st_drop_geometry(newdata)))
}
registerS3method("predict", "lm_fit", predict.lm_fit)
cv <- cv_spatial(site, "price", "elev", fit_fn = lm_fit, k = 3, seed = 1)
#> cv_spatial(): no folds supplied -- using spatial block k-fold CV (k=3).
cv$overall
#> RMSE MAE MAPE SMAPE R2 Adj_R2 n_pred n_MAPE n_SMAPE
#> 1 3.472328 3.009688 21.29953 20.27029 0.2341257 NA 80 80 80
# Compare these before reading the metrics above; fold_status says why
# any fold is missing from fold_metrics.
c(attempted = cv$n_folds_attempted, succeeded = cv$n_folds_succeeded)
#> attempted succeeded
#> 3 3
cv$fold_status
#> fold status message
#> 1 1 ok
#> 2 2 ok
#> 3 3 ok
# 3. A metric of your own beside the built-in ones: per fold and pooled.
med_ae <- function(y, yhat) c(MedAE = stats::median(abs(y - yhat)))
cv2 <- cv_spatial(site, "price", "elev", fit_fn = lm_fit, k = 3, seed = 1,
metrics = med_ae)
#> cv_spatial(): no folds supplied -- using spatial block k-fold CV (k=3).
cv2$overall$MedAE
#> [1] 2.981173
cv2$fold_metrics[, c("fold", "RMSE", "MedAE")]
#> fold RMSE MedAE
#> 1 1 3.793601 3.536960
#> 2 2 3.198481 2.590833
#> 3 3 3.359840 2.678648