Skip to contents

Selects predictors by repeatedly adding whichever candidate most improves a cross-validated score, stopping when the best candidate's improvement is no larger than the fold-to-fold noise in it (rule = "se", the default) or no larger than tol (rule = "tol"). The first variable has a baseline to beat with every learner, so the selection can be empty.

Usage

select_features_forward(
  train_sf,
  response_var,
  candidate_vars,
  fit_fn,
  k = 5,
  method = c("block_kfold", "random_kfold"),
  block_size = NULL,
  metric = c("RMSE", "MAE", "R2"),
  tol = 0,
  max_vars = NULL,
  max_fits = 5000L,
  seed = 123,
  quiet = FALSE,
  auto_range = FALSE,
  select_on = c("all", "split"),
  rule = c("se", "tol"),
  se_mult = 1,
  null_fn = NULL
)

Arguments

train_sf

Training data (sf). With select_on = "split" a feature that is not a point is reduced to one, and the layer is read in x and y (see spatialkit-geometry).

response_var

Character(1).

candidate_vars

Character vector of predictors to choose among.

fit_fn

A function (train_sf, predictor_vars) returning a spatial_fit. The signature takes two arguments because selection has to refit with different predictor sets.

k

Inner fold count. Default 5.

method

Inner fold method: "block_kfold" (default) or "random_kfold".

block_size

Minimum inner block edge, passed to make_folds(); NULL (default) for the geometric grid. Pass the outer block size so inner and outer blocks are on the same scale.

metric

Score to optimise: "RMSE", "MAE" (minimised) or "R2" (maximised). Default "RMSE".

tol

Minimum improvement of the pooled score required to accept a variable, in the units of metric. Default 0. It is a floor under both rules: with rule = "tol" it is the whole test, and with rule = "se" the per-fold test applies as well. The first variable is judged against a baseline with every learner; see "The baseline for the first variable". Any single number is accepted: Inf leaves nothing selected, and -Inf puts no floor under a step.

max_vars

Optional cap on how many predictors to select; NULL (default) for no cap.

max_fits

Abort if the sweep would exceed this many model fits. Default 5000.

seed

RNG seed, default 123. It governs both the inner fold construction and the cross-validation itself: it is forwarded to cv_spatial(seed = ), which draws one RNG stream per fold from it, so it also seeds the learner inside every fold. A stochastic fit_fn is therefore reproducible from this one value.

quiet

Logical; suppress this function's progress message()s. It does not silence R warnings, nor the package's console log echo (see spatialkit_quiet for that). Default FALSE.

auto_range

Logical. If TRUE and method is "block_kfold", the autocorrelation range is estimated from the response (detrended on the candidates) and used as the minimum block size of the inner folds, exactly as in make_folds(): a block_size given takes precedence, and the range is then only compared with it. Default FALSE, which keeps the geometric blocks. With inner blocks smaller than the range a candidate can be chosen for tracking nearby values of the response, the case the random_kfold caution above is about, so the warning make_folds() raises on blocks smaller than the range applies to the inner folds too.

select_on

"all" (default) runs the sweep on every row of train_sf. "split" runs it on one spatially blocked half, then fits the selected set on that half and scores it on the other: score_holdout is then the selected model's metric on rows whose response the sweep never read (the sweep's own score is not; see "Selection score"). It is one estimate from one half of the area: the halves share a border with no buffer, so rows near it are still correlated with the selection half. Both halves come back in $split. See the "Post-selection inference" section of determine_optimal_levels for the trade: coverage for half the sample.

rule

How a step is judged. "se" (default) accepts the best candidate when its pooled gain exceeds tol and its mean per-fold gain exceeds se_mult standard errors of that mean; "tol" accepts any pooled gain above tol. See "When a variable is accepted".

se_mult

Number of standard errors the mean per-fold gain must exceed under rule = "se". Default 1; 0 asks only that the mean per-fold gain be positive.

null_fn

Optional function (train_sf) returning a fitted model with no predictors, used as the baseline for the first variable when fit_fn cannot fit an empty predictor set. Default NULL: fit_fn(train_sf, character(0)) is tried, and the training-fold mean is used when that fails. The example builds one.

Value

A list of class "feature_selection" (so that plot.feature_selection() draws the selection path) with these elements:

selected

The chosen predictors, in the order they were added; possibly none.

score

The winning set's cross-validated metric at the final step: the optimum inside the selection, optimistically biased because it was chosen as the best of many (see "Selection score"), and NA when nothing was selected.

score_holdout

NA unless select_on = "split", and then the selected set's metric when fitted on the selection half and predicted on the estimation half (\(R^2\) against the selection half's mean, the out-of-sample convention); NA when nothing was selected or the prediction failed.

history

A data.frame with step, variable, score, n_pred, gain, fold_gain and fold_gain_se, holding every candidate evaluated at every step, and a step = 0 row named "<none>" giving the baseline, so the first variable's gain can be read off directly. gain is the candidate set's improvement in the pooled score over the set it was added to (the baseline at step 1), fold_gain the mean of the per-fold improvements and fold_gain_se its standard error: the three numbers the rule reads. n_pred is how many rows the set's cross-validation predicted.

fold_scores

A data.frame with step, variable, fold and score: the metric of every set in every fold.

baseline

A list with type ("null model", "null_fn" or "fold mean"), score, path (the set the sweep reached), gain, fold_gain and fold_gain_se (that set against the baseline), kept (FALSE when the path did not beat a fold-mean baseline and selected is empty for that reason; NA when the path is empty) and first. first is NULL unless the baseline is the fold mean and a path of two or more variables was kept. It is then the test of the first variable, a list with:

  • variable;

  • gain, fold_gain and fold_gain_se: the path against the path without that variable;

  • rest, a list with vars, score, gain, fold_gain and fold_gain_se: the path without it, and that set against the baseline;

  • dropped: TRUE when the variable was taken out (selected is then path[-1] and score is rest$score);

  • note, the verdict in words.

The set scored for that test is not a step of the sweep, so it is not in history or fold_scores.

split

NULL, or a list with selection and estimation, integer row positions in train_sf as passed; rows the completeness filter above dropped are in neither.

params

The settings: metric, method, k, tol, seed, auto_range, select_on, n_candidates, estimated_fits, n_scored, rule, se_mult and baseline (its type).

Every set is scored on the same rows: those the null model's cross-validation predicted or, when the baseline is the fold mean, those any step-1 set predicted; params$n_scored counts them (a warning says so when that is fewer than all). A set that left some of the scored rows unpredicted, because a fold failed for it, has score NA, with a warning naming it: scored on the rows it did predict it would be compared on fewer, usually easier, rows than its rivals. A factor with a level found in one spatial block only is the usual case, and cannot be selected.

Details

To get a performance estimate for the selected model, call this function inside the fit_fn you pass to cv_spatial(). cv_spatial() calls fit_fn(train_sf) on each fold's training rows only, so a selection run inside fit_fn sees no held-out data. The cost grows fast: a sweep over p candidates costs roughly p^2 / 2 * k model fits, and nesting it inside n outer folds multiplies that by n. max_fits guards against it.

Inner and outer folds

The inner folds should hold points out as the outer ones do. A cross-validated score is the error at the distance the folds put between the held-out points and their training data, so nested selection chooses the variables that predict best at the inner folds' distance. Random inner folds inside blocked outer folds choose for predicting among the samples, where a variable that only tracks nearby values of the response can win, and the outer loop then scores the chosen set at the blocks' distance, for which it was not chosen. method therefore defaults to "block_kfold", as the folds of the cv_*() functions do, and logs a caution (not an R warning) when set to "random_kfold".

When a variable is accepted

Each candidate set is scored by cross-validation on the same inner folds, and the candidate with the best pooled score is the one considered. Under rule = "se" it is accepted when two things hold: the pooled score improves by more than tol, and the mean of the per-fold improvements (the metric computed in each fold for the set with and without the candidate, and differenced) exceeds se_mult times the standard error of that mean across the folds. This is the one-standard-error idea of Breiman et al. (1984) applied to a step: a gain the folds do not agree on is not evidence that the variable predicts. The fold is the unit because under spatially blocked folds the errors within a block are correlated, so a standard error over rows would understate the noise. Against rule = "tol", which accepts any pooled gain above tol, the rule adds fewer noise variables and costs power on weak predictors. With few folds the standard error is itself noisy, so raise k before lowering se_mult. history records each candidate's pooled gain, mean per-fold gain and its standard error, and fold_scores the per-fold scores themselves, so every decision can be checked.

The baseline for the first variable

The first variable is compared with a model that has no predictors, recorded as step 0 of history. Which model that is depends on the learner, and baseline$type says which was used:

baseline$typeThe baselineHow the first variable is tested
"null model"fit_fn(train_sf, character(0)), when the learner can fit itat step 1, like any other step
"null_fn"the model null_fn(train_sf) returns, for a learner that cannot fit an empty setat step 1
"fold mean"the mean response of each fold's training rows, when neither is available (fit_rf_model() refuses an empty predictor set)the whole set at the end of the path, then the set without its first variable

Under a fold-mean baseline the first variable is provisional: it is added without a test, the sweep continues by the rule, and the finished set is kept only if it beats the fold mean by the same rule; otherwise nothing is selected, and baseline$path holds the set that was reached. The test is at the end of the path because a forest on a single variable can lose to the mean when two variables together beat it clearly. A kept path of two or more variables is then scored once more without its first variable (one more set, k more fits), and that variable is dropped from selected when the set is no better with it by the rule and the rest of the path still beats the fold mean by the rule. baseline$first records the test, and selected is baseline$path[-1] after a drop. When the set is no better with the first variable but the rest of the path does not beat the fold mean either, the two cannot be told apart: the set is kept whole, a caution is logged, and baseline$first$note says the variable is unproven. So in a selection of two or more variables from a learner that falls back to the fold mean, selected[1] has been shown to improve the set, not to predict on its own.

For a GWR the fold mean is a weak baseline: every GWR carries a local intercept and beats a global mean on any smooth response, whatever its predictors are. The baseline that matters is the kernel-weighted local mean at the same bandwidth, a GWR with a local intercept and no predictors, which fit_gwr_model() fits when its predictor_vars is character(0). A fit_fn that hands its variables to fit_gwr_model() therefore gets it as the null model. Give such a fit_fn a fixed bandwidth, so that every set is fitted under the kernel the baseline has; with bandwidth = NULL each set, the empty one included, searches for its own. A GWR fit_fn that refuses an empty set falls back to the fold mean, and the sweep logs a caution: read a selection of one variable made that way as unproven, or supply the local mean as null_fn, as the second half of the example does.

predict() on a GWR warns about held-out locations it cannot estimate and about those beyond the reach of a fold's fit (classes "spatialkit_gwr_not_estimable" and "spatialkit_gwr_extrapolation"; see spatial_fit-predict). The sweep raises each once, under this function's name, with the totals over the candidate sets it scored; every set is predicted on the same rows, so a location can be counted once per set.

Selection with a random forest

A forest confounds selection with tuning: a predictor that carries no signal still changes how the trees split, and with ranger's default node size that can regularise the forest and lower the cross-validated error. No acceptance rule refuses that gain, so a variable in a forest selection may be there as a tuning effect, wherever in the order it was added. Setting the node size in fit_fn reduces this without removing it, and comparing the selected set with a tuned forest on the variables you have other reasons to believe in shows how much of the gain is tuning:


fit_fn <- function(train_sf, vars)
  fit_rf_model(train_sf, "z", vars, min_node_size = 20)

Selection score

$score is the cross-validated metric of the winning set at the final step: the best of every candidate set the sweep scored. A number optimised over many candidates is optimistically biased by construction, and Cawley and Talbot (2010) show the bias can exceed the differences between the models being compared. Quote it as the selection criterion, not as the performance of the selected model. A performance estimate needs data the selection never saw. Two ways to get one: run this function inside the fit_fn of cv_spatial(), so the outer folds score a model whose predictors were chosen on the inner ones alone; or pass select_on = "split", which selects on one spatial half of train_sf and returns the selected set's score on the other as score_holdout.

References

Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth.

Cawley, G. C. and Talbot, N. L. C. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11, 2079–2107. https://jmlr.org/papers/v11/cawley10a.html

See also

plot.feature_selection() to draw the selection path, gwr_model_selection() for a fast in-sample screen for GWR, and cv_spatial() to nest the selection in outer folds.

Other predictor selection: gwr_model_selection(), plot.feature_selection(), plot.gwr_model_selection()

Examples

library(sf)
set.seed(1)
n <- 120
pts <- st_as_sf(
  data.frame(x = 5e5 + runif(n, 0, 1000), y = 5e6 + runif(n, 0, 1000),
             a = rnorm(n), b = rnorm(n), noise = rnorm(n)),
  coords = c("x", "y"), crs = 32632
)
pts$resp <- 3 * pts$a + 2 * pts$b + rnorm(n, 0, 0.3)

# The learner: a GWR with an adaptive bisquare kernel reaching the 30th
# nearest training point.  With no variables it is the kernel-weighted
# local mean, which is the baseline: a GWR's local intercept beats a
# global mean on any smooth response, so the first variable has to beat
# the local mean under the same kernel.
fit_fn <- function(tr, vars) fit_gwr_model(tr, "resp", vars, bandwidth = 30)
sel <- select_features_forward(pts, "resp", c("a", "b", "noise"), fit_fn,
                               k = 3, quiet = TRUE)
sel$selected
#> [1] "a" "b"
sel$baseline$type        # "null model": step 1 was tested against it
#> [1] "null model"
sel$history[sel$history$step <= 1, c("step", "variable", "score", "gain")]
#>   step variable    score       gain
#> 1    0   <none> 3.523273         NA
#> 2    1        a 2.181982 1.34129082
#> 3    1        b 3.477749 0.04552313
#> 4    1    noise 3.487167 0.03610585

# A learner that cannot fit an empty set takes its no-predictor model
# through `null_fn`.  Here the same local mean, written as a spatial_fit
# of its own with a predict() method, for a learner that insists on a
# variable.  Without `null_fn` that sweep would start from the mean of
# each training fold.
needs_one <- function(tr, vars) {
  if (!length(vars)) stop("this learner needs a predictor")
  fit_gwr_model(tr, "resp", vars, bandwidth = 30)
}
local_mean <- function(tr, bandwidth = 30) {
  new_spatial_fit("local_mean_fit",
                  engine = list(xy = st_coordinates(tr), y = tr$resp,
                                bandwidth = bandwidth),
                  formula = resp ~ 1, response_var = "resp",
                  predictor_vars = character(0), data_sf = tr)
}
predict.local_mean_fit <- function(object, newdata = NULL, ...) {
  e  <- object$engine
  at <- if (is.null(newdata)) e$xy else st_coordinates(newdata)
  vapply(seq_len(nrow(at)), function(i) {
    d <- sqrt((e$xy[, 1] - at[i, 1])^2 + (e$xy[, 2] - at[i, 2])^2)
    h <- sort(d)[min(e$bandwidth, length(d))]  # the kernel's reach here
    w <- ifelse(d > h, 0, (1 - d^2 / h^2)^2)   # bisquare weights
    sum(w * e$y) / sum(w)
  }, numeric(1))
}
registerS3method("predict", "local_mean_fit", predict.local_mean_fit)
sel2 <- select_features_forward(pts, "resp", c("a", "b", "noise"), needs_one,
                                k = 3, quiet = TRUE, null_fn = local_mean)
sel2$baseline$type       # "null_fn"
#> [1] "null_fn"
all.equal(sel2$baseline$score, sel$baseline$score)  # the same baseline
#> [1] TRUE