Select predictors by forward selection on spatial folds
Source:R/feature-selection.R
select_features_forward.RdSelects predictors by repeatedly adding whichever candidate most improves a
cross-validated score, stopping when the best candidate's improvement is
no larger than the fold-to-fold noise in it (rule = "se", the
default) or no larger than tol (rule = "tol"). The first
variable has a baseline to beat with every learner, so the selection can
be empty.
Usage
select_features_forward(
train_sf,
response_var,
candidate_vars,
fit_fn,
k = 5,
method = c("block_kfold", "random_kfold"),
block_size = NULL,
metric = c("RMSE", "MAE", "R2"),
tol = 0,
max_vars = NULL,
max_fits = 5000L,
seed = 123,
quiet = FALSE,
auto_range = FALSE,
select_on = c("all", "split"),
rule = c("se", "tol"),
se_mult = 1,
null_fn = NULL
)Arguments
- train_sf
Training data (
sf). Withselect_on = "split"a feature that is not a point is reduced to one, and the layer is read in x and y (see spatialkit-geometry).- response_var
Character(1).
- candidate_vars
Character vector of predictors to choose among.
- fit_fn
A function
(train_sf, predictor_vars)returning aspatial_fit. The signature takes two arguments because selection has to refit with different predictor sets.- k
Inner fold count. Default 5.
- method
Inner fold method:
"block_kfold"(default) or"random_kfold".- block_size
Minimum inner block edge, passed to
make_folds();NULL(default) for the geometric grid. Pass the outer block size so inner and outer blocks are on the same scale.- metric
Score to optimise:
"RMSE","MAE"(minimised) or"R2"(maximised). Default"RMSE".- tol
Minimum improvement of the pooled score required to accept a variable, in the units of
metric. Default 0. It is a floor under both rules: withrule = "tol"it is the whole test, and withrule = "se"the per-fold test applies as well. The first variable is judged against a baseline with every learner; see "The baseline for the first variable". Any single number is accepted:Infleaves nothing selected, and-Infputs no floor under a step.- max_vars
Optional cap on how many predictors to select;
NULL(default) for no cap.- max_fits
Abort if the sweep would exceed this many model fits. Default 5000.
- seed
RNG seed, default 123. It governs both the inner fold construction and the cross-validation itself: it is forwarded to
cv_spatial(seed = ), which draws one RNG stream per fold from it, so it also seeds the learner inside every fold. A stochasticfit_fnis therefore reproducible from this one value.- quiet
Logical; suppress this function's progress
message()s. It does not silence R warnings, nor the package's console log echo (seespatialkit_quietfor that). DefaultFALSE.- auto_range
Logical. If
TRUEandmethodis"block_kfold", the autocorrelation range is estimated from the response (detrended on the candidates) and used as the minimum block size of the inner folds, exactly as inmake_folds(): ablock_sizegiven takes precedence, and the range is then only compared with it. DefaultFALSE, which keeps the geometric blocks. With inner blocks smaller than the range a candidate can be chosen for tracking nearby values of the response, the case therandom_kfoldcaution above is about, so the warningmake_folds()raises on blocks smaller than the range applies to the inner folds too.- select_on
"all"(default) runs the sweep on every row oftrain_sf."split"runs it on one spatially blocked half, then fits the selected set on that half and scores it on the other:score_holdoutis then the selected model'smetricon rows whose response the sweep never read (the sweep's ownscoreis not; see "Selection score"). It is one estimate from one half of the area: the halves share a border with no buffer, so rows near it are still correlated with the selection half. Both halves come back in$split. See the "Post-selection inference" section ofdetermine_optimal_levelsfor the trade: coverage for half the sample.- rule
How a step is judged.
"se"(default) accepts the best candidate when its pooled gain exceedstoland its mean per-fold gain exceedsse_multstandard errors of that mean;"tol"accepts any pooled gain abovetol. See "When a variable is accepted".- se_mult
Number of standard errors the mean per-fold gain must exceed under
rule = "se". Default 1; 0 asks only that the mean per-fold gain be positive.- null_fn
Optional function
(train_sf)returning a fitted model with no predictors, used as the baseline for the first variable whenfit_fncannot fit an empty predictor set. DefaultNULL:fit_fn(train_sf, character(0))is tried, and the training-fold mean is used when that fails. The example builds one.
Value
A list of class "feature_selection" (so that
plot.feature_selection() draws the selection path) with
these elements:
selectedThe chosen predictors, in the order they were added; possibly none.
scoreThe winning set's cross-validated
metricat the final step: the optimum inside the selection, optimistically biased because it was chosen as the best of many (see "Selection score"), andNAwhen nothing was selected.score_holdoutNAunlessselect_on = "split", and then the selected set'smetricwhen fitted on the selection half and predicted on the estimation half (\(R^2\) against the selection half's mean, the out-of-sample convention);NAwhen nothing was selected or the prediction failed.historyA data.frame with
step,variable,score,n_pred,gain,fold_gainandfold_gain_se, holding every candidate evaluated at every step, and astep = 0row named"<none>"giving the baseline, so the first variable's gain can be read off directly.gainis the candidate set's improvement in the pooled score over the set it was added to (the baseline at step 1),fold_gainthe mean of the per-fold improvements andfold_gain_seits standard error: the three numbers the rule reads.n_predis how many rows the set's cross-validation predicted.fold_scoresA data.frame with
step,variable,foldandscore: the metric of every set in every fold.baselineA list with
type("null model","null_fn"or"fold mean"),score,path(the set the sweep reached),gain,fold_gainandfold_gain_se(that set against the baseline),kept(FALSEwhen the path did not beat a fold-mean baseline andselectedis empty for that reason;NAwhen the path is empty) andfirst.firstisNULLunless the baseline is the fold mean and a path of two or more variables was kept. It is then the test of the first variable, a list with:variable;gain,fold_gainandfold_gain_se: the path against the path without that variable;rest, a list withvars,score,gain,fold_gainandfold_gain_se: the path without it, and that set against the baseline;dropped:TRUEwhen the variable was taken out (selectedis thenpath[-1]andscoreisrest$score);note, the verdict in words.
The set scored for that test is not a step of the sweep, so it is not in
historyorfold_scores.splitNULL, or a list withselectionandestimation, integer row positions intrain_sfas passed; rows the completeness filter above dropped are in neither.paramsThe settings:
metric,method,k,tol,seed,auto_range,select_on,n_candidates,estimated_fits,n_scored,rule,se_multandbaseline(its type).
Every set is scored on the same rows: those the null model's
cross-validation predicted or, when the baseline is the fold mean,
those any step-1 set predicted; params$n_scored counts them (a
warning says so when that is fewer than all). A set that left some of
the scored rows unpredicted, because a fold failed for it, has
score NA, with a warning naming it: scored on the rows it
did predict it would be compared on fewer, usually easier, rows than its
rivals. A factor with a level found in one spatial block only is the
usual case, and cannot be selected.
Details
To get a performance estimate for the selected model, call this function
inside the fit_fn you pass to cv_spatial().
cv_spatial() calls fit_fn(train_sf) on each fold's training
rows only, so a selection run inside fit_fn sees no held-out data.
The cost grows fast: a sweep over p candidates costs roughly
p^2 / 2 * k model fits, and nesting it inside n outer folds
multiplies that by n. max_fits guards against it.
Inner and outer folds
The inner folds should hold points out as the outer ones do. A
cross-validated score is the error at the distance the folds put between
the held-out points and their training data, so nested selection chooses
the variables that predict best at the inner folds' distance. Random
inner folds inside blocked outer folds choose for predicting among the
samples, where a variable that only tracks nearby values of the response
can win, and the outer loop then scores the chosen set at the blocks'
distance, for which it was not chosen. method therefore defaults
to "block_kfold", as the folds of the cv_*() functions do,
and logs a caution (not an R warning) when set to "random_kfold".
When a variable is accepted
Each candidate set is scored by cross-validation on the same inner folds,
and the candidate with the best pooled score is the one considered. Under
rule = "se" it is accepted when two things hold: the pooled score
improves by more than tol, and the mean of the per-fold
improvements (the metric computed in each fold for the set with and
without the candidate, and differenced) exceeds se_mult times the
standard error of that mean across the folds. This is the
one-standard-error idea of Breiman et al. (1984) applied to a step: a gain
the folds do not agree on is not evidence that the variable predicts. The
fold is the unit because under spatially blocked folds the errors within
a block are correlated, so a standard error over rows would understate
the noise. Against rule = "tol", which accepts any pooled gain
above tol, the rule adds fewer noise variables and costs power on
weak predictors. With few folds the standard error is itself noisy, so
raise k before lowering se_mult. history records each
candidate's pooled gain, mean per-fold gain and its standard error, and
fold_scores the per-fold scores themselves, so every decision can be
checked.
The baseline for the first variable
The first variable is compared with a model that has no predictors,
recorded as step 0 of history. Which model that is depends on the
learner, and baseline$type says which was used:
baseline$type | The baseline | How the first variable is tested |
"null model" | fit_fn(train_sf, character(0)), when the
learner can fit it | at step 1, like any other step |
"null_fn" | the model null_fn(train_sf) returns, for a
learner that cannot fit an empty set | at step 1 |
"fold mean" | the mean response of each fold's training rows,
when neither is available (fit_rf_model() refuses an
empty predictor set) | the whole set at the end of the path, then the set without its first variable |
Under a fold-mean baseline the first variable is provisional: it is added
without a test, the sweep continues by the rule, and the finished set is
kept only if it beats the fold mean by the same rule; otherwise nothing is
selected, and baseline$path holds the set that was reached. The
test is at the end of the path because a forest on a single variable can
lose to the mean when two variables together beat it clearly. A kept
path of two or more variables is then scored once more without its first
variable (one more set, k more fits), and that variable is dropped
from selected when the set is no better with it by the rule and the
rest of the path still beats the fold mean by the rule.
baseline$first records the test, and selected is
baseline$path[-1] after a drop. When the set is no better with the
first variable but the rest of the path does not beat the fold mean
either, the two cannot be told apart: the set is kept whole, a caution is
logged, and baseline$first$note says the variable is unproven. So
in a selection of two or more variables from a learner that falls back to
the fold mean, selected[1] has been shown to improve the set, not
to predict on its own.
For a GWR the fold mean is a weak baseline: every GWR carries a local
intercept and beats a global mean on any smooth response, whatever its
predictors are. The baseline that matters is the kernel-weighted local
mean at the same bandwidth, a GWR with a local intercept and no
predictors, which fit_gwr_model() fits when its
predictor_vars is character(0). A fit_fn that hands
its variables to fit_gwr_model() therefore gets it as the null
model. Give such a fit_fn a fixed bandwidth, so that every
set is fitted under the kernel the baseline has; with
bandwidth = NULL each set, the empty one included, searches for
its own. A GWR fit_fn that refuses an empty set falls back to the
fold mean, and the sweep logs a caution: read a selection of one variable
made that way as unproven, or supply the local mean as null_fn, as
the second half of the example does.
predict() on a GWR warns about held-out locations it cannot
estimate and about those beyond the reach of a fold's fit (classes
"spatialkit_gwr_not_estimable" and
"spatialkit_gwr_extrapolation"; see spatial_fit-predict).
The sweep raises each once, under this function's name, with the totals
over the candidate sets it scored; every set is predicted on the same
rows, so a location can be counted once per set.
Selection with a random forest
A forest confounds selection with tuning: a predictor that carries no
signal still changes how the trees split, and with ranger's default node
size that can regularise the forest and lower the cross-validated error.
No acceptance rule refuses that gain, so a variable in a forest selection
may be there as a tuning effect, wherever in the order it was added.
Setting the node size in fit_fn reduces this without removing it,
and comparing the selected set with a tuned forest on the variables you
have other reasons to believe in shows how much of the gain is tuning:
fit_fn <- function(train_sf, vars)
fit_rf_model(train_sf, "z", vars, min_node_size = 20)Selection score
$score is the cross-validated metric of the winning set at
the final step: the best of every candidate set the sweep scored. A
number optimised over many candidates is optimistically biased by
construction, and Cawley and Talbot (2010) show the bias can exceed the differences between
the models being compared. Quote it as the selection criterion, not as
the performance of the selected model. A performance estimate needs data
the selection never saw. Two ways to get one: run this function inside
the fit_fn of cv_spatial(), so the outer folds score a
model whose predictors were chosen on the inner ones alone; or pass
select_on = "split", which selects on one spatial half of
train_sf and returns the selected set's score on the other as
score_holdout.
References
Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth.
Cawley, G. C. and Talbot, N. L. C. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11, 2079–2107. https://jmlr.org/papers/v11/cawley10a.html
See also
plot.feature_selection() to draw the selection
path, gwr_model_selection() for a fast in-sample screen for
GWR, and cv_spatial() to nest the selection in outer folds.
Other predictor selection:
gwr_model_selection(),
plot.feature_selection(),
plot.gwr_model_selection()
Examples
library(sf)
set.seed(1)
n <- 120
pts <- st_as_sf(
data.frame(x = 5e5 + runif(n, 0, 1000), y = 5e6 + runif(n, 0, 1000),
a = rnorm(n), b = rnorm(n), noise = rnorm(n)),
coords = c("x", "y"), crs = 32632
)
pts$resp <- 3 * pts$a + 2 * pts$b + rnorm(n, 0, 0.3)
# The learner: a GWR with an adaptive bisquare kernel reaching the 30th
# nearest training point. With no variables it is the kernel-weighted
# local mean, which is the baseline: a GWR's local intercept beats a
# global mean on any smooth response, so the first variable has to beat
# the local mean under the same kernel.
fit_fn <- function(tr, vars) fit_gwr_model(tr, "resp", vars, bandwidth = 30)
sel <- select_features_forward(pts, "resp", c("a", "b", "noise"), fit_fn,
k = 3, quiet = TRUE)
sel$selected
#> [1] "a" "b"
sel$baseline$type # "null model": step 1 was tested against it
#> [1] "null model"
sel$history[sel$history$step <= 1, c("step", "variable", "score", "gain")]
#> step variable score gain
#> 1 0 <none> 3.523273 NA
#> 2 1 a 2.181982 1.34129082
#> 3 1 b 3.477749 0.04552313
#> 4 1 noise 3.487167 0.03610585
# A learner that cannot fit an empty set takes its no-predictor model
# through `null_fn`. Here the same local mean, written as a spatial_fit
# of its own with a predict() method, for a learner that insists on a
# variable. Without `null_fn` that sweep would start from the mean of
# each training fold.
needs_one <- function(tr, vars) {
if (!length(vars)) stop("this learner needs a predictor")
fit_gwr_model(tr, "resp", vars, bandwidth = 30)
}
local_mean <- function(tr, bandwidth = 30) {
new_spatial_fit("local_mean_fit",
engine = list(xy = st_coordinates(tr), y = tr$resp,
bandwidth = bandwidth),
formula = resp ~ 1, response_var = "resp",
predictor_vars = character(0), data_sf = tr)
}
predict.local_mean_fit <- function(object, newdata = NULL, ...) {
e <- object$engine
at <- if (is.null(newdata)) e$xy else st_coordinates(newdata)
vapply(seq_len(nrow(at)), function(i) {
d <- sqrt((e$xy[, 1] - at[i, 1])^2 + (e$xy[, 2] - at[i, 2])^2)
h <- sort(d)[min(e$bandwidth, length(d))] # the kernel's reach here
w <- ifelse(d > h, 0, (1 - d^2 / h^2)^2) # bisquare weights
sum(w * e$y) / sum(w)
}, numeric(1))
}
registerS3method("predict", "local_mean_fit", predict.local_mean_fit)
sel2 <- select_features_forward(pts, "resp", c("a", "b", "noise"), needs_one,
k = 3, quiet = TRUE, null_fn = local_mean)
sel2$baseline$type # "null_fn"
#> [1] "null_fn"
all.equal(sel2$baseline$score, sel$baseline$score) # the same baseline
#> [1] TRUE