Computes the dissimilarity index (DI) of Meyer and Pebesma (2021) for a set of prediction locations and flags those that fall inside the model's area of applicability (AOA), the region of predictor space where the model's cross-validated performance estimate can be expected to hold.
Usage
area_of_applicability(
newdata,
model = NULL,
train_sf = NULL,
predictor_vars = NULL,
weights = "auto",
folds = NULL,
threshold = NULL,
normalizer_max_n = 5000L,
seed = 123L,
chunk_size = NULL,
use_fnn = requireNamespace("FNN", quietly = TRUE),
coords = c("auto", "yes", "no"),
categorical = c("refuse", "indicator", "drop")
)Arguments
- newdata
Prediction locations: an
sfobject (typically frompredict_surface) or a data.frame, carrying the predictor columns. When location is measured (agwr_fit, abayesian_fit, a forest fitted withinclude_coords = TRUE, orcoords = "yes") ansfobject is required, non-POINTgeometry is reduced to representative points, and a CRS mismatch with the training data is reconciled; see Models whose predictions depend on location.- model
A fitted
spatial_fit, supplying the training data and predictor names, the importanceweights = "auto"reads, and whether its predictions depend on location. Optional iftrain_sfandpredictor_varsare given directly, which lets this be used with any model.- train_sf
Training data, if not taken from
model. When location is measured it is read in x and y, asnewdatais.- predictor_vars
Predictor names, if not taken from
model.- weights
"auto"(default),NULL, or a named numeric vector of predictor importances."auto"usespmax(model$info$importance, 0)whenmodelcarries a finite importance with at least one positive value, naming every predictor in use, from a model with out-of-bag skill; otherwise every predictor counts equally and a logged note says why (see Weights).NULLis the explicit request for equal weights. For a vector of your own, any positive scale works. Whenmodelwas fitted withinclude_coords = TRUE, the coordinates"..x"and"..y"are measured as predictors too, besides the location index, which no weight changes (see Models whose predictions depend on location). You are not expected to supply importances for them: any you leave out default to the mean of the weights you did supply, so in the predictor index the coordinates count about as much as a typical predictor. Naming them explicitly overrides that. An unnamed vector may have one value per predictor either with or without the two coordinate columns. Weights must be finite and non-negative, so pass permutation importance aspmax(importance, 0). (A forest with no out-of-bag rows,replace = FALSEwithsample_fraction = 1, hasNaNimportance, whichpmax()keeps and which is refused when passed by hand;"auto"uses equal weights for it.)pmax(importance, 0)is all zero when the model found no predictor useful, and then the weights cannot say anything: with a single predictor any weight gives the same index and zero is accepted; with several, all of them are weighted equally, as withweights = NULL, and a warning says so. The coordinate default above is the mean of the supplied weights, so a zero weight on the only covariate of a coordinate-using model zeroes the coordinates too, and that equal weighting applies.- folds
Cross-validation folds: a
make_foldsresult, a list oftrain/testsplits, or a vector of fold labels with one entry per training row. DefaultNULL(plain nearest neighbour). The folds you passed tocv_*(), built on the layermodelwas fitted from, may name rows thatprep_model_data()removed (a missing or non-finite value, or no location): as incv_*(), they are dropped from the folds, and a label vector with one entry per row of that layer loses theirs. Also as incv_*(), amake_folds()result built on other data (another layer, or these rows in another order) is refused. That check is skipped when the folds were built on polygons and the training data are the points a fit reduced them to.- threshold
Optional numeric override for the DI threshold of the predictor index. The location index, when there is one, always takes its threshold from the rule.
- normalizer_max_n
Subsample the training data to this many points when computing the mean pairwise distance, which is quadratic. Default 5000. The subsample is drawn from the training rows in a fixed order (by the predictors, and for the location index by the coordinates), so it does not depend on the row order of the training data.
- seed
Seed for that subsample. Default 123.
- chunk_size
Query rows per distance block on the dense path. Default
NULL(chosen from the training size). Otherwise a single number of at least 1; a fractional value is truncated to a whole number of rows.- use_fnn
Use FNN for nearest-neighbour search when available. Exposed so the dense fallback can be tested. The two paths return the same distances: the dense one searches with a fast formula and then recomputes the nearest distance exactly, so an exact copy of a training row is at distance 0 on both.
- coords
Whether location is measured:
"auto"(default) measures it for a model whose predictions depend on it (agwr_fit, abayesian_fit, a forest fitted withinclude_coords = TRUE) and not otherwise;"yes"measures it whatever the model, and with nomodel;"no"never does, and logs a caution for a model that depends on it (for a forest it also leaves"..x"and"..y"out of the predictor index). See Models whose predictions depend on location.- categorical
What to do with factor and character predictors:
"refuse"(default) stops with a message naming them and the two alternatives;"indicator"codes each level as a 0/1 column and marks a prediction row on a level no training row has as outside (DI = Inf);"drop"computes the index on the other predictors and logs which were left out. See Limitations for the scaling"indicator"uses.
Value
An object of class aoa: a list with
aoa:newdatawith a numericDIcolumn and a logicalAOAcolumn added, and between them a numericDI_locationcolumn when location has an index of its own. For a forest fitted withinclude_coords = TRUE, unlesscoords = "no", it is the object the computation ran on:newdataafter pointizing, CRS reconciliation and the addition of the"..x"and"..y"columns. In every other case it isnewdataas passed, also when location has an index of its own. A row whose predictors are not all finite getsNAinDI, and a row outside the training range of a predictor indropped_vars, or on a level no training row has, getsDI = InfandAOA = FALSE(see Limitations).AOAisDI <= threshold, and with a location index alsoDI_location <= location$threshold: a row outside on either index isFALSE, and a row with one index missing and the other inside isNA.threshold: the DI cut-off used.train_DI: the training points' own DI values.normalizer: the mean pairwise training distance.weights: the weight vector actually applied, named bypredictor_vars.predictor_vars: the predictors used, including"..x"/"..y"when a forest uses the coordinates and excludingdropped_varsand, undercategorical = "drop",categorical_vars.dropped_vars: predictors dropped for negligible variance.scaling: a list withcenterandscale, each named bypredictor_vars: the training means and standard deviations the index is computed in, so a location's DI can be traced to the predictor that put it outside. A predictor coded as indicators has one entry per level, named"<predictor>=<level>".n_outliers: the number of training DI values above theQ3 + 1.5 * IQRfence, which the default threshold rule sets aside (the "outlier-removed" in its name); computed whether or notthresholdwas supplied.n_duplicates: withfolds = NULL, the number of training rows that repeat an earlier row's predictor values exactly and so share its reference distance;NAwhen folds were supplied (see Training rows with identical predictor values).location:NULL, or for a location index a list withthreshold,train_DI,normalizer(the mean distance between training points, in CRS units, sothreshold * normalizeris the cut-off as a distance),n_duplicates(training rows at an earlier row's location;NAwhen folds were supplied) andn_outside(prediction rows beyond the location threshold).categorical_vars: the factor and character predictors found, coded or left out asparams$categoricalsays.n_train,n_new,n_inside,n_outside,n_na: row counts.n_traincounts the training rows that survived the finite-value filter;n_newis every row ofnewdata, son_new = n_inside + n_outside + n_na, withn_nathe rows whoseAOAisNA.n_outside_predictorscounts the rows beyond the predictor threshold, which isn_outsideunless there is a location index.paramsrecords the call:folds_supplied,n_folds,folds_method,threshold_supplied,normalizer_max_n,normalizer_n_used,normalizer_subsampled,weights_supplied(the caller passed a vector),weights_source("model importance","supplied"or"equal"),weights_note(the kind of importance, or why the weights are equal),coords("location index";"location index and predictors"for a forest fitted withinclude_coords = TRUE, whose coordinates are also among the predictors; or"none"),categoricalandseed.folds_methodis themethodof amake_folds()result ("block_kfold","random_kfold", ...),"labels"for a vector of fold labels,"splits"for a bare list of train/test splits, andNAwhen no folds were supplied. It is printed with the object, because the threshold's meaning depends on it: the cross-validated DI that sets it comes from the same kind of hold-out as the CV error it should be quoted beside, so an AOA built onrandom_kfoldfolds is logged as a caution and does not belong next to a blockedcv_*()result.
What the index is for
A fitted model returns a number for any location you hand it, including locations whose predictor values look nothing like anything it was trained on. Those predictions are extrapolations, and a cross-validation score says nothing about them, because the held-out folds were drawn from the same predictor distribution as the training data. The AOA marks where the score applies.
How it is computed
Predictors are centred and scaled using the training data's own means and
standard deviations, then weighted by variable importance when the model
carries one (see Weights). For a
prediction point \(p\), the DI is the distance to its nearest training
point in that space, divided by the mean pairwise distance among training
points. The same quantity is computed for the training data itself, using
each point's nearest neighbour among the training rows of the fold
that holds it out. That means everything outside its own fold for random
and block folds, and the smaller training set that buffered and NNDM folds
leave (see "The fold scheme"); without
folds it is the nearest training row with different predictor values. The
threshold is then the largest
training DI that is not an upper outlier, i.e. not above the fence
Q3 + 1.5 * IQR of the training DI, with the quartiles of
stats::quantile()'s default type 7. Prediction points at or below
that threshold are inside the AOA.
That is the paper's "outlier-removed maximum". CAST, the reference
implementation, computes the same fence with the same quartiles but uses the
fence itself as the threshold, capped at the largest training DI. The two
agree whenever no training DI lies above the fence (n_outliers is 0);
otherwise CAST's threshold is the larger, and so is its AOA. (Earlier
CAST releases used grDevices::boxplot.stats(), which gives the
rule used here but with Tukey's hinges as the quartiles, so they can also
differ when the number of training points is even.) To apply the current
CAST rule to the same training DI, pass
threshold = min(quantile(res$train_DI, 0.75) + 1.5 * IQR(res$train_DI),
max(res$train_DI)) for an earlier result res.
The DI is invariant to the overall scale of weights: the numerator
and the normaliser carry the same factor. Importance values can be passed
as-is.
Training rows with identical predictor values
Repeat visits to a site with static covariates, or covariates read off a raster coarser than the sampling, give training rows that are exact copies of each other in predictor space. A copy is the same point of that space, not a neighbour. Counted as a neighbour it gives a training DI of 0, and once about three quarters of the rows have one the threshold is 0 and only exact copies of a training row count as inside.
With folds = NULL each training row's reference is therefore its
nearest row with different values: the search runs on the distinct
rows and every row takes its point's distance. n_duplicates counts
the rows that repeat an earlier one, and a layer with none is unaffected.
The normaliser stays the mean pairwise distance over
all the training rows, so a prediction point's DI is the same with and
without folds.
With folds supplied the definition is applied as it stands: a
held-out row whose copy is on the training side of its fold has a DI of 0,
because that hold-out did have an identical row to learn from. If the
threshold comes out 0 that way, a warning of class
spatialkit_aoa_zero_threshold gives the count, so a script or a
cross-validation loop can catch it with
tryCatch(..., spatialkit_aoa_zero_threshold = ). Folds that keep
the copies together (make_folds(method = "leave_location_out",
group_var = ...)) give the threshold its meaning back. The same applies
to the location index when training rows share a location.
The fold scheme
With folds = NULL the training reference is each point's nearest
neighbour anywhere in the training data, which for clustered data is very
close, giving a small threshold and a conservative AOA. Passing the folds
you validated with makes the reference distances larger and the AOA
wider. The AOA is defined relative to a performance estimate, and a
spatially blocked estimate is a claim about predicting further away. Pass the same make_folds() result you passed
to cv_spatial. Buffered and NNDM folds use the training set
they left available, not everything outside the fold.
Weights
Meyer and Pebesma weight the predictors by the model's variable
importance, and with the default weights = "auto" so does this
function whenever model carries an importance that means
something: pmax(model$info$importance, 0), which a forest from
fit_rf_model has. Equal weights are used, with a logged
note, when it does not: no model; a model with no importance
(importance = "none", a GWR, a Bayesian fit); an importance that is
not finite (NaN for a forest with no out-of-bag rows); one that
names fewer predictors than are in use; no positive value; or a forest
with no out-of-bag skill (oob_r_squared missing or at most 0),
whose positive importances are noise. print() says which it was.
weights = NULL asks for equal weights whatever the model carries.
Importance weights make the index react to shifts along the predictors
the model uses and much less to shifts along those it does not, which
leave its error unchanged. The cost is on the weak predictors. A weight
multiplies the scaled column, so its square enters the distance, and a
weak but real predictor is down-weighted by the square of its relative
importance. A shift along such a predictor can then go unseen although
it raises the error. weights = sqrt(pmax(importance, 0)) sits
between equal weights and importance weights: a weak predictor loses less
of its weight.
Limitations
Categorical predictors (factor or character columns) have no distance of
their own, and by default they are refused and never silently
dummy-coded. categorical = "indicator" codes each level that
occurs in the training rows as a 0/1 column. The columns of one predictor
share one scale, sqrt(1 - sum(p^2)) with p the levels'
shares of the training rows, so two rows on different levels are the same
distance apart whichever levels they are, two on the same level are 0
apart, and the predictor adds as much to the mean squared distance between
training rows as one standardised numeric predictor. That scaling is a
choice, so it is not the default. A prediction row
on a level no training row has gets DI = Inf, is outside the AOA,
and a warning gives the count. categorical = "drop" leaves the
categorical predictors out and computes the index on the others; a level
the training data never has then goes unseen.
Predictors whose training values do not vary beyond their rounding are
dropped from the distance. A predictor is kept when
sd > sqrt(.Machine$double.eps) * max(abs(x)), and otherwise when its
standard deviation is more than 3 units in the last place of
max(abs(x)). The same variable in metres and in gigametres is
treated identically, and a predictor at an offset (a timestamp in seconds)
is kept as it is at 0. A prediction point taking a value there outside the
training range by more than 3 units in the last place of
max(abs(x)) (of 1 for a column of zeros) is extrapolation along a
direction
the training data never varied in: its scaled distance along it is
infinite, so it gets DI = Inf, is outside the AOA, and a warning
gives the count. A point missing that value is judged on the other
predictors. With equal weights every predictor counts the same, which
overstates dissimilarity along directions the model barely uses; see
Weights.
Models whose predictions depend on location
For three kinds of model the prediction is a function of where the point
is, not only of its covariates, so the index has to measure location too.
Without that a prediction point far outside the training extent but with
ordinary covariate values reads as inside the area of
applicability, which is exactly the extrapolation this index exists to
catch. coords = "auto" (the default) measures location for all
three; coords = "no" turns it off, with a logged caution;
coords = "yes" measures it for any model, or for none.
A GWR (gwr_fit) has coefficients that vary over the map, a
Bayesian GP fit (bayesian_fit) has a Gaussian-process term in the
coordinates, and a forest fitted with include_coords = TRUE splits
on the coordinates. For these, and under coords = "yes", location
gets an index of its own, built like the predictor index but in the
coordinates:
DI_location is the
distance to the nearest training point over the mean distance between
training points; the training points' own values use the same folds; and
the threshold is the same outlier-removed maximum. A prediction point is
inside the AOA only if it is inside on both indices, and both DI columns
are returned so the cause can be read off. Location is kept apart from
the covariates because, as two more equally weighted columns, it would be
diluted as covariates are added, and points well beyond the training
extent would still read as inside.
Inside the extent the index marks an unsampled gap only where the gap is
wider than the distance its threshold allows, and that distance depends
on folds. With folds = NULL it is the spacing of the
training points (the outlier-removed largest nearest-neighbour distance).
With blocked folds it is the largest distance from a held-out point to
the nearest row its fold trained on, outliers removed, which grows with
the block size; a gap narrower than about twice that distance has no
cell beyond it and reads as inside. This follows from the definition:
the cross-validated error the index is paired with was earned on
hold-outs that far from their training rows. With blocked folds the
location index therefore says nothing about a gap narrower than about
twice the cut-off; location$threshold * location$normalizer is the
cut-off in CRS units, and print() shows it.
A forest splits on the coordinates like on any other column, so for a
forest they also stay in the predictor index as "..x" and
"..y", centred, scaled and weighted with the rest. Under
weights = "auto" that weight is the forest's importance, which is
small for a forest that barely splits on its coordinates, so points
beyond the extent can read as inside on the predictor index. The
location index counts location whatever weight the coordinates get.
Geometry is needed on both sides, so train_sf and
newdata must both be sf objects; a data.frame is refused
rather than quietly measured without location. Non-POINT
newdata (grid polygons, say) is reduced to representative points
first, as coerce_to_points would. If exactly one side carries
a CRS the other is brought into it: reprojected when its coordinates look
like longitude/latitude, stamped otherwise, with a warning either way; a
CRS in longitude and latitude is not stamped on coordinates outside
longitude -180 to 360 and latitude -90 to 90, which is refused (see
spatialkit-geometry). Degrees fed into an index in metres would
silently understate the distances. The location index measures distances in
the coordinates and refuses training data in a geographic CRS. Both
layers are read in x and y for the location: a layer whose geometries
mix XY with XYZ or XYM (as rbind() of an XY and an XYZ layer gives), or carry M (XYM or XYZM), is read without its Z
and M. Only one change reaches the result: for a forest fitted with
include_coords = TRUE, newdata that mixes dimensions and is
reprojected into the training CRS comes back in XY, with a warning of
class "spatialkit_zm_dropped", as ensure_projected()
returns such a layer; the points that non-POINT newdata is
reduced to are in XY when they would mix dimensions or carry M.
A row without a location (an empty geometry, or a coordinate that is not
finite or lies beyond 1e150) is read as one both as given and after a
move into the other layer's CRS, which can bring a coordinate of 1e200
back finite. Such a row of newdata gets NA in
DI_location, and for a forest fitted with
include_coords = TRUE in "..x" and "..y" and so in
DI; such a training row is left out of the reference distances,
with a log line. A row of newdata without a location as given that
is reprojected into the training CRS comes back in aoa with an
empty geometry.
References
Meyer, H. and Pebesma, E. (2021). Predicting into unknown space? Estimating the area of applicability of spatial prediction models. Methods in Ecology and Evolution 12(9), 1620–1633. doi:10.1111/2041-210X.13650
See also
predict_surface() to build the grid,
make_folds() for the fold scheme, and
plot.aoa() to draw the dissimilarity distribution.
Other prediction:
plot.aoa(),
predict_surface(),
print.aoa()
Examples
suppressPackageStartupMessages(library(sf))
set.seed(1)
n <- 120
train <- st_as_sf(
data.frame(x = 5e5 + runif(n, 0, 1000), y = 5e6 + runif(n, 0, 1000),
a = rnorm(n), b = rnorm(n)),
coords = c("x", "y"), crs = 32632
)
train$z <- 2 * train$a - train$b + rnorm(n, 0, 0.3)
# Prediction points, some of them well outside the training predictor range
newpts <- st_as_sf(
data.frame(x = 5e5 + runif(50, 0, 1000), y = 5e6 + runif(50, 0, 1000),
a = c(rnorm(40), rnorm(10, 8)), b = rnorm(50)),
coords = c("x", "y"), crs = 32632
)
res <- area_of_applicability(newpts, train_sf = train,
predictor_vars = c("a", "b"))
res
#> Area of applicability (Meyer & Pebesma 2021)
#>
#> predictors : 2 (a, b)
#> weighted : no (all predictors count equally: no model was given)
#> training : 120 points
#> reference : nearest other training point (no folds supplied)
#> normaliser : 1.7777 (mean pairwise distance)
#> threshold : 0.2899 (outlier-removed max of training DI)
#> location : not measured
#>
#> 39 of 50 prediction points inside the AOA (78.0%)
#>
#> Predictions outside the AOA are extrapolations; the cross-validated
#> performance estimate does not cover them.
table(res$aoa$AOA)
#>
#> FALSE TRUE
#> 11 39