Measure how far the held-out points sit from their training data
Source:R/fold-separation.R
fold_separation.RdReturns, for each fold, how far the held-out points sit from their nearest
training point. make_folds() only compares the block size with the
estimated autocorrelation range; this measures the folds that were built.
Nothing is estimated: the distances come from the geometry, and the range,
when one is shown, is the one the folds carry (make_folds(auto_range = TRUE) records it) or the one you pass as sac.
Arguments
- folds
A
make_folds()result, or its$foldselement (a list oftrain/testsplits).- data_sf
The layer the folds were built on. Row identifiers are matched through
..row_idwhen the layer carries one, and by row position otherwise, as inmake_folds()and everycv_*()function. As incv_*(), amake_folds()result whose recorded rows sit at other locations here (folds built on another layer, such as the points beforeassign_features_to_polygons()dropped some) is refused; the location check is skipped when one of the two layers is POINT and the other is not. A layer that mixes dimensions or carries M is read in x and y, for the distances and for that check (spatialkit-geometry). A row without a location, as given or in the CRS the distances are measured in, is left out of the distances.- sac
Optional: an
estimate_sac_range()result or a single number. Defaults to the range the folds carry, if any. Supplying one adds themean_corcolumn and, with a prediction target, the gap on the correlation scale. A range that records its CRS (the folds' own, or anestimate_sac_range()result) is compared with distances measured in that CRS, whatever CRSdata_sfis in. A bare number is taken to be in the units the distances are otherwise measured in: those ofdata_sfif it is projected, and for lon/lat input metres, in the CRSensure_projected()chooses (as for theblock_sizeofmake_folds()), not degrees. Aunitsobject is refused.- prediction_points
Optional sf or sfc layer of the locations the model will predict at: the grid from
predict_surface(), or a grid over the region of interest (sf::st_make_grid(region, what = "centers")). A layer of other geometry is reduced to one representative point per feature, as inmake_folds(method = "nndm"), so pass grid cells or points, not one polygon of the region. It is brought into the CRS the distances are measured in, and read in x and y when it mixes dimensions or carries M (spatialkit-geometry). DefaultNULL: no target and no reading, except for NNDM and kNNDM folds (method = "nndm"and"knndm"), which carry the distances from the prediction points they were built for and are compared against those, measured to all sites. The carried distances are lengths in the CRS the folds were built in (folds$params$crs). When the distances here are measured in another CRS (data_sfis in another projected CRS, orsacrecords another), the two are not compared,print()says so, and passingprediction_pointshas the target measured here. A prediction point without a location, as given or in the CRS the distances are measured in, takes no part in the target and is not counted.
Value
A data.frame of class fold_separation, one row per fold:
fold (the fold's number: for the $folds of a
cv_*() result, the fold_id its fold_metrics use,
which differs from the list position once a fold has been dropped),
n_train, n_test, n_blocks (NA
for a scheme with no blocks), min_dist and median_dist
(distance from a held-out point to its nearest training point, in the
units of the CRS the crs attribute names), mean_cor,
target_median_dist (the median distance from the prediction
points to the fold's training data) and dist_gap (the fold's mean
held-out distance minus its mean prediction distance); the last two are
NA without a target, and print() shows them under the
headings target and gap.
mean_cor is a mean correlation, NA without a range: the
mean over the fold's held-out points of \(\rho(d)\), the
correlation the range leaves between a point and its nearest training
point at distance \(d\). It is 1 for a point on top of training data
and falls towards 0 beyond the range, so a point at a twentieth of the
range counts for more than one at nine tenths of it.
\(\rho(d)\) is \(\exp(-3 d / range)\),
the exponential model whose effective range is the range given (0.05 at
the range), or the spherical model when the range comes from an
estimate_sac_range() fit that was spherical; the
range_model attribute says which.
Attributes: method, sac_range, range_model,
crs (the CRS the distances were measured in),
n_unknown_ids (the number of distinct rows the folds name that
data_sf does not have, each counted once however many folds
mention it, as in the cv_*() field of the same name),
pooled (a list with the number, minimum, median and mean of the
held-out distances over all folds, and the pooled mean_cor)
and target: NULL without a prediction target, otherwise a
list with n_prediction_points, source, basis
("each fold's training data" or "all sites"),
held_out_mean, held_out_median, target_mean,
target_median, target_all_mean, target_all_median
(the prediction distances to all sites), mean_diff, w1,
cor_gap, cor_w1 (NA without a range), scale
("correlation" or "distance"), tolerance,
reading (see the section "Reading the gap") and
nndm_out_of_reach (for NNDM and kNNDM folds, the
params$target_out_of_reach that make_folds() recorded;
NA for other schemes). not_compared is NULL, or,
for NNDM or kNNDM folds whose carried target was not compared because the
distances here are measured in another CRS, one sentence naming the two
CRSs; target is then NULL and print() gives the
sentence where it would otherwise say that no target is given.
Details
The distances do not say by themselves which way a score errs. A
cross-validated error reads low when the held-out points sit closer to
their training data than the locations the model will predict at, and high
when they sit farther, so the same held-out distances can go with either,
depending on the target. Random folds on evenly sited data, scored for a
map of the sampled area, are close to unbiased although nearly every
held-out point has a training neighbour inside the range; range-sized
blocks on the same data read high; random folds on clustered data scored
for a map of the whole region read low. So without a prediction target
the function reports the distances and draws no conclusion. Give it
prediction_points and it sets the held-out distances beside the
distances from those points to the same training data, and words its
reading by the signed gap between the two.
cv_block_size_sweep() gives the cross-validated error at each block
size, but not which size matches a map. This function does, read on the
folds of each size (the $folds of each element of
attr(sweep, "results")) with the map's locations as
prediction_points.
Reading the gap
With a target, the held-out distances and the prediction distances are compared as two distributions. Each fold's prediction distances are measured to that fold's own training rows, so that a five-fold design is not read as too far only because each fold trains on four fifths of the sites; the distances to all sites are reported beside them. With more than 50 folds, as in the leave-one-out schemes, every training set is nearly the whole sample, and the prediction points are measured against all sites once. Folds count by their share of the held-out points.
Two numbers summarise the comparison: the signed difference of the means,
held-out minus prediction (the signed area between the two cumulative
distributions), and the Wasserstein distance \(W_1\) (the unsigned
area). When a range is available both are also given on the correlation
scale, after each distance \(d\) is replaced by \(1 - \rho(d)\), with \(\rho\) the correlation the range implies (see
mean_cor under Value). That scale counts a difference between two
distances well inside the range and ignores one between two distances
beyond it.
The reading is taken on the correlation scale when there is a range, with
a tolerance of 0.1, and otherwise on the mean distance, with a tolerance of
a quarter of the prediction mean. When the prediction points are measured
to all sites and every one lies on a site, as when the sites themselves
are passed as prediction_points to a leave-one-out design, the
prediction mean and with it that tolerance are 0: held-out distances above
0 then read "farther", and held-out distances that are all 0 as well
read "matched". With 50 folds or fewer the points are measured to each
fold's own training rows, which leave out that fold's held-out sites, so
the mean is above 0 and the quarter rule gives the reading.
"closer"The gap is below minus the tolerance and at least half of \(W_1\) in size: the held-out points sit closer to their training data than the prediction points do, and the cross-validated error tends to read low for that target.
"farther"The gap is above the tolerance and at least half of \(W_1\): the held-out points sit farther, and the error tends to read high for that target.
"matched"The gap is within the tolerance and \(W_1\) within twice the tolerance.
"mixed"The two distributions cross, closer at some distances and farther at others (the gap is under half of \(W_1\), or it is within the tolerance while \(W_1\) is not), so the distances give no single direction.
The reading is a statement about nearest-neighbour distances, not a measured bias. A learner that draws on several neighbours can still score low on clustered samples whose nearest-neighbour distances match.
See also
make_folds() for the fold schemes and the block sizing this
measures the outcome of, plot_folds() to map the folds, and
cv_block_size_sweep() for the cross-validated error at each block size.
Other folds:
cv_block_size_sweep(),
make_folds(),
plot_folds()
Examples
library(sf)
set.seed(1)
n <- 200
pts <- st_as_sf(
data.frame(x = 5e5 + runif(n, 0, 1000), y = 5e6 + runif(n, 0, 1000)),
coords = c("x", "y"), crs = 32632
)
# Random folds put a training point close to every held-out one; blocked
# folds hold out whole neighbourhoods, so the distances grow. The median
# distance from a held-out point to its nearest training point, per fold:
random <- make_folds(pts, k = 4, method = "random_kfold", seed = 1)
blocked <- make_folds(pts, k = 4, method = "block_kfold",
block_size = 250, seed = 1)
round(fold_separation(random, pts, sac = 200)$median_dist)
#> [1] 37 42 39 46
round(fold_separation(blocked, pts, sac = 200)$median_dist)
#> [1] 112 169 118 116
# Which of the two suits a map of the sampled square? Compare each with
# the distances from the cells of that map to the training data.
cells <- st_make_grid(pts, n = c(20, 20), what = "centers")
print(fold_separation(random, pts, sac = 200, prediction_points = cells))
#> Fold separation: random_kfold, 4 fold(s), 200 held-out point(s) (EPSG:32632)
#> autocorrelation range: 200 (in metres, like the distances)
#> prediction target: 400 point(s), measured to each fold's training data
#>
#> fold n_train n_test min_dist median_dist target gap mean_cor
#> 1 150 50 4.696 36.95 37.82 -0.9056 0.57
#> 2 150 50 3.835 42.14 38.02 0.2229 0.55
#> 3 150 50 3.835 38.52 36.00 -1.8310 0.58
#> 4 150 50 4.696 45.79 37.28 3.9360 0.53
#>
#> Held-out points sit a median of 41.61 from their nearest training
#> point; the closest is 3.835 away. Under the range of 200 that leaves a
#> mean correlation of 0.56 between a held-out point and its nearest
#> training point (exp(-3 d / range) averaged over the hold-out: 1 at
#> distance 0, 0.05 at the range).
#> The prediction points sit a median of 37.48 from the same training data
#> (32.16 from the nearest of all sites). Held-out minus prediction:
#> +0.3554 in mean distance (W1 3.371); +0.01 on the correlation scale (W1
#> 0.02). The held-out points sit about as far from their training data as
#> the prediction points do. Matched nearest-neighbour distances do not
#> make a score unbiased by themselves: a learner that draws on several
#> neighbours can still read low on clustered samples.
print(fold_separation(blocked, pts, sac = 200, prediction_points = cells))
#> Fold separation: block_kfold, 4 fold(s), 200 held-out point(s) (EPSG:32632)
#> autocorrelation range: 200 (in metres, like the distances)
#> prediction target: 400 point(s), measured to each fold's training data
#>
#> fold n_train n_test n_blocks min_dist median_dist target gap mean_cor
#> 1 152 48 2 23.02 111.9 39.37 58.97 0.23
#> 2 143 57 3 29.04 169.5 44.55 89.14 0.14
#> 3 152 48 2 21.03 118.3 40.95 65.42 0.22
#> 4 153 47 2 21.03 115.9 38.61 67.19 0.23
#>
#> Held-out points sit a median of 123.5 from their nearest training
#> point; the closest is 21.03 away. Under the range of 200 that leaves a
#> mean correlation of 0.20 between a held-out point and its nearest
#> training point (exp(-3 d / range) averaged over the hold-out: 1 at
#> distance 0, 0.05 at the range).
#> The prediction points sit a median of 40.55 from the same training data
#> (32.16 from the nearest of all sites). Held-out minus prediction:
#> +71.05 in mean distance (W1 71.15); +0.30 on the correlation scale (W1
#> 0.30). The held-out points sit farther from their training data than
#> the prediction points do, so the cross-validated error tends to read
#> high for these prediction points.