Score candidate cell counts on several criteria
Source:R/resolution-profile.R
resolution_profile.RdScores candidate numbers of cells (levels) for a layer of points, so that
the count chosen, by select_resolution() or by eye, can be
defended with the whole profile in place of one criterion's optimum.
Usage
resolution_profile(
data_sf,
response_var = NULL,
predictor_vars = NULL,
levels = NULL,
n_levels = 20L,
min_cell_n = 9L,
sample_n = 1500L,
nstart = NULL,
seed = 123L,
sac = "auto",
select_on = c("all", "split"),
range_floor = FALSE,
partition = c("density", "kmeans"),
init = c("kmeans++", "random"),
density_power = 1,
density_k = 8L,
iter_max = NULL,
estimate = "auto",
obs_per_cell = "auto"
)
# S3 method for class 'resolution_profile'
print(x, digits = 3L, ...)Arguments
- data_sf
An sf object of points (other geometries are reduced to representative points with
coerce_to_points()). Features without a location, as given, once projected or in the CRS ofsac, are dropped with a warning, and a layer that mixes XY and XYZ geometries or carries M is read in x and y (see spatialkit-geometry).- response_var
Name of the response column (numeric or logical), or
NULL. Enablescpandmoran_z, and, when it is numeric, the pooled estimate the default profile scores byloo. A variogram estimated from it also gives the range floor the bounds report andreliability; it is estimated on a ladder of counts or whensacasks for it, and not on the default ladder of one size, socpandreliabilityareNAthere. Rows where it, or a predictor, is missing or non-finite stay in the geometry and are left out of the OLS fit, the RSS,cp,moran_zand the count the support ceiling divides bymin_cell_n; a logged warning gives their number, and an R warning says so when they lower the ceiling.- predictor_vars
Optional predictor column names (numeric or logical). With them,
cpscores the OLS residuals of the response on the predictors, the variogram is estimated from those residuals, andmoran_zregresses the cell means on the cell-mean predictors.- levels
Optional integer vector of level counts to score, replacing the ladder; values below 2, above the number of distinct locations, or at or above the number of points, are dropped (k-means cannot place more centres than there are distinct points, and
stats::kmeans()refuses as many centres as points). When none is left the call stops with an error, before the variogram is estimated or read, so it raises no warning about the variogram.- n_levels
Number of levels on the ladder. Default 20. The ladder has fewer when rounding puts two of them on one count, and one more when the range floor is added as a level (see "The ladder and its bounds"). Supplying it, or
min_cell_n,sample_n,levelsorrange_floor, gives that ladder in place of the ladder of sizesobs_per_cell = "auto"would set.- min_cell_n
Minimum average number of rows per cell that a level must keep (scored rows when
response_varis given: those with a response and every predictor; points otherwise); sets the ceiling. Default 9. A layer that cannot give two cells that many is profiled at two cells, or at thelevelsasked for, withsupported = FALSEand a warning. It is floored to a whole number, and it is not read on a ladderobs_per_cellsets (see "The ladder and its bounds").- sample_n
Points are subsampled to this many before the k-means fits, as in
determine_optimal_levels(), on the ladder of "The ladder and its bounds"; on a ladder of sizes, the default, the centres are fitted to every row. Default 1500. The columns read off the fitted cells (wss, thecell_andcells_columns,rss,moran_i,moran_z) describe the subsample; the bounds,supported, the variance term ofcpandreliabilitydescribe the whole layer, so the answer does not change withsample_nexcept through the fits. The fits are those of the subsample, whichseeddraws, so on a layer of more thansample_npoints the count read off the profile can move withseedand withsample_n. The levelselect_resolution()chooses off such a profile without an estimate is refitted to every point before its seeds are handed on (seerefit_rowsunder "Value"); a profile with an estimate scores it on every row in the cells of the centres fitted, and hands those centres on.- nstart
Starts per level; the best is kept. They are k-means++ starts unless
initsays otherwise.NULL(default) is the kind's own number: 3 for the kind"density"and 25 for the kind"kmeans". A number is run under eitherpartition.attr(, "nstart")holds the number that was run.- seed
RNG seed for the subsample and the restarts; restored afterwards. Default 123. The rows are put in coordinate order before either, so the profile does not depend on the order they come in. Each level's restarts run on a stream of their own, seeded from
seedand the level, so for oneseeda level gets the same cells on any ladder that holds it and is fitted to the same rows, and with them the samewss,rss,moran_zand the other columns read off its cells. On a layer of more thansample_nrows a ladder of counts is fitted to a subsample and a ladder of sizes to every row, so there the two can give a level other cells.cpalso reads its noise variance, which without a usable variogram comes from the ladder's finest level (attr(, "cp_noise")).- sac
Optional
sac_rangeobject fromestimate_sac_range()to take the range, nugget and correlation function from. Pass one fitted withdetrend = "reml", say. It must describe the variable the profile scores: the raw response withoutpredictor_vars, the residuals on them with; a sac whosedetrendedattribute says otherwise is used with a warning. Its range is read in its own CRS (attr(sac, "crs")), to which the points are transformed first, so the area, the floor andcell_diam_medianare then in that CRS's units. A sac whose range was rejected (NAwith arejected_reason) givescpits nugget (held to the semivariance of nearest-neighbour pairs, and not at all when it is at the fit's lower bound), with a warning, and leavesreliabilityNA, since that needs the range; one whose model did not converge (its fit has no minimum inside the ranges searched, so its nugget and sill are not fitted values), or whose range is below the shortest lag fitted (a structure that cannot be told from a nugget, so the nugget is not identified either), gives neither, andcptakes its noise variance from the finest level (seecpabove). Whether the model converged is read from the model (attr(attr(sac, "variogram_model"), "converged"), whichestimate_sac_range()sets and describes), not from the reason: the reason names one ground for the refusal, and such a fit is refused as past the largest lag. The warning then names both. A sac that carries no such flag (one made by hand, or built from REML parameters) is judged by its reason. Underselect_on = "split"the sac must come from the selection half alone: run the profile once without it, fit the sac ondata_sf[attr(p, "split")$selection, ]and pass it to a second call with the sameseed, which makes the same split. WhenNULLand a response is given, one is estimated on the subsample (its selection half under"split") with the samepredictor_vars. A plain number is taken as the range alone, in the units of the CRS the profile is computed in (metres for lon/lat input): it gives the range floor,reliability, which needs a fitted model, isNA, andcptakes its noise variance from the finest level whose cells hold at least two scored rows on average (seecpabove). A sac refused because the response shows no spatial structure (either of the tworejected_reasonvalues ofestimate_sac_range()that say so; the model it attaches is a pure nugget) givescpno nugget either, andselect_resolution()then returnsstructured = FALSE. Aunitsobject is refused rather than read as a number in whatever unit it was written in. The string"auto", the default, isNULLon every ladder but one thatobs_per_cellsets. On that ladder no variogram is estimated or read:cpandreliabilityareNAat every level,cptakes no noise variance from the finest level in its place,attr(, "cp_noise")is a list whosesourceis"none"and whosereasonsays why, and thevariogramandsacattributes areNULL. The other columns do not read the variogram and are the numberssac = NULLgives.sac = NULLestimates the variogram on a ladder of sizes too, and givescpandreliabilitythere.- select_on
"all"(default) profiles every point;"split"reads the response on one spatially blocked half only (the OLS fit, the variogram,rss,cpandmoran_z) and returns the other half as the set to estimate on, in the"split"attribute. The cells,wss,elbow, and the extent and point counts behind the bounds,cp's variance term andreliabilitystill come from every point, because the tessellation the count is for is built on every point: the levels are cell counts for the whole layer, and the estimation half's response never touches them. See the "Post-selection inference" section ofdetermine_optimal_levels; the profile reads the response wheneverresponse_varis given.- range_floor
FALSE(default) starts the ladder at 2 whatever the range, and the range floor is reported in the bounds and the print without being applied.TRUEstarts the ladder at the floor when the data support it and the floor is below the ceiling; an estimated range then sets the fewest cells any criterion can choose (see "The ladder and its bounds").- partition
What the centres of the cells minimise, at every level of the ladder.
"density"(default): the sum of squared distances from the points to their nearest centre, each point weighted by its local density to the powerdensity_power; compact cells that, at the default power, hold about the same number of points each once there are more cells than dense places, and counts about as uneven as"kmeans"gives below that."kmeans": the same sum unweighted; compact cells whose counts are several times higher in dense places than in sparse ones, from 25 starts undernstart = NULL. See "The partition".- init
How each start is drawn.
"kmeans++"(default): the centres one at a time, each in proportion to its squared distance from those already drawn, times its weight underpartition = "density"."random":kdistinct locations at random, askmeans()draws them.- density_power
For
partition = "density": the power the local density is raised to for a point's weight, a number from 0 to 2. Default 1, which evens out the cell counts once the cells outnumber the dense places (see "The partition").0weights every point alike, the objective of"kmeans"(reached by Lloyd's iteration, so the centres are close to that kind's and not equal to them); values between give counts between the two; values above 1 give the cells of dense places fewer points than those of sparse places. Setting it underpartition = "kmeans", which does not read it, is a warning.- density_k
For
partition = "density": the number of nearest neighbours the local density is read from. Default 8. It is held to one fewer than the distinct locations. A small number follows the density closely and noisily; a large one smooths it. Setting it underpartition = "kmeans"is a warning.- iter_max
The most iterations a start runs for: a single positive whole number, or
NULL(default) for the kind's own limit, 50 forpartition = "kmeans"from k-means++ starts and 100 for random starts and forpartition = "density". A kept start that was cut off at the limit is a warning of class"spatialkit_not_converged", except for the kind"kmeans"from k-means++ starts underiter_max = NULL, which reports nothing.- estimate
The cell value the profile scores, which is the one
summarize_by_cell()is to report:"auto"(default; see below),NULLfor none,"mean"for the plain mean of a cell's rows,"pooled"for the mean pulled toward the mean of the 8 nearest cells. With an estimate the profile gains five columns andattr(, "estimate")(see "Value"), and every column and attribute it has without one is unchanged.loois the leave-one-out mean squared error of that estimate: each row predicted by the value of its cell computed without it. Under"mean"a row in a cell of two or more is predicted by the mean of the others, and a row alone in its cell by the mean of the cell of its nearest other row. Under"pooled"the cell's mean and its own weight are computed again without the row, and the spread inside the cells, the spread between a cell and its target, and the targets are held. It is computed on every row of the layer with a finite response (of the selection half underselect_on = "split"), not on the subsample the centres were fitted to, and each row is scored in the cell of its nearest centre, the cellget_voronoi_seeds()hands on. It needs no variogram. Rows at one location are left out one at a time."auto"is"pooled"when there is a numericresponse_varwith a finite value and nopredictor_vars, andNULLotherwise.NULLscores none and gives the profile without the five columns."mean"and"pooled"are errors without aresponse_var, withpredictor_vars(the profile would then score residuals, and the summary reports cell values of the response), with a logical response, and with a response that has no finite value. Underselect_on = "split"a selection half with no finite response is not an error: the five columns areNAat every level, with a warning of classspatialkit_no_scored_rows.- obs_per_cell
Rows per cell:
"auto"(default; see below);NULLfor the ladder described under "The ladder and its bounds", whatever the other arguments are; or one number or several, each at least 1 (fractions are allowed), for a ladder of sizes. The level of a size \(s\) ismax(2, round(n / s))cells, with \(n\) the rows that have a response (and, withpredictor_vars, every predictor), or every point when there is noresponse_varor no row has a response. Underselect_on = "split"\(n\) counts the rows of both halves, since the cells are built on every point. On a ladder of sizes the centres are fitted to every row, no subsample is drawn, andn_levels,min_cell_n,sample_nandrange_floorare not read: supplyingn_levels,min_cell_n,sample_norrange_floor = TRUEbeside a numericobs_per_cellis a warning of classspatialkit_ladder_unused, andlevelsbeside it is an error. A level is held to the most centres k-means can place (the distinct locations, and one short of the points)."auto"isNULLwhenlevelsis given or the caller supplies any of those four arguments; otherwise 4.5 when the estimate is"pooled", the seven sizes 18, 12, 9, 6, 4.5, 3 and 2 when it is"mean", andNULLwithout an estimate. The cost of a ladder of sizes is bounded before any fit (see "Conditions").- x
A
resolution_profile, forprint().- digits
For
print(): significant digits for the table. Default 3.- ...
Ignored.
Value
A data.frame of class resolution_profile, one row per
level. Columns a missing input leaves undefined are NA.
levelsThe number of cells.
wssThe within-cell sum of squares of the coordinates, every point counted once, under either
partition.wss_spreadThe relative spread of WSS across the restarts; under
partition = "density", of the weighted sum the starts minimise.elbowThe sag of the WSS curve below a power law (see "The criteria").
cell_n_min,cell_n_medianThe fewest and the median points per cell.
cell_diam_medianTwice the median RMS radius of the cells, in coordinate units: about 0.8 of the side of a square cell of the same area, and less for finer cells, so a size to compare levels by rather than a width.
rss,cpThe residual sum of squares of the scored rows about their cell means, and Mallows' \(C_p\).
moran_i,moran_zMoran's I of the cell means and its standardised deviate.
reliabilityThe between-cell signal's share of the spread in the cell means.
cells_respThe cells that hold at least one scored row.
cell_resp_meanThe scored rows per such cell; compare it with
min_cell_n.
Like the other columns read off the fitted cells, cells_resp
and cell_resp_mean count rows of the subsample when the layer is
larger than sample_n: the cells of the layer then hold about
n_response / n_scored times as many scored rows as
cell_resp_mean shows (under select_on = "split",
split$n_response / n_scored: both count the rows of the
selection half).
With an estimate the frame has five more columns, after
cell_resp_mean:
scored_per_cellThe rows
looread over the cells that hold one; the layer's counterpart ofcell_resp_mean, which counts the subsample's rows.looThe leave-one-out mean squared error of the estimate; lower is better.
loo_seThe standard error over rows of that level's mean squared error. It is not the standard error of a difference between two levels, whose errors come from the same rows.
share_aloneThe share of the scored rows that are the only scored row of their cell.
own_weightThe median over cells of the weight a pooled value gives the cell's own mean;
NAunder"mean".
loo, loo_se and own_weight are NA at a level
where no cell holds two scored rows, at every level when the response
is constant over the rows loo reads, and at every level when
those rows' values lie beyond what the arithmetic of the estimate holds
(spatialkit_pool_unavailable). All five are NA at every
level when no row of the selection half has a response
(select_on = "split"; spatialkit_no_scored_rows).
Attributes:
boundsA list:
floor,ceiling,ceiling_from("min_cell_n","min_cell_n (rows with a response)","distinct locations","sample_n","fewer than two supported cells"or"fewer than two distinct locations", whichever bound it),supported,area,range,n(the points in the layer),n_sample(the points the k-means fits ran on),n_distinct,min_cell_n,range_floor,n_response(the rows of the layer that carry a response and, withpredictor_vars, every predictor, which the support ceiling counts;NAwithout aresponse_var),n_scored(the rows of the subsample the response criteria read;NAwithout aresponse_var),levels_given(whetherlevelsset the ladder),predictors(whetherpredictor_varswere given) and, where rows have a response and none of them every predictor,n_response_only(the rows with a finite response). On a ladder thatobs_per_cellsets,boundsdescribes that ladder:ceilingis the finest level;ceiling_fromis"obs_per_cell","distinct locations","fewer than two cells of the size"or"fewer than two distinct locations";supportedisFALSEwhen the finest size gives fewer than two cells or the range floor is above the finest level;levels_givenisTRUE;n_sampleis every point;min_cell_nholds the argument, which is not read; and two fields are added,obs_per_cell(the sizes, largest first) andsize_levels(the level of each, named by size;NAfor a size of"auto"that gives fewer than two cells and was left out).variogramA list with
nugget,psill,range(NAwhen rejected),modelanddetrended(whether the sac says it is a variogram of residuals;NAwhen it does not say);NULLwhen none was usable.variable"response","residuals"orNA.wss_bumps,nstartThe number of rising steps on the WSS curve, and the starts run per level.
sacThe range object used.
splitWith
select_on = "split": aspatialkit_splitwithselectionandestimation(integer row positions indata_sf), themethodandseedthat made them, and with a responsen_response, the rows of the selection half with a finite response and every predictor, counted on the layer before any subsample.cp_noiseA list with
source("variogram nugget","finest-level residual mean square"or"short-lag semivariance"),value(the \(\tau^2\) used) andlevel(the level it was read at, orNA). Under the short-lag semivariance it also hascap, that semivariance, anduncapped, a list of thesource,valueandlevelit held. When the finest level whose cells hold two scored rows on average has a residual mean square of 0 to within rounding,sourceandvalueareNA, and the list haslevel(that level),rows_per_cell(its scored rows a cell),rms(its residual mean square: 0, or the rounding it was read as) andreason, whichselect_resolution()gives. On a ladder of sizes undersac = "auto"it is a list whosesourceis"none"and whosereasonsays why.NULLwhencphas no noise variance for another reason; seecpunder "The criteria".one_cellWhat one cell for the whole layer scores, the reference a level has to beat: a list with
n(the scored rows),rss(their total sum of squares),cp(the same formula at one cell, orNAwhencphas no noise variance),constant(whether the scored variable is constant to within rounding),cp_levels(thecpcolumn it belongs to) andf_p(named by level, the p-value of the F test of that level's cell means against one mean, \((TSS - RSS) / (L - 1)\) over \(RSS / (m - L)\) on the \(L\) cells that hold a scored row and the \(m\) scored rows).NULLwithout a response.centres,centres_crs,hullA list named by level of the two-column matrices of the scored partitions' centres, the CRS they are in, and the convex hull of the points profiled (an
sfcin that CRS). A k-means partition is the Voronoi partition of its centres, so these seed the cells each level was scored on:select_resolution()returns the chosen level's as$seeds, andget_voronoi_seeds()andvoronoi_seeds_kmeans()use them when given the selection or the profile. The centres carry the hull as their"cover"attribute, which those two functions hand on so that Voronoi cells built on the seeds reach the edge of the points.refit_rowsWhen the centres were fitted to a subsample and the profile has no estimate: a list with
xy, the coordinates of every point in the CRS of the centres, anditer_max, the iteration limit of the fits, from whichselect_resolution()refits the level it chooses to every point. The seeds are then the chosen level's centres refitted to every point, whose cells are not the cells scored (seeselect_resolution, "Value").partitionOn a profile built with
partition = "density"orinit = "random": a list withkind,init,density_poweranddensity_k(bothNAforkind = "kmeans"); forkind = "density",objective, the weighted sum the centres minimise, named by level;iter_max, the iteration limit; anditerandconverged, the iterations each level's kept start took and whether it stopped before the limit, named by level. The selection carries the first four as$partition, its seeds and the seeds the seeding functions return asattr(, "partition"), and a Voronoi tessellation built on them asparams$partition. A profile of the kind"kmeans"from k-means++ starts has no such attribute, and neither has anything made from it.estimateWith an
estimate: a list withvalue("mean"or"pooled"),neighbours(8, orNAunder"mean"),response(the column),n(the rowslooread),rows("all"or"selection half"),features("points", or"other"when the layer held features that were reduced to points),seed,constant,loo_one_cellandloo_one_cell_se(the same error with one mean for all those rows, the reference a level has to beat;NAfor a constant response, for fewer than two rows and for values beyond the arithmetic of the estimate),loo_levels(theloocolumn those references belong to, named by level), and, named by level,sigma2(the spread of the rows about their cell's mean),tau2(the mean squared distance between a cell's level and its target;NAunder"mean"),cells(the cells that hold a scored row) andf_p(the p-value of the F test of those cells' means against one mean).constantsays whether the response is constant over those rows by the estimate's test, which reads the raw values and the values centred on their median and scaled, so that 1e15 plus a spread of 1 is not constant while values that differ only by their rounding, 0.1 beside 0.3 / 3, are; it isFALSEfor fewer than two rows.one_cell$constant, the test forcp, reads the scored rows of the subsample the same way, and residuals against the rounding of the regression and of the response.
Details
Called with a numeric response, no predictors and none of
levels, n_levels, min_cell_n, sample_n or
range_floor, the profile is one level: the count that gives 4.5
rows per cell (obs_per_cell = "auto"), scored for pooled values
(estimate = "auto") by the leave-one-out error loo, with
no variogram fitted (sac = "auto"), in cells of the density kind
(partition = "density") kept from 3 k-means++ starts
(nstart = NULL). cp and reliability need the
variogram and elbow needs a curve, so the three are NA
there; moran_z is computed. select_resolution()
reads the count off it by the size.
Any of those five ladder arguments, obs_per_cell = NULL, or a
call with no estimate (no response, predictor_vars, or a logical
or wholly missing response) gives a ladder of counts (see "The ladder
and its bounds"). At every level the profile then scores the
within-cluster sum of squares of the coordinates and its elbow, Mallows'
\(C_p\) of the piecewise-constant approximation of the response by
cell means, Moran's \(z\) of the cell means and their reliability, and,
with an estimate, the leave-one-out error of the cell value. Mallows
(1973) presents \(C_p\) itself as a display of the bias-variance
trade-off across candidates, and the profile is that display, with the
other criteria alongside.
print() gives a header that states the ladder and its bounds, the
variogram (or why none was fitted), what was scored and on how many
starts, what one cell for the whole layer scores, the estimate, whether
the WSS curve has an elbow and, under select_on = "split", the
rows of each half; then the table, one row per level, at digits
significant digits.
The partition
The cells at every level are the Voronoi cells of a set of centres: a
point belongs to the centre nearest to it. partition says what the
centres minimise, and every level of the ladder is built with the same
kind.
"kmeans"The sum, over the points, of the squared distance to the nearest centre. The centres follow the square root of the point density. The cells are compact, smaller where the points are dense and larger where they are sparse, and a cell in a dense place holds several times the points of a cell in a sparse place.
"density"(default)The same sum with each point weighted by its local density to the power
density_power. The local density at a point is the number of points in the disc that reaches itsdensity_k-th nearest neighbour, over the area of that disc. Withdensity_power = 1the centres follow the point density itself. The cells are compact: many small ones where the points are dense, a few large ones where they are sparse. They hold about the same number of points each once there are more cells than dense places (clusters of points). With fewer cells than dense places a cell takes in whole clusters, large and small, and the counts are about as uneven as under"kmeans".density_power = 0weights every point alike, which is the objective of"kmeans"; a power above 1 goes past equal counts and gives the cells of dense places fewer points than those of sparse places.
A ladder starts at 2 cells, so its low levels are usually below the
number of dense places, and the columns cell_n_min and
cell_n_median show where the two kinds part.
The density kind is the default. The default size, 4.5 rows per cell,
puts the count above the number of dense places on a layer whose
clusters hold more than a few observations each, so the default kind and
the default count go together. A coarse ladder the caller sets (with
levels, large sizes, or the low levels of the automatic ladder)
may not, and at those levels the density kind does not even the counts.
partition = "kmeans" gives plain k-means cells, from 25 k-means++
starts unless nstart says otherwise.
init says how each of the nstart starts is drawn before it
is refined; the best start is kept. "kmeans++" draws the centres
one at a time, each a point taken in proportion to its squared distance
from the centres already drawn (times its weight, under
"density"), which spreads the start over the points.
"random" draws k distinct locations at random, as
kmeans() does.
A start of the kind "kmeans" is refined by
kmeans() (Hartigan and Wong), for at most 50
iterations from a k-means++ start and 100 from a random one. A start of
the kind "density" is refined by Lloyd's iteration, for at most 100
steps: each point goes to its nearest centre, each centre moves to the
weighted mean of its points, and a centre left without a point stays
where it is. iter_max sets another limit. The start with the
lowest sum is kept (under the density kind a later start replaces an
earlier one only when its sum is lower by more than one part in ten
million, so of starts whose sums agree to rounding the first is kept), and
a kept start that was cut off at the limit is reported (see
"Conditions"): its centres are those of the last step. A large layer can
need more steps than the default limit.
The density kind depends on the points alone. It works on the distinct
locations in coordinate order (see below for first coordinates equal but
for their last bits), so the order of the rows does not matter. Its
tolerances do not depend on the unit of the coordinates: the neighbour
distance is not taken below a millionth of the extent of the points (the
diagonal of their bounding box), the iteration stops when no centre moves
by \(10^{-12}\) of it, and the tie rules below are fractions of the
extent or of the quantities they compare. On gridded locations many
points are equally near two neighbours or two centres, and the last bits
of the coordinates, which move with the unit and the origin, would settle
those ties. So squared distances within one part in ten million of each
other count as equal, a tie between neighbours goes to the lower first
coordinate and then the lower second, a tie between centres goes to the
lower-numbered one (between centres a squared distance ties with the least
when it is no more than the square of the least distance lengthened by a
margin, plus one part in ten million of the least squared distance; the
margin is one part in 1e10 of the largest coordinate of the centres, at
most a millionth of the diagonal of their bounding box and at least four
times the spacing of doubles at that coordinate), first coordinates
within that part of the extent count as equal when the rows and the
locations are put in order, and so do weights within that part of each
other when the first k-means++ centre is drawn. The cells are then the
same in another unit or with another origin as long as no coordinate
moves by more than about a two-hundred-millionth of the distance between
neighbouring locations, but for one case: the margin of the centres grows
with their largest coordinate (about 0.46 mm at a UTM northing near
4,550 km, about half a micrometre once the origin lies next to data 5 km
across), and the rows are given to the centres in the layer's own
coordinates, so a row whose distances to two centres differ by less than
the margin with one origin and by more with another goes to the
lower-numbered centre with the first and to the nearer with the second.
The rounding PROJ leaves when it changes the unit or the false origin,
about \(10^{-9}\) m, is below that for locations more than about a
quarter of a metre apart. Coordinates rounded more coarsely can give
other cells: a lattice at 250 m with its coordinates stored in feet to
four decimals does. Points at one location count as that many points at
it and share a cell; the neighbour distance is measured to the
density_k-th nearest other location, so it is not zero
where locations repeat. The neighbour searches use FNN when it is
installed. The result is the same without it; the search then takes
time that grows with the square of the number of points.
The kind "kmeans" meets the same ties inside
kmeans(), which settles them by the last bits of the
coordinates and takes no tolerance. So it is handed the coordinates
centred on the middle of their bounding box, divided by its diagonal and
rounded to \(2^{-30}\) of it (about a billionth); the starts are drawn
from those coordinates and ranked by their sums, and the cells it returns
are read back in the layer's own coordinates, so the centres, wss
and every column are in its unit. The rows are put in order as for the
density kind. The cells are then the same in another unit or with
another origin unless the change moves a coordinate across the middle of
a rounding step, which the rounding PROJ leaves (about \(10^{-9}\) m)
does more often the smaller the extent. Distinct locations that would
round to one point keep their coordinates unrounded, so they stay
distinct. stats::kmeans() also settles such ties by the last bits
of its own compiled arithmetic, which differ between platforms, so on a
layer where ties decide cells the kind "kmeans" can return other
cells on another platform from the same coordinates and seed, and
with them every column read off the cells.
The columns keep their meaning under "density", because each is
read off the cells and not off the way they were drawn. wss is the
within-cell sum of squares of the coordinates with every point counted
once, taken about each cell's mean, so wss and elbow
compare between kinds; the weighted sum the centres minimise is kept in
attr(, "partition")$objective, and wss_spread is its spread
over the starts. cp, moran_z and the test against one cell
need cells drawn without the response, which both kinds give.
reliability is computed for square cells of the level's average
area and count and never reads the cells, so it is the same number under
either kind. The elbow is found more readily under
"density" where the points cluster: once each cluster has its
centres, further centres go into the clusters, the sparse ground between
them keeps its sum of squares, and the curve flattens.
A profile of the density kind, or of either kind from random starts,
carries attr(, "partition"), and
select_resolution(), the seeding functions and
build_tessellation() hand that record on with the cells
(see "Value"). A result with no such record was built by plain k-means:
from k-means++ starts when it comes from a profile, and from random
starts when a seeding function fitted it.
The ladder and its bounds
This is the ladder of a call that sets levels, n_levels,
min_cell_n, sample_n or range_floor, or passes
obs_per_cell = NULL, and of a call with no estimate: one without
a response, with predictor_vars, or with a logical or wholly
missing response. Cells are clusters of the (projected)
coordinates of the kind partition names, fitted at each level as
the best of nstart k-means++ starts (see
determine_optimal_levels for why) unless init says
otherwise (see "The partition"). Levels are spaced
logarithmically, because cell diameter scales as \(L^{-1/2}\): a unit
step wastes fits at large \(L\) and starves resolution at small. The
ladder runs from 2 to a ceiling the data impose. The ceiling is
floor(n / min_cell_n): cells with fewer than min_cell_n
rows on average have too little support. With a response_var,
\(n\) is the number of rows of the layer (not of the subsample) that
are scored: those with a response and, with predictor_vars, every
predictor, since the cell means the criteria read are built from those
rows alone. ceiling_from is then "min_cell_n (rows with a
response)" when some rows are not scored (the label is the same when it
is a predictor they lack), and a warning of class
spatialkit_response_ceiling gives both counts and names the bound
in force. Without a
response_var, or when no row has a response, \(n\) is every
point. The ceiling is also held to the number of distinct locations, one
cell on each, and to one short of the number of points, which
stats::kmeans() needs. The count \(n / L\) is the rows per cell
only when the scored rows are spread over the whole extent.
Where they sit in one part of it, the cells elsewhere hold none and the
occupied cells hold more than \(n / L\), so the ceiling is lower than
those cells need: cells_resp and cell_resp_mean give the
cells that hold a scored row and the scored rows per such cell at each
level (of the subsample, when the cells are fitted to one), and
profiling the scored rows alone bounds the ladder by the area they
cover.
When fewer than two cells meet min_cell_n
(floor(n / min_cell_n) < 2), or the layer has one distinct
location, there is no ladder: the profile scores two cells so that level
is still visible (or the levels asked for, as given),
supported is FALSE, ceiling_from
is "fewer than two supported cells" (or "fewer than two
distinct locations") and a warning of class
spatialkit_not_supported names both numbers.
min_cell_n sets the ceiling of this ladder. It is a mean number
of rows per cell and not a minimum: a cell of a level at the ceiling can
hold fewer (cell_n_min). It is floored to a whole number, so 4.5
is read as 4. With levels it does not choose the levels scored,
and it still sets bounds$ceiling, supported and the
spatialkit_not_supported warning.
The range floor is ceiling(area / range^2) when an autocorrelation
range is available: cells wider than the range average over more than one
patch of the field. It is reported in the bounds and the print, and by
default it is not applied: the ladder starts at 2. When the layer
supports the floor and it lies between the ladder's ends, the floor is
one of the ladder's levels all the same (so the ladder can have one more
level than n_levels): select_resolution(criterion =
"reliability") reads from the floor up and returns the floor itself, not
the next level the spacing happened to put above it. The range is an
estimate, and a response with no spatial structure can still be given
one, so a ladder started at the floor can leave out the coarse counts
such a response supports. range_floor = TRUE starts the ladder
at the floor when the layer supports it and it lies below the ceiling.
When
the floor exceeds the ceiling the data cannot support a tessellation
that respects their own correlation structure; that is reported as a
finding (a warning of class spatialkit_not_supported, and
attr(x, "bounds")$supported is FALSE) and the ladder runs
from 2 to the ceiling (or the levels asked for are scored, as
given), so the profile still shows what each level costs.
A floor equal to the ceiling would leave one level and nothing to compare
it with, so that ladder starts at 2 as well. On a layer larger than
sample_n the ceiling is also held to half the subsample's scored
rows (two per cell, the least a fitted cell can be scored on; half the
subsample without a response); ceiling_from is then
"sample_n", a floor above that is logged with a request to raise
sample_n, and it does not make supported FALSE.
The floor moves with the range estimate, and the ceiling with \(n\).
To compare profiles of similar data (the folds of a cross-validation,
say), pass the same levels to each. reliability is the one
criterion that still reads the floor: select_resolution()
holds it to the levels at or above the floor by default, so its pick can
differ widely between two such profiles when the floor is supported on
one and not on the other; pass range_floor = FALSE
there to read the whole ladder.
A ladder that obs_per_cell sets is a ladder of sizes (see
"Observations per cell"). It has one level per size \(s\),
max(2, round(n / s)) cells, held to the most centres k-means can
place: the distinct locations, and one short of the points. Its centres
are fitted to every row. No subsample is drawn, and n_levels,
min_cell_n, sample_n and range_floor are not read.
The finest size takes the part min_cell_n has on the ladder above:
bounds$ceiling is the finest level, and supported is
FALSE when the finest size gives fewer than two cells or the range
floor is above the finest level. The range floor is reported and not
applied.
A ladder of sizes under sac = "auto" fits no variogram, so
cp and reliability are NA at every level there, and
attr(, "cp_noise") says why. sac = NULL, or a sac of
the caller's own, gives them. sac = "auto" and
obs_per_cell = "auto" are the defaults, so the default profile
has neither column.
Observations per cell
obs_per_cell sets the ladder by a size: the mean number of rows a
cell is to hold. The level of a size \(s\) is
max(2, round(n / s)) cells, with \(n\) the rows the size
divides (see the argument). A size says how many observations a cell
value is to stand on, and that is the caller's choice: no criterion makes
it. Under a size neither the cells, which are fitted to the coordinates,
nor their number, which is a count of rows, depend on the values of the
response. Under select_resolution(criterion = "loo") the count
does.
For pooled values the leave-one-out error tends to keep falling as the
cells shrink, so its minimum tends to be the finest size offered, and the
end of the ladder would then choose the count (see "How this was
checked"). select_resolution(criterion = "size") therefore sets
the count of pooled values from a size, 4.5 unless another is given, and
criterion = "loo" reads the lowest error for plain means.
Two columns say what a size leaves a cell value to stand on.
share_alone is the share of the scored rows that are the only
scored row of their cell: the plain mean of such a cell is one
observation, and its pooled value has the smallest own weight a cell can
have. own_weight is the median weight a pooled value gives the
cell's own mean: at 1 the pooled values are the plain means, and at 0
each is the mean of the cell's neighbours.
obs_per_cell = "auto", the default, is resolved against the other
ladder arguments, in this order. Whether the caller supplied an argument
is what counts, so a value a wrapper passes on counts as supplied.
With
levels,"auto"isNULL: the levels are scored as given. A numericobs_per_cellbesidelevelsis an error, since each sets the ladder.With
n_levels,min_cell_n,sample_norrange_floorsupplied,"auto"isNULL: those arguments describe the ladder of "The ladder and its bounds", and the caller gets it, with the estimate's columns when there is an estimate.Otherwise
"auto"is 4.5 when the estimate is"pooled", the seven sizes 18, 12, 9, 6, 4.5, 3 and 2 when it is"mean", andNULLwithout an estimate.A numeric
obs_per_cellbesiden_levels,min_cell_n,sample_norrange_floor = TRUEsets the ladder, with a warning of classspatialkit_ladder_unusedthat names the arguments that are not read.
On a ladder of sizes the centres are fitted to every row and the count of
cells grows with the rows, so a step that compares every point with every
centre grows with the square of the rows. partition = "kmeans"
has such a step in every pass and in a k-means++ start. partition
= "density" has it in a k-means++ start and, without FNN, in every
pass; with FNN and init = "random" it has none. The profile
stops before any fit, with an error of class
spatialkit_work_limit, when the points times the cells of the
finest level times nstart (times iter_max for the density
kind without FNN) is above
getOption("spatialkit.fit_work_max", 1e10). The message names the
ways on: fewer starts, which keep the best of fewer; a coarser size;
partition = "density" with init = "random", which leaves a
cell without a point more often (see spatialkit_empty_cells under
"Conditions") and, where FNN could not be loaded, needs it
reinstalled first; or a higher limit. The work and the limit are written
in full, so that a work just above the limit does not print as the limit.
The limit counts the finest level, and a ladder of several sizes fits
every one of its levels. At the default size the work grows with the
square of the rows, and a call just below the limit runs for a long time
and prints nothing while it runs. spatialkit-options describes
both limits.
Without FNN the density kind also finds its weights by comparing
every pair of distinct locations. A profile on a ladder of sizes stops,
with the same class, when the square of their number is above
getOption("spatialkit.exact_pairs_max", 2e9), which is about
44,700 locations. The result of every search is the same with and
without the package. moran_i and moran_z need FNN
and Matrix at a level where more than 5,000 cells hold a scored
row. Without them a ladder of sizes has NA in both columns
there, with a warning of class spatialkit_moran_skipped, and so
does a ladder set any other way (see "Conditions").
At a level of up to 5,000 such cells, when either package is missing, the
neighbour weights of the cell means are held as a dense matrix of cells
by cells.
The criteria
select_resolution() reads cp by default on a profile
without an estimate; on one with an estimate it reads the size (pooled
values) or loo (plain means). reliability
needs a variogram range and is NA without one; elbow is
NA unless the points cluster; and moran_z is a diagnostic
of the cell means, which no function reads as a count rule by default.
With an estimate the profile has a fifth column, loo, which
is the last entry. Each entry below says what the column measures and
what it returns.
elbowHow far the WSS curve sags below a power law: on log-log axes, \(\log\) WSS below the straight line from \(k = 1\) (the total sum of squares) to the last level, in natural-log units (larger is better). Points with no cluster structure have a WSS close to \(c/k\), which is straight on those axes, so the column is
NAat every level unless the largest sag reaches \(\log 1.25\), and the print says there is no elbow (the rule is given indetermine_optimal_levels). The classical chord on linear axes found a "knee" on such a layer anyway, at about \(\sqrt{L_{first} L_{last}}\), where the ladder's ends put it. A level whose WSS is 0 (to within \(10^{-12}\) of the total), one cell on every distinct location, which the ladder reaches when locations repeat, is left out of the line; when the other levels have no elbow, that fall to zero is the elbow, and its sag is measured with the WSS floored at \(10^{-12}\) of the total, which puts it far above any other level's. Geometry only; it knows nothing of the response. It isNAunless the points cluster, and where they do it names the number of clusters, which is usually the first level or two of the ladder.cpMallows' \(C_p\) of the piecewise-constant approximation of the response (or of its OLS residuals on
predictor_vars) by cell means: \(RSS(L)/n + 2 \tau^2 L / n\), with \(\tau^2\) the nugget of the fitted variogram (lower is better). It estimates the error of re-measuring the sampled sites: each scored row observed again at its own location, with fresh noise of variance \(\tau^2\), and predicted by the mean of its cell. For an observation at a new location the penalty needs the spread of the field inside a cell as well, which the nugget leaves out. This \(\tau^2\) is the nugget, the noise of one measurement; it is not thetau2of a pooled value, which is the mean squared distance between a cell's level and its target. When the cells are fitted to a subsample of \(m\) of the \(N\) points with a response, the penalty is split between the two: \(RSS(L)/m + \tau^2 L_m / m + \tau^2 L / N\), where the first two terms estimate the approximation error from the subsample (adding back the optimism of its own cell means, over the \(L_m\) cells its scored points fall in) and the last is the variance of cell means built from all \(N\), which is what the tessellation will carry. With no subsample it is the formula above. On a smooth field with little noise the approximation keeps improving as cells shrink and the penalty is too small to stop it, so the minimum of \(C_p\) is the support ceiling andmin_cell_nis what is choosing;select_resolution()says so when that happens. \(C_p\) has an interior minimum when the nugget is a large share of the sill. A nugget of 0, or of at most \(10^{-4}\) of the sill, whichestimate_sac_range()marks asnugget_at_bound, is the fit's lower bound and not an estimate, and as \(\tau^2\) it would leave no penalty, so that \(C_p\) would descend to the ceiling whatever the field: \(C_p\) does not take it. When the variogram gives no nugget (none was fitted, its nugget is at that bound, its fit has no minimum inside the ranges searched, its range is below the shortest lag, it found no spatial structure, orsacis a range alone), \(\tau^2\) is Mallows' own choice instead: the residual mean square \(RSS / (m - L_m)\) of the finest level whose cells hold at least two scored rows on average. It counts the structure within those cells as noise too, so on average it is no smaller than the nugget and errs towards fewer cells. When no level's cells hold that many, or the finest that does has a residual mean square of 0 to within the rounding of the values scored (as when each of its cells holds one location whose rows carry one value),cpisNAat every level, as it is for a constant response, which the profile warns about. Whatever its source, \(\tau^2\) is held to the semivariance of nearest-neighbour pairs the variogram carries (attr(sac, "short_lag_semivariance"), seesac_nugget): a semivariance does not fall as the lag shrinks, so it bounds the nugget from above, and a larger \(\tau^2\) is contradicted by the closest pairs and stops \(C_p\) early. A semivariance of 0 holds nothing. When \(\tau^2\) is not the nugget as it stands, a warning of classspatialkit_cp_noisesays where it came from, andattr(, "cp_noise")records the source and, when the semivariance held it, the number it replaced. Its minimum is often a bound, the support ceiling when the nugget is a small share of the sill, andselect_resolution()then names the bound inedge. The same formula at one cell, with the total sum of squares as its RSS, is kept inattr(, "one_cell"), with the p-value at every level of the F test of that level's cell means against one mean. A level has to beat one cell:select_resolution()returnsstructured = FALSEwhen \(C_p\) at one cell is withintolof the optimum, or when the cell means at the optimum differ from one mean by no more than chance allows, because one mean for the whole layer then predicts as well as the cells and the count is not identified (seeselect_resolutionfor the rule).moran_zA diagnostic of the cell means: the standardised deviate of Moran's I on the cell means, or with
predictor_varson the residuals of the cell means regressed on the cell-mean predictors, over the 8 nearest neighbours of each cell. It tests whether neighbouring cell means are independent. It does not measure what the cells leave unexplained, and it does not say how well they represent the response. On any autocorrelated response it grows as the cells get finer than the range, because neighbouring cells then resemble each other, and it has almost no power below about 20 cells, where the 8-neighbour graph between cells is close to complete and Moran's I sits near \(-1/(L - 1)\) whatever the data. The level with the smallest \(|z|\) is therefore among the coarsest it is computed at on a structured response, whatever the layer and its ceiling, and arbitrary on a response with no structure, where \(z\) is calibrated and flat in \(L\).select_resolution()refuses to read it as a criterion unless asked withdiagnostic = TRUE, and neithersummary()nor the functions that take a profile as a cell count read it by default.NAat nine cells or fewer.reliabilityThe between-cell signal's share of the spread in the cell means, from the fitted variogram alone via Krige's additivity relation (Cressie 1996), for square cells of the level's average area holding the level's average share of the layer's points with a response (larger is better). This is the shrinkage factor of Fay and Herriot (1979). The curve has an interior optimum, a broad one, which can lie below the range floor.
select_resolution()reads it at the levels at or above the floor (itsrange_floorargument), and what it returns is then often the floor, which is a level of every ladder the profile builds: the coarsest count the range allows, with the range choosing rather than the curve. Report the flat region with the count. The domain term is taken over the convex hull the area is measured on, so rotating the layer does not move it.NAwithout a usable variogram, and when the variogram's range was rejected (seesac).looWith an
estimate: the leave-one-out mean squared error of that cell value (lower is better). It estimates the squared error of predicting one further observation, at a location drawn the way the data's locations were, by the value of the cell it falls in. That error depends on the cell value, so the column is computed for the estimate the profile is given and for no other. Each scored row of the layer is predicted by the value of its cell computed without it, in the cell of its nearest centre, which is the cell the seeds will build. Under"mean"a row in a cell of two or more is predicted by the mean of the others, and a row alone in its cell by the mean of the cell of its nearest other scored row. Under"pooled"the cell's mean and its own weight are computed again without the row; the spread inside the cells, the spread between a cell and its target and the targets are held; and a cell left empty takes its target. It needs no variogram. It describes locations among the observations and says nothing of a location away from them: it is a criterion for comparing levels where data are near, not an estimate of the error of a map.loo_seis the standard error, over rows, of that level's mean squared error. It is not the standard error of a difference between two levels, whose errors come from the same rows.scored_per_cellcounts the rowslooread, every scored row of the layer, over the cells that hold one;cell_resp_meancounts the rows the other response criteria read, which are the subsample's when the cells are fitted to one. For pooled values its minimum tends to be the finest size offered (see "How this was checked"), so a size sets the count, andselect_resolution(criterion = "size")reads it.
cp and reliability answer different questions: how well the
cells represent the field, and whether the cell values are distinguishable
from noise. They can disagree, and the profile shows both.
Conditions
What changes the meaning of a result is raised as an R warning with a
class, so tryCatch(), withCallingHandlers() and
options(warn = 2) see it without matching message text; the rest
is logged (see spatialkit_quiet). These are the classes
this function raises; spatialkit-conditions lists every class the
package raises.
spatialkit_not_supportedsupportedisFALSE, because the range floor exceeds the ceiling or because fewer than two cells meetmin_cell_n.spatialkit_response_ceilingRows that are not scored (no response, or with
predictor_varsa predictor missing) lowered the ceiling. The message names the bound in force,min_cell_nor, on a subsample,sample_n; the condition carriesn,n_response,ceiling,ceiling_from,n_sampleandn_scored.spatialkit_constant_responseThe scored variable is constant to within rounding (a response the predictors fit exactly leaves such residuals), so
cp,moran_iandmoran_zareNAat every level. On a profile with anestimatewhose rows are constant too, so arelooandloo_se, and for pooled valuesown_weight; the message names them.summary()of such a profile stops with an error that says the response is constant. Both tests read the raw values and, where those show no spread, the values centred on their median and scaled, so 1e15 plus a spread of 1 is not constant while values that differ only by their rounding (0.1 beside 0.3 / 3) are; residuals are read against the rounding of the regression and of the response, and where the residuals of the raw values show no spread, they are those of a fit to the response less its median. The test forcpreads the scored rows of the subsample and the estimate's every row of the layer (of its selection half underselect_on = "split") with a value: where the first finds the response constant and the second does not, the message says thatloois scored, and the print does not call the response a constant.spatialkit_empty_cellsUnder
partition = "density": a centre holds no point, so there are fewer occupied cells than the count. A centre that loses its points stays where it is, and the start that is kept is the one with the lowest sum, whether or not all its centres hold points. It happens at high counts, mostly frominit = "random". The condition carrieslevel, the count, andempty.spatialkit_not_convergedThe start that was kept was cut off at the iteration limit, so its centres are those of the last step and a longer run can still move points between cells. Not raised for the kind
"kmeans"from k-means++ starts underiter_max = NULL. The condition carrieslevel(every level it happened at),iteranditer_max. Raiseiter_max.spatialkit_ladder_unusedA numeric
obs_per_cellsets the ladder, and the caller also suppliedn_levels,min_cell_n,sample_norrange_floor = TRUE, which are then not read. The condition carriesunused(the argument names) andobs_per_cell.spatialkit_size_not_reachedOn a ladder of sizes: a size does not give the count it asks for. Either a size given as a number gives fewer than two cells while a finer one does not, and its level is 2 (
reasonis"fewer than two cells"); or a size asks for more cells than k-means can place, and its level is the most that can (reasonis"more cells than can be placed"). It is raised once for each reason, naming every size held for it. The condition carriesobs_per_cellandwanted(one element for each such size, in the order of the ladder),usedandreason. When the finest size gives fewer than two cells the warning isspatialkit_not_supported, withobs_per_cellamong its fields andmin_cell_nNA; andspatialkit_response_ceilingis not raised on such a ladder, since a size already counts the rows with a response.spatialkit_repeated_locationsWith an
estimate: scored rows share a location. Each row is left out alone and the others at its location stay, soloothen describes a further observation at a place already observed. Raised once, after the levels are scored, and not whenlooisNAat every level. The condition carriesn(the rows that share a location),total(the rowslooread) andcolumn.spatialkit_no_scored_rowsUnder
select_on = "split": no row of the selection half has a response, and the rows of the layer with one are all in the estimation half.rss,cp,moran_iandmoran_z, which read the selection half, areNAat every level. With anestimate, whose check that it has rows to read is made on the layer,looand the estimate's other columns read no row and areNAat every level too. Withpredictor_varsa row counts when it has a response and every predictor. Not raised when no row of the layer has a response. Raised once, after the levels are scored. The condition carriesreason("no response in the selection half"),n(0, the rows of the selection half scored),selection(the rows of the selection half),n_response(the rows of the layer with a response) andcolumn. Raised too, with or without the split, when rows of the layer have a response and none of them every predictor: thenreasonis"no row with a response and every predictor",n_responsethe rows with a finite response, andselectionNAwithout the split.spatialkit_pool_unavailableWith an
estimate: the values of the response lie beyond what the arithmetic of the estimate holds, values so large, or so close to their median, that the squares of many such values can overflow or lose their precision, soloo,loo_seandown_weightareNAat every level. Where the test forcpfinds such a response constant,cp,moran_iandmoran_zareNAfor the same reason, and this message names them in place ofspatialkit_constant_response. The message gives the bounds; the response rescaled by a power of ten is scored. Raised once, after the levels are scored. The condition carriesreason("values out of range") andcolumns, the response, assummarize_by_cell(estimate = "pooled")raises the class for its pooled columns.spatialkit_moran_skippedWith a level of more than 5,000 cells, on any ladder, when FNN cannot be loaded or Matrix is not installed:
moran_iandmoran_zareNAat a level where more than 5,000 cells hold a scored row, and every other column is computed. Raised once, before the first fit, when such a level can occur: the cells that hold a scored row are counted after a level's fit, so where rows without a response leave fewer of them the two columns are computed after all. The condition carrieslimit,levels(the levels above the limit) andmissing.spatialkit_variogram_unusableThe variogram, the one estimated or
sac, gives no usable range (rejected_reasonofestimate_sac_range()), soreliabilityisNA, and the message says whethercpmay still take its nugget. The condition carriesreason,converged(whether the variogram model converged;NAwhen it does not say),nuggetandnugget_used.spatialkit_cp_noisecpdoes not take the variogram's nugget as it stands. Its noise variance is the residual mean square of the finest level (the variogram gives no nugget, or its nugget is at the fit's lower bound), or the semivariance of nearest-neighbour pairs, which holds the nugget or that residual mean square where they are larger; or there is none andcpisNAat every level: the finest level whose cells hold two scored rows on average has a residual mean square of 0 to within rounding, or, with the nugget at its bound, no level's cells hold that many. The condition carriessource,valueandlevel(asattr(, "cp_noise")holds them; with no noise variancesourceandvalueareNA, andlevelis the level whose residual mean square is 0 to within rounding, orNA),cap(the semivariance,NAwhen the variogram carries none),uncapped_sourceanduncapped_value(whatcpwould take without the hold, and with no noise variance what it refused: the nugget at its bound, or that residual mean square).
The other warnings of this function (points dropped, a variogram of another variable) are R warnings without a class; rows left out for a missing response, a rising WSS curve and a subsample too small to reach the floor are logged only.
One refusal is an error with a class, spatialkit_work_limit:
resolution_profile() stops before a step whose count of
comparisons is above a limit. The condition carries work (the
count), limit and what (the step). On a ladder that
obs_per_cell sets, the count of cells grows with the rows, so two
costs are bounded before the variogram is estimated and before any fit.
The fits: the points times the cells of
the finest level times nstart may not pass
getOption("spatialkit.fit_work_max", 1e10)
(what = "partition fit"; the condition also carries kind,
init, nstart and fnn). That product counts the
step in which every point is compared with every centre, which
partition = "kmeans" has in every pass and in a k-means++ start,
and partition = "density" has in a k-means++ start. Without
FNN the density kind compares every point with every centre in
every pass, and the product is multiplied by iter_max; with
FNN and init = "random" it has no such step and nothing is
counted. The density kind's weights: without FNN they come from a
search that compares every pair of distinct locations, and the square of
their number may not pass getOption("spatialkit.exact_pairs_max",
2e9) (what = "density weights"); density_power = 0 has no
weights and no such search. With an estimate and without
FNN, on any ladder, the same limit bounds the three exact searches
the estimate's columns make: each scored row's nearest centre, each cell's
nearest cells, and, under "mean", the nearest other location of a
row alone in its cell. The result of every search is the same with and
without FNN.
The warnings of the pooled values and of the count record, which
summarize_by_cell() raises, are listed on its page.
How this was checked
The sizes for pooled values were compared in a simulation of 200 layers,
each 1,500 clustered observations of a field with covariance
exp(-d / 100) plus noise on a square of side 1,000
(dev/sim-pooled-values.R
in the package's GitHub repository; summarize_by_cell
describes it), profiled at the sizes 18, 12, 9, 6, 4.5, 3 and 2 with
partition = "density". The lowest loo of the pooled value
was at the size 2 on 176 layers, at 3 on 22 and at 4.5 on 2. That of the
plain mean was at 2 on none: it was at 6 on 86 layers, at 4.5 on 51, at 9
on 49, at 12 on 12, and at 18 and at 3 on one each. These figures are of
those seven sizes and say nothing of another ladder.
References
Cressie, N. (1996). Change of support and the modifiable areal unit problem. Geographical Systems, 3(2–3), 159–180.
Fay, R. E. and Herriot, R. A. (1979). Estimates of income for small places: an application of James-Stein procedures to census data. Journal of the American Statistical Association, 74(366), 269–277. doi:10.1080/01621459.1979.10482505
Mallows, C. L. (1973). Some comments on \(C_p\). Technometrics, 15(4), 661–675. doi:10.1080/00401706.1973.10489103
See also
select_resolution() to read a level and its flat
region off the profile; summary.resolution_profile() for
every criterion's pick side by side;
determine_optimal_levels() for a count from the
coordinates alone; build_tessellation() and
get_voronoi_seeds(), which accept the profile or a
select_resolution() result directly as approx_n_cells
and n.
Other resolution:
determine_optimal_levels(),
plot.resolution_profile(),
select_resolution(),
summary.resolution_profile()
Examples
library(sf)
# An exponential field with range parameter 200 (true effective range
# 600 m) on a 1 km square, with a nugget of 0.6 on a unit sill.
set.seed(4)
n <- 400
xy <- data.frame(x = 5e5 + runif(n, 0, 1000), y = 5e6 + runif(n, 0, 1000))
D <- as.matrix(dist(xy))
xy$z <- as.numeric(t(chol(exp(-D / 200) + diag(0.6, n))) %*% rnorm(n))
pts <- st_as_sf(xy, coords = c("x", "y"), crs = 32632)
# The default profile: one level at 4.5 rows per cell, scored for
# pooled values, in density-weighted cells, with no variogram.
prof1 <- resolution_profile(pts, response_var = "z")
prof1
#> Resolution profile: 1 level on 400 points
#> ladder : 89 cells (obs_per_cell = 4.5 on the 400 rows with a response; centres fitted to every row; floor 2 (no range))
#> variogram : none fitted: a size ladder fits one only when `sac` is given (cp and reliability are NA)
#> scored on : response; 3 density-weighted k-means++ starts (power 1, 8 neighbours) per level
#> one cell : rss 490 (one mean for the layer; a level has to beat it)
#> estimate : pooled toward the 8 nearest cells; loo on 400 rows of the layer; one cell: loo 1.23
#>
#> levels wss wss_spread elbow cell_n_min cell_n_median cell_diam_median rss
#> 89 502000 0.0698 NA 1 4 64 255
#> cp moran_i moran_z reliability cells_resp cell_resp_mean scored_per_cell loo
#> NA 0.389 8.28 NA 89 4.49 4.49 0.903
#> loo_se share_alone own_weight
#> 0.0645 0.01 0.397
select_resolution(prof1) # the count the size gives
#> Resolution by size: 89 cells
#> size : 4.5 rows per cell, the default (obs_per_cell); 400 rows / 89 cells = 4.49
#> cells : density-weighted k-means++ starts (power 1, 8 neighbours)
#> estimate : pooled toward the 8 nearest cells
#> at this size: 1.0% of the rows are alone in their cell; median own weight 0.40
#> loo : 0.903 (one cell for the layer: 1.23)
#> note : the count is set by obs_per_cell = 4.5, the default; no
#> criterion chose it.
# A ladder of counts for plain means: n_levels asks for the ladder and
# estimate = NULL scores no cell value, so cp is the rule that goes
# with it.
prof <- resolution_profile(pts, response_var = "z", n_levels = 12,
estimate = NULL)
prof
#> Resolution profile: 12 levels on 400 points
#> ladder : 2 to 44 cells (floor 4 from range 516.8, not applied, ceiling 44 from min_cell_n = 9)
#> variogram : nugget 0.582, partial sill 0.819, range 516.8
#> scored on : response; 3 density-weighted k-means++ starts (power 1, 8 neighbours) per level
#> one cell : rss 490, cp 1.23 (one mean for the layer; a level has to beat it)
#> elbow : none; the WSS curve falls as it does with no cluster structure
#>
#> levels wss wss_spread elbow cell_n_min cell_n_median cell_diam_median rss
#> 2 41800000 0.000000 NA 182 200.0 646 482
#> 3 25400000 0.268000 NA 122 134.0 473 480
#> 4 16900000 0.000112 NA 77 106.0 410 469
#> 5 14100000 0.048400 NA 65 74.0 373 445
#> 6 11600000 0.125000 NA 53 68.5 347 432
#> 8 8550000 0.057700 NA 39 51.0 288 413
#> 11 5950000 0.084200 NA 28 36.0 239 394
#> 14 4430000 0.084100 NA 19 28.0 207 367
#> 19 3360000 0.028100 NA 11 22.0 168 343
#> 25 2530000 0.178000 NA 7 16.0 155 338
#> 33 1790000 0.110000 NA 7 12.0 130 348
#> 44 1250000 0.063900 NA 3 9.0 106 314
#> cp moran_i moran_z reliability cells_resp cell_resp_mean
#> 1.210 NA NA 0.839 2 200.00
#> 1.210 NA NA 0.861 3 133.00
#> 1.180 NA NA 0.863 4 100.00
#> 1.130 NA NA 0.860 5 80.00
#> 1.100 NA NA 0.855 6 66.70
#> 1.060 NA NA 0.844 8 50.00
#> 1.020 -0.0248 1.400 0.827 11 36.40
#> 0.957 -0.0668 0.137 0.810 14 28.60
#> 0.914 0.0123 0.915 0.784 19 21.10
#> 0.918 0.0142 0.743 0.756 25 16.00
#> 0.965 0.1870 3.110 0.722 33 12.10
#> 0.914 0.2490 4.240 0.681 44 9.09
# Three of the criteria, one row per level.
as.data.frame(prof)[, c("levels", "cp", "reliability", "moran_z")]
#> levels cp reliability moran_z
#> 1 2 1.2097988 0.8393165 NA
#> 2 3 1.2097026 0.8607721 NA
#> 3 4 1.1829553 0.8626250 NA
#> 4 5 1.1273510 0.8597290 NA
#> 5 6 1.0976257 0.8551213 NA
#> 6 8 1.0550201 0.8441615 NA
#> 7 11 1.0158319 0.8269075 1.4018098
#> 8 14 0.9569884 0.8101464 0.1367780
#> 9 19 0.9136533 0.7840438 0.9149028
#> 10 25 0.9183700 0.7556101 0.7431945
#> 11 33 0.9648884 0.7217720 3.1141763
#> 12 44 0.9139798 0.6811883 4.2441007
attr(prof, "bounds")$floor # the range floor: reported, not applied
#> [1] 4
select_resolution(prof, criterion = "cp")
#> Resolution by cp: 19 cells
#> flat region : 19 to 25, 44 (3 of 12 levels)
#> cells : density-weighted k-means++ starts (power 1, 8 neighbours)