Skip to contents

Aggregates an sf point dataset into one row per cell. Each cell gets a count, the mean of each response and predictor column (or the functions in agg_funs), and that column's within-cell standard deviation and standard error. By default each numeric column also gets a pooled value, the cell's mean pulled toward the mean of the cells around it (estimate = "pooled"; estimate = "mean" leaves it out). On request the frame adds an interval for each mean (conf_level) and an interval for one further observation (obs_level).

Usage

summarize_by_cell(
  assigned_points_sf,
  response_var = NULL,
  predictor_vars = NULL,
  id_col = "poly_id",
  agg_funs = list(mean = function(x) mean(x, na.rm = TRUE)),
  cells_sf = NULL,
  deff = 1,
  sac = NULL,
  deff_max_n = 500L,
  quiet = TRUE,
  conf_level = NULL,
  area = FALSE,
  estimate = c("pooled", "mean"),
  obs_level = NULL
)

Arguments

assigned_points_sf

An sf object with a cell identifier column.

response_var

Optional response column name for per-cell aggregation: a single character string. A name that is not a column, or a column that is not numeric, is skipped with a warning.

predictor_vars

Optional predictor column names for per-cell aggregation. Names that are not columns, and columns that are not numeric, are skipped with a warning.

id_col

Name of the cell ID column. Default "poly_id". When assigned_points_sf has no column of that name, the first of "poly_id", "polygon_id" and "cell_id" it carries is used; the message that names the column shows only with quiet = FALSE. The column of cells_sf is found as cells_sf says.

agg_funs

Named list of aggregation functions. Default list(mean = \(x) mean(x, na.rm = TRUE)). Additional common options: median, sum, sd. A single function (agg_funs = median) or a character vector of function names (c("median", "sum")) is also accepted. A single function is named after the expression passed: a name gives that name (median, or f for a variable f holding a function), stats::median gives median, and any other expression gives agg1; pass a named list to choose the name. Anything else falls back to the default mean with a warning. The functions receive each column with its infinite values already set to NA (see "Output columns"), so they need na.rm = TRUE as they do for missing values.

cells_sf

Optional polygon sf layer to join cell geometries onto the output. When supplied, the return value is an sf object with the polygon geometry from cells_sf, with one row per cell in cells_sf. Cells that no feature fell in are kept, with NA summaries. Duplicate ID values in cells_sf would multiply those rows, so they are reported with a warning. Its ID column is the first of id_col, "poly_id", "polygon_id", "id", "cell_id" and "grid_id" it carries, the list and order assign_features_to_polygons() reads the polygons' IDs from. A summarised ID that matches no cell is reported with a warning, since the join drops it with its points; a cells_sf with none of those columns, or one that is not an sf object, gives a warning and a plain data frame (an error with area = TRUE). When NULL (default), a plain data.frame/tibble is returned.

deff

Design-effect adjustment for standard errors. One of:

1 (default)

No adjustment; the classic IID standard error sd / sqrt(n), which assumes the observations within a cell are independent. No "deff_applied" attribute is attached, so a result carrying none was computed this way.

"variogram"

Compute a per-cell design effect from a fitted variogram: for n points in a cell with correlation matrix R, the effective sample size of the mean is n^2 / sum(R), so deff = sum(R) / n. This generalises Kish (substituting a constant off-diagonal correlation recovers 1 + (n - 1) * rho exactly) but lets correlation decay with distance, which matters increasingly as cells get larger and Kish's single-rho assumption degrades. Supply the fit via sac, or it is estimated when response_var is given and gstat is available. The distances are taken in x and y: a layer whose features, or the points taken from them, do not all have the same dimensions (some with a Z or M coordinate) is read without its Z and M. Exponential, spherical and Gaussian models are supported, with a nugget and with several structured components (each weighted by its partial sill) and with gstat's 2-D geometric anisotropy (vgm(..., anis = c(angle, ratio))), applied as gstat applies it. A request that cannot be met falls back to "kish", with a warning of class "spatialkit_deff_fallback" that gives the reason and the ICCs: a model of any other family (named in the warning), or no usable model (none supplied and none could be estimated, or a rejected fit that could not be replaced by an estimate). The standard errors, intervals and cell_weight are then Kish's, the uncorrected ones where Kish's own ICC is 0 and NA where it cannot be estimated; the deff_method column says which on every row (see "Value"). The variogram route needs an identified range, which a layer does not always have, so read deff_method for the fallback. A layer whose geometries carry M (XYM or XYZM), which GEOS does not read, is read in x and y too. A model with no structured component (a pure nugget) implies that distinct observations are uncorrelated, so it is applied as a design effect of 1 in every cell, not treated as a fallback. That includes the model estimate_sac_range() returns when it finds no spatial structure (an NA range whose rejected_reason says so): the answer is then a design effect of 1, deff_method reads "variogram", and no fallback warning is raised. Predictor columns summarised under that verdict get the same design effect of 1, and a warning of class "spatialkit_deff_no_structure" names them (fields columns and reason). Points with empty geometry count towards their cells' values but not towards the correlation, with a warning.

"kish"

Estimate an intra-class correlation (ICC) for each summarised column from the grouped data using a one-way random-effects ANOVA decomposition (one ICC for the response and one for each predictor), then apply Kish's formula per cell: deff_i = 1 + (n_i - 1) * rho. The ICC is the ANOVA (method-of-moments) estimator with Donner's n0 for unequal cell sizes, not the REML estimate a mixed model returns, so on cells of unequal sizes the two estimates can differ. Each column's ICC is used for that column's SEs and for no other: the response's ICC is never applied to a predictor, nor one predictor's to another. It is the response's ICC that sets cell_weight whenever a response was given, and the first predictor's own ICC otherwise. An ICC needs at least 2 cells with 2+ observations of the column and at least 2 residual degrees of freedom (N - k >= 2). Where a column has fewer (all its values in one cell, say), its ICC cannot be estimated: it is NA, attr(, "icc")$reason says why, and the column's standard errors, ..neff_* and ..ci_* are NA in every cell of two or more observations (..df_* stays n - 1), as is cell_weight when that column sets it, with a warning of class "spatialkit_icc_unavailable" (fields columns and reason). Rows with no cell id (kept by keep_unassigned = TRUE in assign_features_to_polygons()) give a summary row of their own but no cell to the ICC, so a column whose values all fall in them, or differ only there, has an ICC that cannot be estimated. An estimate at or below 0 is another case: the ICC is 0 and the column's standard errors are the uncorrected ones. A column that does not vary (all its values equal, those of rows with no cell id included) has an ICC of NA and standard errors of 0, with no warning: its mean has no error whatever the correlation. A column whose values differ only by their rounding (0.1 beside 0.3 / 3) is read the same way, as one that does not vary, at any size of its values; its standard errors are 0 in a cell whose values are equal and of the size of that rounding in one whose values differ. The "deff_applied" attribute is attached when any ICC is positive, so when all are 0 or NA there is none.

A positive number

Applied as a uniform design effect to every cell, as sd * sqrt(deff / n), exactly sqrt(deff) times the naive SE. Use when you have an external estimate of the design effect. Anything that is not a single finite number >= 1 (including a value below 1, which would shrink the standard errors, and NA or Inf) is refused with a warning and replaced by 1.

sac

Optional sac_range object from estimate_sac_range(), used when deff = "variogram". Supplying one avoids re-fitting the variogram and lets you inspect the fit the design effect is based on. A sac_range whose fit was rejected (its value is NA and it carries a rejected_reason attribute) carries no usable correlation function, so it is set aside rather than correcting by a shape that was not trusted enough to report a range: the variogram is then estimated as if no sac had been given, with a plain warning saying so, or, where that is not possible, deff falls back to "kish" with the fallback warning, which names the rejection. The exception is a sac refused because no spatial structure was found, whose model is a pure nugget: it is used, and gives a design effect of 1. A sac with no variogram_model attribute, such as a plain number or a units object, is a range without a correlation function, and is set aside the same way. A sac fitted to residuals (attr(sac, "detrended") TRUE) is used as given, with a warning when it corrects response columns (see "Standard errors and design effects").

deff_max_n

Cells with more than this many points are subsampled before forming the n x n correlation matrix used by deff = "variogram". Default 500. It must be a single number of at least 2 when deff = "variogram"; anything else is an error. The subsample is drawn with a fixed seed from the cell's points in coordinate order, so it does not depend on the row order of assigned_points_sf, and the caller's random-number stream is left where it was.

quiet

Logical; suppress this function's progress message()s. It does not silence R warnings, nor the package's console log echo (see spatialkit_quiet for that). Default TRUE.

conf_level

Optional confidence level in (0, 1), such as 0.95. When given, every numeric response and predictor column also gets ..neff_*, ..df_*, ..ci_lo_* and ..ci_hi_* (see "Confidence intervals"). Default NULL: no such columns.

area

Logical, default FALSE. With TRUE, and cells_sf supplied, the result gains cell_area (each cell's planar area in the squared units of cells_sf's CRS) and n_per_area (the count of rows in the cell over that area, a point density; a rate of anything else is that thing's agg_funs sum over cell_area). The request is refused with an error, not answered with a number, when the cells' CRS distorts areas across them by more than 1 percent, measured as the spread of planar-to-geodesic area ratios over the cells (over 200 of them when there are more, taken at even spacing along the cells sorted by their bounding boxes in that CRS and then by the means of their vertices, together with the cells that reach furthest along x and y either way, so the order of the rows does not matter): a density is a comparison between cells and is meaningless where the map scale differs from one cell to the next. Inside a UTM zone the spread is under 0.3 percent and the request goes through; the conterminous United States forced into one zone (14 percent), or a few degrees of latitude in Web Mercator (4 percent at 48N), does not. Exact copies of one cell count once. A layer of one distinct cell gives one ratio and no spread, so its distortion is measured on a 6 by 6 grid over the cell's bounding box. Projected cells are first checked with sf::st_is_valid() in their CRS, every non-empty cell before any summary: a cell GEOS calls invalid (edges that cross, a spike, parts that overlap) has no planar area to vouch for, and the request is refused with an error that names the first such cells and GEOS's reason and says to repair the cells with sf::st_make_valid(). A valid projected cell with two vertices that fall on one point once in degrees, where s2 refuses it, is measured there without the repeat, which moves no area; when s2 refuses a valid cell it measures for another reason, the spread cannot be measured and the request is refused. ensure_projected() with purpose = "area" chooses an equal-area CRS for lon/lat input; build the cells in it. Lon/lat cells are measured geodesically instead: their cell_area is sf::st_area()'s area in square metres (on the sphere with s2, sf's default), not a planar area in squared degrees, and on the cells s2 measures the distortion check passes by construction. Every cell's area is taken before any summary, and when s2 refuses a cell that is not valid on the sphere, such as one with two corners it reads as one point (cells returned in lon/lat can hold them; see crs in create_voronoi_polygons()), the request stops with an error of class "spatialkit_cells_degenerate" that gives the rows of cells_sf s2 refuses (rows on the condition) and says to transform the cells to a projected CRS, such as the one they were built in or the equal-area one ensure_projected(purpose = "area") picks. With s2 switched off, sf needs lwgeom for that area and the request is refused without it. The measured spread is attached as attr(, "area_error"). Cells that mix XY and XYZ (or XYM) geometries are measured, and their spread taken, without Z and M, since sf reads a layer in the dimension of its first geometry; so are cells that carry M throughout, which GEOS does not read. The cells are returned with their own geometry. In a projected CRS whose unit sf has no name for (a +to_meter such as 1e6), sf::st_area() gives square metres; cell_area is the planar area in that CRS's unit squared all the same.

estimate

Which value of a cell the frame reports: "pooled" (default) or "mean". With "mean" the frame holds the plain columns only. With "pooled" every numeric response and predictor column also gets a pooled value, w * m + (1 - w) * t: the cell's own mean m and a target t, the mean of the rows of the 8 cells whose observations' centroids are nearest (all the other cells when there are fewer than 10; the polygons are not read). The own weight is w = tau2 / (tau2 + sigma2 / n), where n is the cell's count, sigma2 the spread of the observations about their cell's mean and tau2 the mean squared distance between a cell's true mean and its target. Each column is pooled on its own rows: those with a value, a cell and a location. agg_funs, deff, sac and conf_level do not enter the pooled columns, and every other column is computed as with "mean". Lon/lat coordinates, and coordinates without a CRS that look like degrees, are projected for the distances between centroids, as resolution_profile() projects them. The projection is chosen from every row of the layer, those without a value or a cell included, so on such input the rows a column does not use can still decide the CRS its distances are measured in; a layer projected beforehand is used in its own CRS. When the input carries the record of its cells' count (see "Value") and that record names a projected CRS, the one the profile worked in, the distances are measured in it, whatever CRS the layer is in. The distances are taken in x and y: a layer whose features, or the points taken from them, do not all have the same dimensions (some with a Z or M coordinate) is read without its Z and M. So is a layer whose geometries carry M (XYM or XYZM), which GEOS does not read. An abbreviation such as "m" is accepted. See "Pooled values" for what the pooled value estimates and what its numbers rest on, "Value" for the columns and the attributes, and "Conditions" for the warnings.

obs_level

Optional level in (0, 1), such as 0.90, of an interval for one further observation in a cell, located like the data. It needs estimate = "pooled": the interval is the pooled value plus the (1 - obs_level) / 2 and (1 + obs_level) / 2 quantiles of the leave-one-out residuals of the pooled values, so its two offsets are the same in every cell. It is a statement about observations on average, not about each cell. It is a separate argument from conf_level, whose interval is for a mean and is not a range for an observation. Any level strictly between 0 and 1 is computed the same way; only 0.90 has been checked (see "Intervals: which one answers which question"). Default NULL: no such interval.

Value

A tibble/data.frame (or sf if cells_sf given) with per-cell summaries: the ID column, n (rows in the cell), one column per agg_funs entry per variable (resp_<name>_<var> for the response and pred_<name>_<var> for each predictor), ..sd_* / ..se_* for every numeric response and predictor, ..neff_* / ..df_* / ..ci_lo_* / ..ci_hi_* for the same columns when conf_level is given, cell_weight (always), deff_applied and deff_method when a design effect was requested (any deff other than 1), and cell_area / n_per_area when area = TRUE ("Output columns" gives the order). An input column also called n is not allowed to shadow the count.

deff_applied is TRUE on every row when a design-effect correction was applied (exactly when the "deff_applied" attribute below is attached; under "kish", when any standard-error column was corrected) and FALSE when none was: a refused deff, or "kish" ICCs of 0 or NA for every column (the standard errors are then the uncorrected ones, or NA where an ICC cannot be estimated). It is NA on a cells_sf row no point fell in. deff_method names the correction the standard errors carry, since their column names do not change with it: "variogram", "kish" or "fixed" when the one requested was applied, "none" when none was, and "kish (variogram fallback)" or "none (variogram fallback)" when a "variogram" request could not be met and Kish stood in for it (the second when Kish's ICCs were 0 or NA too). So deff_applied is TRUE on a Kish fallback row: test deff_method == "variogram" to keep only rows the variogram corrected. Unlike the attribute, both columns survive rbind() and dplyr::bind_rows() of many results. A fallback is also signalled by a warning of class "spatialkit_deff_fallback" (a Kish ICC of 0 under a "kish" request is reported on attr(, "icc") instead), which tryCatch(spatialkit_deff_fallback = ) catches without matching the message. It is raised when the correction requested was not the one applied: a refused number (the standard errors are then the uncorrected ones) or a "variogram" request that fell back to Kish; the condition of the second carries requested and used. Two more classes say what a design effect could not do: "spatialkit_icc_unavailable" names the columns whose Kish ICC cannot be estimated, and "spatialkit_deff_no_structure" the predictor columns given a design effect of 1 by a variogram that found no spatial structure.

When a correction was actually applied, an attribute "deff_applied" is attached recording it: method plus icc_resp/icc_pred and deff for "kish" (icc_pred is a vector named by predictor column, one ICC each; deff is the primary variable's per-cell design effect, all 1 when only a predictor's ICC was positive, except that it is NA in each cell of two or more of the primary variable's values when that variable's ICC cannot be estimated, and its entry in icc_resp or icc_pred is then NA too; requested and fallback_reason are added when a "variogram" request fell back to it), deff/deff_rows/rbar/crs/max_n for "variogram" (deff is the design effect at the primary variable's non-missing count per cell, deff_rows at the cell's row count, which is the vector the log line summarises as a median and a max), and deff alone for a fixed number. When cells_sf is supplied, every per-cell vector in that attribute (deff, deff_rows and rbar alike) is realigned to the joined row order, so deff[i] and rbar[i] still describe row i; cells with no observations carry NA. No attribute is attached when no correction was applied: deff = 1, or a deff = "kish" request (or Kish fallback) whose ICCs are all 0 or NA. A deff = "kish" request, and a "variogram" request that fell back to it, always records the ICCs it estimated on an attribute "icc" (resp, and pred, a vector named by predictor column; NA for a variable type with no numeric column and for a column whose ICC cannot be estimated; reason, a character vector named by column, is then added and says why), whether or not they were positive enough to apply, so a result with no "deff_applied" still says what the ICCs came out as.

The ID column keeps its input type when cells_sf's ID column and the summarised IDs already have the same class. When the classes differ, both are coerced to character in order to join (logged as a warning), and the returned ID column is therefore character. Whole numbers are written out in full for that ("100000", never "1e+05"), so an integer and a double ID of the same cell still match.

With estimate = "pooled", each numeric column v with its prefix (resp_ or pred_) gains ..pooled_<prefix><v> (the pooled value), ..pooled_w_<prefix><v> (the own weight, 0 to 1; it is not a regression weight) and ..pooled_target_<prefix><v> (the target), and with obs_level also ..obs_lo_<prefix><v> and ..obs_hi_<prefix><v>. They follow every column listed above and precede cell_area and n_per_area, the response's first, then each predictor's. A cell with no row that has a value, a cell and a location gets NA, never a value made from its neighbours. Two attributes are attached. attr(, "pooled"), of class "spatialkit_pooled", records what was fitted: toward, neighbours, crs (the CRS the centroid distances were measured in), obs_level, conf_level, intervals and columns, a list with one entry per pooled column holding column, role ("response" or "predictor"), n (rows used), cells, df (values beyond one per cell, which sigma2 rests on), neighbours (the cells in each target), sigma2, tau2, own_weight (the median weight), share_alone (the share of rows that are the only one in their cell), loo and loo_se (the leave-one-out mean squared error of the pooled value, and its standard error over rows), loo_one_cell (the same error with one mean for all rows), structured and structure_p (whether the cells predict a left-out observation better than one mean, and the p-value of the F test of the cell means against one mean, which has to be below 0.05 as well), offsets (the two offsets of the interval), repeated, skipped, constant and status. attr(, "intervals") is the same data frame as attr(, "pooled")$intervals: one row per family of interval columns in the frame, saying what it covers, what it is centred on, its level and its method. Away from the observations no interval here is calibrated.

Where the pooled columns of a column cannot be computed they are NA, with a warning of class "spatialkit_pool_unavailable" and not an error, and the plain columns are returned. "Conditions" lists that class and the other warnings of the pooled columns, with what each condition carries.

When assigned_points_sf carries attr(, "count_rule"), or cells_sf does (the assigned layer's is read first), the result carries that record with one more element, check, whatever estimate is. The record is the one select_resolution() wrote: rule, estimate (the cell value the count was chosen for), cells, obs_per_cell, response, scored (what the profile scored at the count: the rows, the cells that hold one, the leave-one-out error among them, and the centres the rows were given to), partition, crs and scored_cells. It has class "spatialkit_count_rule" and a print method. check$estimate is "match" or "mismatch": whether the summary reports the estimate the count was chosen for (NA for a count chosen for neither). A mismatch is a warning of class "spatialkit_estimate_mismatch" (fields chosen_for, reported, rule, cells), and the summary is returned as asked. check$cells is "match", "differ" or "not checked: <reason>": the response is fitted again on the rows and cells given, with the estimate the profile scored, and its scored rows, its occupied cells and its leave-one-out error are compared with the record's, the error to a relative tolerance of 1e-8. The first that differs is a warning of class "spatialkit_cells_differ" (fields what: "n", "cells" or "loo"; scored; found): rows were dropped or moved between cells, or the layer is another one. The same class with what = "record" (fields field, scored, found) is raised when the assigned layer and cells_sf carry different records, so the rows were assigned to other cells than the cells_sf given: the cells of another selection, or the scored cells of a selection beside other cells built from its count, such as a lattice sized by it. check$cells is then "differ", whatever the comparison of the rows finds. The comparison is not made, and check$cells says why, when the response is another column or was not summarised, the profile read the selection half (select_on = "split"), the cells are not the ones the profile scored (scored_cells is FALSE or NA), the profile scored no estimate, the input has no coordinates, the features are not points (see assign_features_to_polygons()), or the fit was stopped by the limit on an exact search without the FNN package. Cells and rows without a record, such as polygons the caller brings, are not checked and raise neither warning.

A row that is the same distance from two centres lies on the boundary between their cells, which happens on gridded locations. resolution_profile() scores such a row in the cell of the lower-numbered centre (a distance counts as the same as the least when its square is no more than the square of the least distance lengthened by a margin, plus one part in ten million of the least squared distance; the margin is one part in 1e10 of the largest coordinate of the centres, at most a millionth of the diagonal of their bounding box and at least four times the spacing of doubles at that coordinate), and assign_features_to_polygons(), on cells built from the centres the profile scored, gives a point on that boundary the same cell, measured in the CRS the record names, so the comparison reads "match" on gridded locations too. It does so on cells that carry the record, for rows that have a CRS exactly when the record names one. Rows without a CRS beside a record that names one, rows with a CRS beside a record that names none, rows assigned to the cells after they lost the record (written to a file and read back, say) and summarised with cells_sf set to the cells that carry it, and rows given their cells by another join are placed by the polygons alone, so a row on a boundary can sit in either cell. build_tessellation() draws those cells in the CRS the record names, whatever CRS the seeds arrive in, unless its crs names another projected CRS. Cells built in another projected CRS are the nearest-centre cells of that CRS, and cells transformed to another projected CRS are joined there, so a row near a boundary, on it or not, can sit in another cell than the profile put it in. So can a row joined in longitude and latitude, as rows and cells in degrees are when the cells do not carry the record or carry one in degrees (the Voronoi cells of the scored centres that build_tessellation() returns in a geographic crs are joined in the record's CRS while they carry it), a row in cells that carry the record but are not the Voronoi cells of its centres (seeds moved, or cells edited, after the selection), and a row that reached the record's CRS by another transformation than the profile's, which can settle a tie the other way. Cells returned in a geographic crs are moved back to the record's CRS for that join, and for a layer wider than a UTM zone the round trip can move a corner by more than the margin of assign_features_to_polygons(), so a row on an edge there can sit in another cell too. The comparison then reports "differ", with what = "loo" when it finds the same rows in the same number of cells with another leave-one-out error, and the warning names these cases. The summary's own numbers describe the cells as assigned.

The comparison uses the fit of the pooled columns when that is a fit of the estimate the profile scored, and fits the response once more otherwise. Without the FNN package that fit compares every pair of cell centroids (pooled values) or, for each row alone in its cell, every other location (plain means), and above options(spatialkit.exact_pairs_max = ) pairs it is not made ("not checked: work limit"). A summary is not compared when neither layer carries the record: attr(x, "count_rule") <- NULL on the assigned layer and on cells_sf removes it.

Details

This is the step after assign_features_to_polygons(), which labels each observation with its cell. Beside every mean the frame keeps the count and the standard error, so a cell mean over 2 observations and one over 200 stay distinguishable in a map or a model, and deff can widen the standard error for correlation inside a cell.

Output columns

One row per cell, with these columns in this order:

  • the ID column (see id_col);

  • n, the rows that fall in the cell;

  • for each variable, the response's first and then each predictor's: one column per agg_funs entry (resp_<name>_<var> for the response and pred_<name>_<var> for a predictor, so resp_mean_val for the default mean of a response val), the within-cell standard deviation (..sd_resp_<var>, ..sd_pred_<var>) and standard error (..se_resp_<var>, ..se_pred_<var>) of a numeric column, and with conf_level its ..neff_*, ..df_*, ..ci_lo_* and ..ci_hi_* (see "Confidence intervals");

  • cell_weight, always;

  • deff_applied and deff_method when a design effect was requested (any deff other than 1);

  • with estimate = "pooled", the pooled columns (see "Pooled values" and "Value");

  • with cells_sf, the cells' geometry;

  • with area = TRUE, cell_area and n_per_area.

cell_weight is the effective sample size of the primary variable: the response when one was supplied, otherwise the first predictor. It counts that variable's non-missing rows, not all rows (a cell of 10 rows with 3 finite responses carries 3 observations' worth of information about the response), and it is divided by that cell's design effect when deff applied one. With deff = 1 and no missing values it equals n. Pass it as the weights argument of a downstream regression.

An infinite value (Inf, -Inf) in a response or predictor column is treated as missing, as NA and NaN are, in every statistic of that column: the agg_funs results, ..sd_*, ..se_*, the interval columns, ..neff_*, cell_weight and the design effect. n still counts the row. A warning of class "spatialkit_nonfinite" names the columns and gives the number of rows and cells affected (the condition carries n and columns). kriging_adequacy(), resolution_profile(), estimate_sac_range() and prep_model_data() leave such rows out too, so the steps of one pipeline agree on a cell's mean. A function in agg_funs therefore never sees an infinite value: max returns the largest finite value, and infinities cannot be counted through agg_funs.

..sd_* is the standard deviation of the cell's values in the units of the column, at any size. Where the raw squares overflow or lose their bits (a standard deviation that is not finite or is below 1e-150), it is taken on the values centred on their median and divided by powers of two, which keeps their spread. A cell whose standard deviation is beyond the largest double (about 1.8e308) has ..sd_* of Inf and standard errors and interval of NA, with a warning of class "spatialkit_sd_overflow" (the condition carries columns and cells, the number of such cells of each column).

Pooled values

With estimate = "pooled" every numeric column also gets a pooled value. That is the default; estimate = "mean" leaves the pooled columns out. A cell's value, plain or pooled, estimates the cell's true mean: the expected value of one further observation located in the cell the way its observations were. It is not the average of a surface over the cell's area, which would give weight to ground that holds no observation.

For one column, write n, S and m = S / n for the count, the sum and the mean of a cell's rows, N for the rows in all and K for the cells that hold one. Then

  • sigma2 = sum((y - m)^2) / (N - K) is the spread of the observed values about their cell's mean;

  • the target t of a cell is the mean of the rows of its neighbours, sum(S) / sum(n) over them. The neighbours are the 8 cells whose centroids are nearest, the centroid of a cell being the mean of the coordinates of its rows. The polygons are not read, so two cells are neighbours whether or not they touch, and a cell is never its own neighbour;

  • tau2 = max(0, (sum(n * (m - t)^2) - K * sigma2) / N) is the mean squared distance between a cell's true mean and its target, the target's own error included;

  • the own weight is w = tau2 / (tau2 + sigma2 / n);

  • the pooled value is w * m + (1 - w) * t.

The own weight is the share of the pooled value that is the cell's own mean. It grows with the cell's count and with tau2. At 1 the pooled value is the plain mean; at 0 it is the target and holds none of the cell's own values. Pooled values are for maps and for predicting observations. ..pooled_w_* is not a regression weight. The plain columns are the same under either estimate, and they are the ones to use, with cell_weight, in a regression on cell values: each column is pulled toward its neighbours by its own amount, so a regression of pooled responses on pooled predictors is not a regression of the true means.

The cells can be any polygons the rows were assigned to, the package's own or ones the caller brings, with IDs of any type: only each row's cell and location are read. Each column is pooled on its own rows, those with a value, a cell and a location, with its own centroids, neighbours, sigma2 and tau2; a missing value in one column changes nothing in another. A cell with one such row gets a value. A cell with none gets NA, never a value made from its neighbours. The pooled columns do not read agg_funs or deff: the mean that is pooled is the arithmetic mean whatever agg_funs computes, and no design effect enters sigma2, tau2, the weights or the interval of obs_level.

Three cases change what the numbers rest on, and each is a warning (see "Conditions"). With 2 to 9 cells the neighbours of a cell are all the others, so its target is the mean of every row outside it, and the overall mean lies between that target and the cell's own mean. A cell whose own weight is below its share of the rows, n / N, then gets a pooled value on the far side of the overall mean from its plain mean, and two cells can change order. When few cells hold two rows, sigma2 is estimated from few values and every weight rests on it: df in attr(, "pooled") is the number of values beyond one per cell. And when tau2 comes out at 0, every own weight is 0 and each pooled value is its target; with two cells that is the other cell's mean, so the two values change places.

Two candidate neighbours are a tie when their squared distances differ by no more than one part in ten million. A tie goes to the centroid with the lower x, then the lower y, coordinates within that part of the distance counting as equal, and then to the cell whose ID sorts first. The centroids are summed with the rows in coordinate order, so a cell's neighbours do not depend on the order of the rows, and on gridded locations they do not depend on the unit or the origin of the coordinates either. The values agree to rounding under another order of the rows, since a cell's sums are taken in the order of its rows, and under another unit or origin where three to eight cells' centroids coincide, since the rounding then sets the order in which a cell's neighbours on that point are summed. A cell whose centroid shares its point with as many other cells as it has neighbours, or more, takes its neighbours from them, the first by ID, without a search. Centroids count as one point there when they are equal, or when they lie within a ten-millionth of the largest coordinate of one another, in x and in y. Where other centroids that close to them chain on further than that, the chain is taken again at half that reach, and so on, until its parts span no more than the reach they were taken at. So those cells keep their neighbours and their values in another unit, and with the origin moved by little against the coordinates. On such a point, cells whose centroids lie that close without coinciding take each other by ID as well, not by distance. The reach follows the largest coordinate, so with the origin moved far, to the corner of the data say, such centroids can lie beyond it, and their cells then take the nearest. Distances are measured in the CRS named under estimate, so the same layer in another projection can still give a cell other neighbours, and so can a lattice whose spacing is below about a ten-millionth of its coordinates.

structured in attr(, "pooled") says whether these cells predict a left-out observation better than one mean for all rows. It is FALSE when the leave-one-out error of one mean (loo_one_cell) is within 2 percent of that of the pooled values (loo) or below it, or when the F test of the cell means against one mean has p >= 0.05 (structure_p). FALSE raises a warning of class "spatialkit_no_structure", and the values are returned. On a response without spatial structure they still make a smooth map: the own weights are small, so each pooled value lies close to the mean of its neighbours ("How this was checked" gives the figures).

The pooling model implies an error variance for a pooled value as an estimate of the cell's true mean, (1 - w) * tau2. It is not returned, and neither is an interval for the true mean, because no check has confirmed an interval built from it; sigma2, tau2 and ..pooled_w_* are in the result for a reader who wants to form it. Away from the observations that variance is too small ("How this was checked").

Beside a sharp boundary the pooled value can be further from a cell's true mean than the plain mean, because its target takes in cells across the boundary.

Intervals: which one answers which question

Two families of interval columns can be in the frame. Each has its own argument, and neither changes the other.

FamilyColumnsTargetCentreAsked for by
observation..obs_lo_*, ..obs_hi_*one further observation in the cellpooled valueobs_level
mean..ci_lo_*, ..ci_hi_*a mean, never one observationplain meanconf_level

The observation of the first is one located in the cell the way the data's were. obs_level needs estimate = "pooled": without it the call is an error, not an interval round the plain mean. The mean the second covers follows deff: at deff = 1 the cell's own mean, when its observations are spread through the cell; under deff = "kish", "variogram" or a number other than 1 the cell mean as an estimate of the population mean (see "Confidence intervals", which also describes ..neff_* and ..df_*). No interval and no standard error is returned for a cell's true mean as the pooled value estimates it. attr(, "intervals") says the same of the frame in hand, one row per family present.

The interval for one observation is the pooled value plus two quantiles of the leave-one-out residuals of the pooled values. The residual e of a row is its value minus the pooled value of its cell computed without it: the cell's mean and own weight are computed again without the row, and sigma2, tau2 and the targets are held. With p = (1 - obs_level) / 2, ..obs_lo = pooled + quantile(e, p) and ..obs_hi = pooled + quantile(e, 1 - p).

  • It is a statement about observations on average: of further observations located like the data, about obs_level of them fall inside the interval of their cell. It does not hold cell by cell.

  • The two offsets are the same in every cell. The interval's centre varies over the map and its width does not. It does not narrow where a cell holds more rows: the offsets are quantiles of the errors of predicting single observations, and the spread of observations inside a cell is in every one of them.

  • It is wide, because one observation is uncertain.

  • Only the level 0.90 has been checked. Another level is computed the same way.

  • Each residual is the error of predicting an observation from the other rows of its cell and from the cells around it, so the interval describes locations among the observations. A cell's value and interval describe the part of the cell where its observations lie, and a Voronoi cell can reach far past them toward the edge of the study area. The interval is too narrow for a cell with no observation, a gap in the layer or ground beyond the layer's edge.

cell_values_at() returns each location's distance from the nearest observation, and its help gives the coverage by that distance.

Away from the observations no interval here is calibrated.

Standard errors and design effects

What the ..se_* columns estimate follows deff. At the default, deff = 1, sd / sqrt(n) is the standard error of the cell's own mean, the mean of the units the cell holds, and it is the right one when the cell's observations are spread through the cell. Under deff = "kish", "variogram" or a number other than 1 the columns are the standard error of the cell mean as an estimate of the population (grand) mean: the unconditional quantity, in which the cell's own realised deviation is part of the error, widened for the correlation inside the cell. That is the right quantity when cells are treated as samples from a common population.

Spatially autocorrelated observations in one cell are positively correlated: they share the cell's departure from the population mean. As an estimate of the population mean, a cell mean then has an effective sample size smaller than n, and the naive standard error is too small. A downstream weighted regression that uses cell_weight or the ..se_* columns for population-level inference then gives overconfident standard errors for cells with strong within-cell correlation.

The corrected standard error is not that of the cell's own mean (the block average over that cell), which is what a cell-level map or a regression on cell values usually wants. For that quantity the naive sd / sqrt(n) is the better of the two on offer. Under exchangeable within-cell correlation it is exact, and the design-effect-corrected SE is too wide by the factor sqrt(deff / (1 - rho)) (4.6 at 20 points a cell and rho = 0.5). Under a spatial covariance model it holds only when the cell's points are spread through the cell; with clustered sampling inside a cell it is too small for the block average too, and the right quantity there is the plain mean's error variance under a variogram, plain_var in kriging_adequacy(). Use deff when the cell means feed a population-level inference; leave it at 1 when they are measurements of the cells themselves and the sampling within cells is close to uniform.

deff = "kish" applies Kish's design effect. An intra-class correlation (ICC) is estimated for every summarised column on its own, the response and each predictor, by a one-way random-effects decomposition across all cells, and each column's ICC is used for that column alone: listing or removing a predictor leaves every other column's standard error, effective n and interval as they were. Each cell's effective sample size is n_i / (1 + (n_i - 1) * rho). This first-order correction needs no spatial covariance model, but it needs enough cells and observations for a stable ICC. A column whose ICC cannot be estimated at all gets NA standard errors (see deff).

deff = "variogram" fits or accepts one correlation function and applies it to every numeric column, because a variogram is a property of the field being modelled rather than of a variable type; a predictor whose spatial structure differs from the response's has its SE corrected by the response's correlation. When that variogram finds no spatial structure (the pure nugget estimate_sac_range() returns with that verdict), every column gets a design effect of 1, the predictors included, and a warning of class "spatialkit_deff_no_structure" names the predictor columns: the verdict is about the variable the variogram was fitted to, not about them. The internally estimated variogram is fitted to the response itself, never to OLS residuals, whatever predictor_vars holds: the ..se_resp_* columns estimate the grand mean of the response, so the correlation to correct for is the response's own. A residual variogram holds only the correlation of the part the predictors do not explain, which is weaker. Pass sac explicitly for a different variogram, such as a residual one from estimate_sac_range(..., predictor_vars = ). A sac whose attr(sac, "detrended") is TRUE is used as given, but when it corrects response columns a warning says that their standard errors are understated, as kriging_adequacy() warns about the same mismatch.

A design effect estimated from the data ("kish" or "variogram") comes with a second, small-sample correction. The within-cell correlation that inflates the variance of the mean to sigma^2 * deff / n also biases the within-cell sample variance downward: under exchangeable correlation rho (Kish's own assumption), with deff = 1 + (n - 1) * rho, E[s^2] = sigma^2 * (n - deff) / (n - 1), so s^2 understates sigma^2 by very nearly the factor by which deff inflates the mean's variance, and the two errors compound. The standard error is therefore s * sqrt(deff / n) * sqrt((n - 1) / (n - deff)), and NA where deff >= n (the cell then holds one observation's worth of information and s carries none about sigma). This is the package's own derivation, not taken from a reference.

A fixed numeric design effect (deff = 2, say) inflates the standard errors uniformly: it is applied as sd * sqrt(deff / n), exactly sqrt(deff) times the naive SE in every cell. The E[s^2] correction of the estimated design effects is derived from within-cell correlation, and is not applied to a number the caller chose.

Even with these corrections the adjusted SE is an approximation. For inference under spatial dependence, fit an explicit spatial covariance model, such as fit_bayesian_spatial_model().

Confidence intervals

With conf_level set, every numeric response and predictor column gains four more columns: ..neff_*, that column's effective sample size in the cell (its non-missing count over its design effect, the per-column version of cell_weight); ..df_*, the degrees of freedom the interval uses; and ..ci_lo_* / ..ci_hi_*, a t interval for a mean. Which mean follows deff, as for ..se_*: at the default, deff = 1, the cell's own mean, when its observations are spread through the cell; under deff = "kish", "variogram" or a number other than 1, the cell mean as an estimate of the population (grand) mean. "Standard errors and design effects" applies to it, including that the corrected interval is not one for the cell's own block average. The interval is mean +/- qt((1 + conf_level) / 2, df) * se, centred on the plain mean of the column's non-missing values whatever agg_funs computes, and is NA wherever the standard error is (a single observation; complete redundancy under deff). ..neff_* and ..df_* are NA where the column has one non-missing value or none, like the standard error; cell_weight still counts such a cell (1, or 1 / deff for a numeric deff), so the two differ there.

At no setting is ..ci_* a range for one observation: it is an interval for a mean, and it narrows as a cell's count grows where the spread of single observations does not. The interval for one further observation in a cell has its own argument and columns, obs_level with estimate = "pooled" and ..obs_lo_* / ..obs_hi_* (see "Intervals: which one answers which question"). Away from the observations no interval here is calibrated.

The degrees of freedom are not neff - 1. The interval's spread comes from the within-cell sample variance, and under exchangeable correlation (deff = "kish") that variance keeps its n - 1 degrees of freedom whatever the design effect. The design effect inflates the mean's variance and biases s^2, both of which the standard error already corrects, and the resulting pivot is exactly t on n - 1 df. The effective-sample-size degrees of freedom of Faes et al. (2009) belong to a mean estimated across many correlated units whose variance is estimated from the spread between them; they do not transfer to a single cell's mean with a variance estimated from inside it. ..df_* is therefore n - 1 at deff = 1, for a numeric deff and for "kish". Under deff = "variogram" correlation decays with distance and the within-cell variance loses degrees of freedom to it; ..df_* is then the Satterthwaite (1946) moment-matched df of s^2 under the fitted correlation matrix, a fraction of n - 1 that shrinks as the range grows. The interval is only as good as the design effect under it: a mis-specified variogram, or an ICC estimated from too few cells, moves the coverage with it.

Conditions

What the pooled columns and the count record have to say is raised as R warnings with a class, so tryCatch(<class> = ) and withCallingHandlers() catch each without matching its text. A warning of the pooled columns is raised once per call and reason, for all the columns it applies to: its text names them and the condition carries them in columns, in the order of the pooled columns. A field said to be by column is a vector named by those columns, and the numbers of each column are in attr(, "pooled").

  • "spatialkit_pool_unavailable": the pooled columns of a column are NA. The plain columns are returned, and this is never an error. reason is "no coordinates" (a layer without geometry), "no located value" (no row with a value, a cell and a location), "no cell holds two values" (the condition also carries n and cells by column), "values out of range" (values so large, or so close to their median, that the squares of many such values can overflow or lose their precision; the message gives the bounds, and the column rescaled by a power of ten is pooled) or "work limit". The last is the case of many cells without the FNN package: the neighbour search then compares every pair of cells and is not made above getOption("spatialkit.exact_pairs_max", 2e9) pairs, which is about 44,700 cells (the condition also carries work, limit, and cells by column). With FNN there is no such limit, and the result is the same.

  • "spatialkit_pool_degenerate": the pooled values were computed and carry no pooling. reason is "one cell" (the pooled value is the plain mean; also n by column), "no spread inside any cell" (the values do not vary inside any cell to their precision, so every own weight is 1; also constant by column) or "no signal between cells" (tau2 is 0 and every own weight is 0; also sigma2 by column).

  • "spatialkit_pool_few_cells": 2 to 9 cells, so each target is the mean of all the other cells, and a cell whose own weight is below its share of the column's rows gets a pooled value on the far side of the overall mean from its plain mean (cells, and own_weight, the smallest, by column). The text also names each column with two cells and tau2 at 0, which has cells 2 and own_weight 0: each cell's pooled value is then the other cell's mean, so the two values change places.

  • "spatialkit_pool_few_df": sigma2 rests on fewer than 30 values beyond one per cell (needed, and df, n and cells by column).

  • "spatialkit_no_structure": structured is FALSE (see "Pooled values"). The text says which of the two tests failed, and beats is TRUE for a column whose pooled values beat one mean while the F test does not tell the cell means from one mean (also criterion = "loo", estimate = "pooled", and by column n, the cells, one_cell, loo and structure_p).

  • "spatialkit_pool_rows_skipped": rows with a value and a cell have no location: an empty point, a coordinate that is not finite or lies beyond 1e150, as given or in the CRS the distances are measured in, or in a geographic CRS a vertex at a latitude beyond 90 degrees. They count in n and the plain columns and take no part in the pooled ones (n, cells).

  • "spatialkit_repeated_locations", with obs_level: rows with a value share a location. A residual leaves one row out and keeps the others at its location, so the interval then describes a further observation at a place already observed (n and total by column).

  • "spatialkit_few_residuals", with obs_level: fewer than five residuals are expected beyond each end of the interval, which at 0.90 is fewer than 100 residuals (needed, level, and n by column).

  • "spatialkit_estimate_mismatch" and "spatialkit_cells_differ": the two checks of the count record, described under "Value".

The two about few cells and few values are raised for columns whose weights were estimated, and the two of obs_level for columns whose interval is read from residuals, so a constant column raises "spatialkit_pool_degenerate" and no other.

How this was checked

The figures on this page that describe pooled values and the two intervals come from one simulation, dev/sim-pooled-values.R in the package's GitHub repository, whose header holds its settings and seeds. It draws 200 layers on a square of side 1,000. Each has 2,100 clustered locations (four fifths of them round 25 parent points with a standard deviation of 40, the rest uniform) and a response that is a field with covariance exp(-d / 100) plus noise of variance 0.3. The 300 locations nearest a random centre are held out as a block, 300 of the others are held out at random, and the 1,500 left are the layer: 333 cells at 4.5 observations a cell from resolution_profile() with partition = "density", select_resolution(criterion = "size"), get_voronoi_seeds(), build_tessellation() and assign_features_to_polygons(), summarised with estimate = "pooled". A held-out observation is given the value of its cell by cell_values_at().

At the observations held out at random the root mean squared error of the pooled value was 0.96 of the plain mean's: 0.84 in cells that hold one of the layer's observations, 0.95 in cells of 2 to 4, 0.98 in cells of 5 to 9 and 0.99 in cells of 10 or more. In the held-out block it was 0.95.

The interval at obs_level = 0.90 held 89.6 percent of the observations held out at random among the others (83.7 to 94.0 percent in single layers) and 75.1 percent of those in the held-out block. ..ci_* at conf_level = 0.90, read as a range for one observation, held 58.9 and 49.2 percent of them, over the held-out observations whose cell has that interval.

With a cell's true mean taken as the mean of the noise-free field over further locations drawn in it like the data's, the squared error of the pooled value was 0.98 of (1 - w) * tau2 among the observations and 6.9 times it in the held-out block.

With white noise as the response on the simulation's 200 layers the median own weight was 0.11, tau2 was 0 on 20 layers, the warning of class "spatialkit_no_structure" was raised on all 200, and a cell's pooled value had a correlation of 0.79 with the mean of the pooled values of its 8 nearest cells, where the plain means had one of -0.02.

References

Faes, C., Molenberghs, G., Aerts, M., Verbeke, G. and Kenward, M. G. (2009). The effective sample size and an alternative small-sample degrees-of-freedom method. The American Statistician, 63(4), 389–399. doi:10.1198/tast.2009.08196

Satterthwaite, F. E. (1946). An approximate distribution of estimates of variance components. Biometrics Bulletin, 2(6), 110–114. doi:10.2307/3002019

See also

assign_features_to_polygons(), which produces the input layer; build_tessellation() for the cells themselves.

Other assign and summarise: assign_features_to_polygons(), cell_values_at(), kriging_adequacy()

Examples

library(sf)
set.seed(1)
n <- 200
east  <- runif(n, 0, 100)
north <- runif(n, 0, 100)
# A response with spatial structure, so the within-cell ICC is not zero.
pts <- st_as_sf(
  data.frame(x = 5e5 + east, y = 5e6 + north,
             val = 5 + 0.02 * east + 0.02 * north + rnorm(n, sd = 0.5)),
  coords = c("x", "y"), crs = 32632
)
bnd <- st_sf(geometry = st_sfc(st_polygon(list(rbind(
  c(5e5, 5e6), c(5e5 + 100, 5e6), c(5e5 + 100, 5e6 + 100),
  c(5e5, 5e6 + 100), c(5e5, 5e6)
))), crs = 32632))
grid <- create_grid_polygons(bnd, target_cells = 9, type = "square")
assigned <- assign_features_to_polygons(pts, grid)

# Nine cells are too few for pooled values (each would be pulled toward
# all the other cells), so these summaries report plain means alone:
# estimate = "mean".
# IID standard errors (default) against the Kish design effect.  The Kish
# correction is large here: about 22 points per cell that share the cell's
# part of the trend carry far fewer than 22 independent pieces of
# information about the cell mean.
naive <- summarize_by_cell(assigned, response_var = "val", estimate = "mean")
kish  <- summarize_by_cell(assigned, response_var = "val", deff = "kish",
                           estimate = "mean")
data.frame(n = naive$n,
           se_naive = naive$..se_resp_val,
           se_kish  = kish$..se_resp_val)
#>    n   se_naive   se_kish
#> 1 25 0.09706775 0.7210227
#> 2 28 0.11815220 0.9279034
#> 3 25 0.11573069 0.8596517
#> 4 21 0.11556510 0.7881134
#> 5 23 0.13192931 0.9407000
#> 6 21 0.12707224 0.8665880
#> 7 10 0.18215988 0.8673238
#> 8 27 0.09534639 0.7355266
#> 9 20 0.14402822 0.9590654
attr(kish, "deff_applied")   # method, icc_resp, icc_pred, per-cell deff
#> $method
#> [1] "kish"
#> 
#> $icc_resp
#> [1] 0.6842466
#> 
#> $icc_pred
#> [1] NA
#> 
#> $deff
#> [1] 17.42192 19.47466 17.42192 14.68493 16.05343 14.68493  7.15822 18.79041
#> [9] 14.00069
#> 

# A 95% interval for each cell mean as an estimate of the grand mean, on
# the Kish-corrected standard error and n - 1 degrees of freedom.
ci <- summarize_by_cell(assigned, response_var = "val", deff = "kish",
                        conf_level = 0.95, estimate = "mean")
ci[, c("poly_id", "n", "resp_mean_val", "..neff_resp_val", "..df_resp_val",
       "..ci_lo_resp_val", "..ci_hi_resp_val")]
#> # A tibble: 9 × 7
#>   poly_id     n resp_mean_val ..neff_resp_val ..df_resp_val ..ci_lo_resp_val
#>     <int> <int>         <dbl>           <dbl>         <dbl>            <dbl>
#> 1       1    25          5.63            1.43            24             4.14
#> 2       2    28          6.30            1.44            27             4.40
#> 3       3    25          6.93            1.43            24             5.16
#> 4       4    21          6.42            1.43            20             4.77
#> 5       5    23          7.11            1.43            22             5.15
#> 6       6    21          7.89            1.43            20             6.09
#> 7       7    10          7.16            1.40             9             5.19
#> 8       8    27          7.77            1.44            26             6.25
#> 9       9    20          8.16            1.43            19             6.15
#> # ℹ 1 more variable: ..ci_hi_resp_val <dbl>

# Pooled values, and a range for one observation: a smooth field plus
# noise at 400 points, on an 8 x 8 grid, so that a cell holds about six.
set.seed(2)
east  <- runif(400, 0, 8000)
north <- runif(400, 0, 8000)
pts2 <- st_as_sf(
  data.frame(x = 5e5 + east, y = 5e6 + north,
             val = 10 + 3 * sin(east / 1500) + 3 * cos(north / 1200) +
               rnorm(400)),
  coords = c("x", "y"), crs = 32632
)
bnd2 <- st_sf(geometry = st_sfc(st_polygon(list(rbind(
  c(5e5, 5e6), c(5e5 + 8000, 5e6), c(5e5 + 8000, 5e6 + 8000),
  c(5e5, 5e6 + 8000), c(5e5, 5e6)
))), crs = 32632))
grid2 <- create_grid_polygons(bnd2, target_cells = 64, type = "square")
assigned2 <- assign_features_to_polygons(pts2, grid2)
# estimate = "pooled" is the default.
pooled <- summarize_by_cell(assigned2, response_var = "val", obs_level = 0.90)
# The plain mean, the pooled value and the weight on the cell's own mean.
head(pooled[, c("poly_id", "n", "resp_mean_val", "..pooled_resp_val",
                "..pooled_w_resp_val")])
#> # A tibble: 6 × 5
#>   poly_id     n resp_mean_val ..pooled_resp_val ..pooled_w_resp_val
#>     <int> <int>         <dbl>             <dbl>               <dbl>
#> 1       1     3          13.6              13.3               0.652
#> 2       2    15          15.0              14.8               0.904
#> 3       3     2          16.6              15.4               0.555
#> 4       4     2          14.0              13.8               0.555
#> 5       5     8          13.4              13.3               0.833
#> 6       6     9          10.9              10.9               0.849
# The range for one further observation in a cell.
head(pooled[, c("poly_id", "..obs_lo_resp_val", "..obs_hi_resp_val")])
#> # A tibble: 6 × 3
#>   poly_id ..obs_lo_resp_val ..obs_hi_resp_val
#>     <int>             <dbl>             <dbl>
#> 1       1             11.3               15.4
#> 2       2             12.8               16.9
#> 3       3             13.4               17.5
#> 4       4             11.8               15.9
#> 5       5             11.3               15.4
#> 6       6              8.86              12.9
attr(pooled, "pooled")   # what was fitted, and what the interval covers
#> Cell values: val, 400 rows in 64 cells
#>   value       : ..pooled_resp_val, pooled toward the 8 nearest cells (centroids of the rows)
#>   variances   : inside cells 1.41 (from 336 values beyond one per cell);
#>                 between a cell and its target 0.88
#>   own weight  : median 0.79 (..pooled_w_resp_val)
#>   alone       : 0.5% of the rows are the only one in their cell
#>   loo         : 1.61 (leave-one-out mean squared error of the pooled value);
#>                 one mean for all rows: 10.8
#>   interval    : 90% for ONE OBSERVATION (..obs_lo_resp_val, ..obs_hi_resp_val):
#>                 value - 2 to value + 2.09, quantiles of 400 leave-one-out residuals
#> Away from the observations no interval here is calibrated.