Draw a sample to screen by hand

Picks documents out of a gr_screen() run for a person to judge without seeing what the model said, and writes them to a CSV to fill in. Feeding the completed file back to gr_calibrate() is what turns "we used an LLM" into a claim with a number attached.

Usage

gr_reference(
  screening,
  n = 50,
  of = c("excluded", "kept", "all"),
  seed = NULL,
  path = NULL,
  blind = TRUE
)

Arguments

screening

A gr_screening from gr_screen().

n

How many to draw: a whole number of at least 1. Inf takes the whole frame, as does any n larger than it.

of

Which rows to sample from: "excluded", "kept" (include and unclear), or "all".

seed

A seed for the draw, so it is reproducible: the sample is drawn after set.seed(seed) and your session's random number stream is put back afterwards. A calibration sample is part of the method and has to be re-drawable.

path

Where to write the CSV. NULL returns the frame without writing.

blind

Leave the model's decision and reason out of the file. On by default: a person shown the answer agrees with it, and the resulting figure measures nothing. A blind sheet does not name its frame either, since "excluded" on every row says what the model decided: sampled_from holds an opaque key that gr_calibrate() reads back as the frame and its size. With blind = FALSE the file has model_decision, model_reason, and the frame in plain words (sampled_from, frame_n, screened_n).

Value

A data frame of class gr_reference_frame, invisibly when written. The human_decision column is empty and is yours to fill with include or exclude.

Which rows to sample

of = "excluded" is usually the right answer and is not the obvious one. Sensitivity failures hide among the exclusions: a study the screener threw away is gone, and nothing downstream will ever mention it, while the records it kept are going to be read by a person anyway. Sampling everything at a realistic inclusion rate spends most of the sample confirming exclusions that were never in doubt, and leaves two or three positives to estimate sensitivity from, which is no estimate at all.

The design that matches how people actually work is to judge every record the screener kept, plus a sample of what it discarded: of = "kept" with n = Inf, and of = "excluded" with whatever hand-screening effort you have. gr_calibrate() knows which frame it was given and will not compute a corpus-wide figure from a sample that cannot support one.

The file

The CSV is written as UTF-8 with a byte-order mark, which is what makes Excel open accented names and titles correctly; save it back as "CSV UTF-8". gr_calibrate() reads it the same way in any locale, reads a sheet a spreadsheet re-saved as Windows-1252 as that, and matches a row whose name still does not agree by its document_id.

See also

gr_calibrate(), gr_screen()

Other corpus functions: gr_calibrate(), gr_inventory(), gr_records(), gr_resolve(), gr_search()

Examples

tab <- data.frame(document = paste0("d", 1:6, ".pdf"),
                  decision = c("include", "exclude", "exclude",
                               "unclear", "exclude", "include"),
                  reason = "because", stringsAsFactors = FALSE)
gr_reference(structure(list(table = tab), class = "gr_screening"),
             n = 3, of = "excluded", seed = 1)
#>   document human_decision human_note                       sampled_from
#> 1   d2.pdf           <NA>       <NA> grd5b346bbe4d898050402719dd5027962
#> 2   d3.pdf           <NA>       <NA> grd5b346bbe4d898050402719dd5027962
#> 3   d5.pdf           <NA>       <NA> grd5b346bbe4d898050402719dd5027962