Draw a sample to screen by hand

Picks documents out of a gr_screen() run for a person to judge without seeing what the model said, and writes them to a CSV to fill in. Feeding the completed file back to gr_calibrate() is what turns "we used an LLM" into a claim with a number attached.

Usage

gr_reference(
  screening,
  n = 50,
  of = c("excluded", "kept", "all"),
  seed = NULL,
  path = NULL,
  blind = TRUE
)

Arguments

screening

A gr_screening from gr_screen().

n

How many to draw. Inf takes the whole frame.

of

Which rows to sample from: "excluded", "kept" (include and unclear), or "all".

seed

Passed to withr::with_seed() so the draw is reproducible. A calibration sample is part of the method and has to be re-drawable.

path

Where to write the CSV. NULL returns the frame without writing.

blind

Leave the model's decision and reason out of the file. On by default: a person shown the answer agrees with it, and the resulting figure measures nothing.

Value

A data frame of class gr_reference_frame, invisibly when written. The human_decision column is empty and is yours to fill with include or exclude.

Which rows to sample

of = "excluded" is usually the right answer and is not the obvious one. Sensitivity failures hide among the exclusions: a study the screener threw away is gone, and nothing downstream will ever mention it, while the records it kept are going to be read by a person anyway. Sampling everything at a realistic inclusion rate spends most of the sample confirming exclusions that were never in doubt, and leaves two or three positives to estimate sensitivity from, which is no estimate at all.

The design that matches how people actually work is to judge every record the screener kept, plus a sample of what it discarded: of = "kept" with n = Inf, and of = "excluded" with whatever hand-screening effort you have. gr_calibrate() knows which frame it was given and will not compute a corpus-wide figure from a sample that cannot support one.

See also

gr_calibrate(), gr_screen()

Other corpus functions: gr_calibrate(), gr_inventory(), gr_records(), gr_search()

Examples

tab <- data.frame(document = paste0("d", 1:6, ".pdf"),
                  decision = c("include", "exclude", "exclude",
                               "unclear", "exclude", "include"),
                  reason = "because", stringsAsFactors = FALSE)
gr_reference(structure(list(table = tab), class = "gr_screening"),
             n = 3, of = "excluded", seed = 1)
#>   document human_decision human_note sampled_from frame_n screened_n
#> 1   d2.pdf           <NA>       <NA>     excluded       3          6
#> 2   d3.pdf           <NA>       <NA>     excluded       3          6
#> 3   d5.pdf           <NA>       <NA>     excluded       3          6