Ask one question of many documents

The counterpart to gr_compare(): that runs several recipes over one document, this runs one recipe over many. Returns one tidy row per document, so the result goes straight into a data frame you can write out, join, or code against.

Usage

gr_read_many(
  sources,
  question,
  recipe = "thorough",
  client = NULL,
  store = NULL,
  on_error = c("continue", "stop"),
  max_total_usd = NULL,
  max_total_calls = NULL,
  keep_answers = TRUE,
  recursive = FALSE,
  trace = NULL,
  ...
)

Arguments

sources: A character vector of file paths, or a single directory, or raw text, or a gr_records() or a gr_screen() result, either of which also carries the search forward to the audit. A directory, and a vector of paths that all exist, are both filtered to the extensions some registered extractor claims, so which files are picked up follows gr_extractors(), including any you registered yourself. Raw text, a mixed vector and a list() of sources are passed through untouched, and a web address is downloaded as gr_ingest() downloads one.
question: The question, asked of every document.
recipe: One recipe, applied to every document.
client: A gr_client. Wrap it in gr_cache_client() for a long run: with a durable cache directory, a restart pays for nothing it has already answered. A closure-backed client (gr_backend_client() or gr_mock_client()) reuses a cache or a store across sessions only if it was given a stable id; see gr_backend_client() for why.
store: Optional directory. Each document's result is written there as it completes and restored on a later run instead of being read again. This is what makes a four-hour run survive being interrupted. Only a document read in full is written, so one that failed (see status below) is read again by the next run. An entry is restored only for the same document, question, recipe, tokenizer and client configuration: model, endpoint, extra_body, embedding model and embedder. A file is the same document when its path, size and content are, whatever its modification time or the session's time zone. An entry that is not plain data, as a file planted in a shared directory could be, is ignored and the document read again.
on_error: "continue" (default) records the failure and moves on; "stop" aborts. One unreadable file in two hundred should not cost you the other hundred and ninety-nine.
max_total_usd: Stop once the run has spent this much, marking the remaining documents "skipped", except any restored from store, which costs nothing. This is a corpus ceiling and is separate from gr_options(max_cost_usd =), which is a limit per document. It needs a model with a registered price: against one without, cost is unknown rather than zero, the ceiling cannot be enforced, and you get a gr_corpus_cost_unknown warning instead of a silent free pass. The one exception is an embedding model with no price (the chat model priced): the ceiling is then checked against the priced spend, which leaves out only the embeddings requests, and a gr_corpus_cost_floor warning names the model and says how to register it. cost_usd stays NA either way, since what those documents cost in full is not known.
max_total_calls: Stop before a document once the run has made this many requests, marking the rest "skipped", except any restored from store, which makes no request. Requests are counted as gr_options(max_calls =) counts them: model calls and requests to an embeddings endpoint alike, so a document read with "needle" through an API embedder spends more of it than its model calls alone, and more than the "model calls" a printed trace shows. The counterpart to max_total_usd for runs whose model has no registered price, and the only ceiling that bounds the run rather than each document: gr_options(max_calls =) is per document, so a corpus can make length(sources) times that many. Checked before each document, because a call ceiling noticed after the calls is not a ceiling. This means the run can overshoot by at most one document's worth, exactly as max_total_usd does.
keep_answers: Keep every gr_answer in the result. Set FALSE for a large corpus, where holding every trace and evidence table is the thing that runs you out of memory.
recursive: Descend into subdirectories when sources is a directory.
trace: A gr_trace() to fold this run's accounting into, so several stages of one review add up to one figure. It is a parent: this run still gets its own trace, which is what $trace returns and what gr_options(max_calls =) is measured against. Running a stage directly on a shared trace would charge the previous stage's calls against this one's ceiling. Omit it and there is no parent.
...: Overrides applied to the recipe, as in answer_document().

Value

An object of class gr_corpus: summary (one row per document), answers (named list, empty when keep_answers = FALSE), sources (the sources as read, aligned row for row with summary; summary$document is a display label and cannot be turned back into a path), records (the gr_records() the corpus came from, or NULL), trace (every call made this run) and store.

The summary

document, document_id, answer, not_found, partial, reader, chunks, chunks_used, calls, cached, tokens_in, tokens_out, cost_usd, seconds, status, duplicate_of, error, warnings.

warnings holds what harnessr warned about while reading that document, joined with " | ", or NA when it raised nothing. The warnings still print as they happen; this column is what lets you tie them to a document afterwards.

With premise = TRUE in the recipe's read spec (see gr_read_spec()) there is one more column, after warnings: premise, the sentence in which the model said what the document states instead of the question's premise, or NA when it reported nothing contradicting it. Such a row's answer is that sentence and its not_found is FALSE.

document is a filename and document_id is the hash of the cleaned text (and of the pages that never became text, when there are any). Cite with the second: a filename changes when the file is renamed, collides between folders, and does not exist at all for a document passed as text, while the id is the same string for the same document in every run and on every machine with the same OCR setup. Two copies of one paper share an id, which is the same fact as the duplicate detection below.

status is "ok", "failed", "skipped" (the corpus ceiling was reached first), "restored" (read from store, not re-read now) or "duplicate" (see below). A document that max_calls or max_cost_usd stopped before it was read in full is "failed" too, with the limit in error and its partial answer in answers; it is not written to store, so a resumed run with a higher limit reads it again. So is a document one of whose model requests failed (a network error, a 5xx or 429 after the retries, a refusal, a reply that was not the JSON asked for): error says how many and gives the first error, the partial answer is in answers, and a resumed run reads it again rather than restoring what the failure left. A request the pipeline recovered from does not count: an embeddings request that failed and was replaced by lexical vectors leaves the document "ok" and stored. partial is set when those vectors ranked the chunks the reader sent (retrieve, rerank, iterative), but not when they only placed a semantic segmenter's cuts; that, like a proposition batch kept as written, shows in warnings, and a contextual header that could not be written only in the trace's errors (see gr_trace()). A row that a limit or a failed request stopped keeps document_id, reader and the chunk counts, since the text was read; only answer and not_found are left NA. A restored row keeps the numbers from when that document was first read, so its cost_usd is what it cost then, not what this run spent. That is why the run's own spend comes from gr_trace_cost(x$trace) and not from summing the column.

Documents that are the same document

The same paper reaches you from three databases under three filenames. Each source is extracted and cleaned, and a document whose cleaned text (and set of unread pages) is identical to one already read this run is not read again: its row is filled in from the first copy, except for warnings, which are its own; status is "duplicate" and duplicate_of names the row it repeats. That is decided afresh in every run, a restored row included: a copy whose first copy is not in this run is the first copy now. Nothing is dropped (every source you passed still has a row), so subset(x$summary, is.na(duplicate_of)) is the deduplicated set and sum(!is.na(x$summary$duplicate_of)) is the number to report as removed.

This is about the table, not the bill: a gr_cache_client() already makes the second copy's calls free. What it could not do is stop the duplicate from appearing in the results as a second, independent document, which is how one study gets counted twice in a synthesis.

The comparison is exact, on cleaned text. Two typesettings of one paper are two documents here; matching those needs bibliographic metadata, not text.

Budgets

Every document gets its own trace, so gr_options(max_calls =) and gr_options(max_cost_usd =) apply per document exactly as they would if you read it alone. One enormous document therefore cannot starve the rest.

That is a deliberate design and it leaves the run itself unbounded: two hundred documents under a 400-call ceiling is a corpus ceiling of eighty thousand calls. The run-level ceilings are max_total_calls, checked before each document, and max_total_usd, checked after each one because what a document costs is not knowable until it has been read. With neither set, the run says once what its worst case is rather than leaving you to multiply.

What this does not do

It reads documents one at a time. Per-document work is embarrassingly parallel, and the per-worker traces the parallel helper already builds would carry the accounting across, so the obstacle is not the trace: it is that duplicate detection, the resume store and both run-level ceilings are all order-dependent, and a parallel loop would have to serialise on each of them. Within a document, gr_options(parallel = TRUE) already applies.

Examples

cl <- gr_mock_client(function(m, p) "Revenue was 45.2 million dollars.")

a <- tempfile(fileext = ".txt"); writeLines("Revenue was 45.2 million.", a)
b <- tempfile(fileext = ".txt"); writeLines("Revenue was 51.8 million.", b)

out <- gr_read_many(c(a, b), "What was revenue?", "fast", client = cl)
#> 2 document(s), and gr_options(max_calls) is 400 PER DOCUMENT, so this run may make up to 800 call(s). Pass max_total_calls = to cap the run.
#> [1/2] file1df638ed1516.txt
#> Extracting 'file1df638ed1516.txt' with the 'txt' extractor.
#> Ingested 1 block(s), ~11 tokens (0 chars removed by cleaning).
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> [2/2] file1df654361ef5.txt
#> Extracting 'file1df654361ef5.txt' with the 'txt' extractor.
#> Ingested 1 block(s), ~11 tokens (0 chars removed by cleaning).
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
out$summary[, c("document", "answer", "not_found", "status")]
#>               document                            answer not_found status
#> 1 file1df638ed1516.txt Revenue was 45.2 million dollars.     FALSE     ok
#> 2 file1df654361ef5.txt Revenue was 45.2 million dollars.     FALSE     ok

# A missing file is one bad row, not a failed run.
bad <- gr_read_many(c(a, "no-such-file.txt"), "What was revenue?", "fast", client = cl)
#> 2 document(s), and gr_options(max_calls) is 400 PER DOCUMENT, so this run may make up to 800 call(s). Pass max_total_calls = to cap the run.
#> [1/2] file1df638ed1516.txt
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> [2/2] no-such-file.txt
#> Warning: Document 'no-such-file.txt' failed: File not found: 'no-such-file.txt'. If you meant to pass document text rather than a path, it must not end in something that looks like a file extension.
bad$summary[, c("document", "status", "error")]
#>               document status
#> 1 file1df638ed1516.txt     ok
#> 2     no-such-file.txt failed
#>                                                                                                                                                       error
#> 1                                                                                                                                                      <NA>
#> 2 File not found: 'no-such-file.txt'. If you meant to pass document text rather than a path, it must not end in something that looks like a file extension.

# What the run actually cost, counting only calls that were really issued.
gr_trace_cost(out$trace)
#>        model calls paid_calls paid_in paid_out usd paid_reasoning
#> 1 mock-model     2          2     190       26   0              0
#>   paid_cached_in
#> 1              0