The result of one reading run

Returned by gr_read() and, with three extra fields, by answer_document().

Fields

answer

Character(1). Always a single string. The sentinel "NOT_IN_DOCUMENT" means the model reported the document does not contain the answer. Test it with is_not_found()-style matching rather than substring search.

premise

Present only when the read spec checked the question's premise (premise = TRUE in gr_read_spec()). NA when the model reported nothing in the document contradicting the question; otherwise the sentence in which it said what the document states instead, which is then answer as well, so is_not_found() is FALSE. It is the model's verdict, not a check harnessr made. Absent (NULL) for a spec that did not ask, and for screen and extract, which answer no question.

partial

Logical(1). TRUE when anything degraded: a call failed, chunks were dropped, a cap was hit, a strategy fell back, text was cut to fit, or pages of the document never became text (a scan read without OCR). Check this before trusting an answer.

notes

List. Why it is partial, and per-reader detail: dropped_chunks, failed_calls, error, merge_levels, degraded_to_bm25, stop_reason, call_cap_reached, cost_cap_reached, summaries_truncated, and so on. Three can be set for every reader: cited_unknown, chunk ids the answer cited that were never sent; unverified_evidence, the number of quoted spans that are not in the chunk they claim to come from; and unread_pages, the pages that never became text. Any of them makes the answer partial.

warnings

Character. What harnessr warned about while the document was ingested, cut and read, named by the warning's class. The warnings still print as they happen; this copy stays with the answer, including when the document came from the ingestion cache and nothing was raised again.

evidence

Data frame or NULL, with columns chunk_id, text, page, section, score, kind. What text holds depends on the reader: verbatim chunk text for stuff, retrieve, rerank and iterative; model-extracted passages for skim; per-chunk model answers for map_reduce. refine and hierarchical return NULL. page is populated only for PDF sources; score only for retrieve (cosine) and rerank (0-10, model-judged). Where the evidence is model-written (skim), three more columns appear: source_text, the chunk the span claims to quote, plus verified and match from checking one against the other; see gr_verify_evidence(). Readers whose evidence is verbatim chunk text do not carry them, because the span and its source are the same string. kind says what that row holds ("verbatim", "extracted" or "answer") per row, because an ensemble mixes them. A blank roxygen line inside a \describe{} item ends the item, which is why this is one paragraph.

chunks_used

Integer vector of chunk_ids that CONTRIBUTED to the answer. For the per-chunk readers this is a subset of the chunks sent: a chunk that answered NOT_IN_DOCUMENT was read and paid for but is not listed. notes$chunks reports how many were sent.

reader, signature

Which strategy ran, and its traversal signature (see gr_reader_signature()).

question

The question, as asked.

trace

The gr_trace for this run. From answer_document() it holds the whole run: ingestion, segmentation and the read. An answer taken from a gr_compare() holds only its own recipe's segmentation and read (its segmentation requests, which semantic, proposition and contextual with context_source = "llm" bill, the pre-flight check and the reader's requests), because each recipe is counted apart. Ingestion and the other recipes are in the comparison's own trace, cmp$trace: cost or save that one as the record of what the comparison did. A trace passed in with trace = is shared, not copied: this field is that trace itself, which still holds every run recorded into it, those before this one and those after it. trace_steps says which of its steps are this run's.

trace_steps

Integer c(first, last): the steps of trace this run made, or NULL when no trace was kept. as_json(), print() and the audit report (gr_audit_report()) use only these, so an answer read with a trace shared across runs does not carry other runs' prompts, calls or cost.

recipe, document, segmentation

Added by answer_document() and gr_compare(): the recipe name, the source and ingestion stats, and the chunk statistics the reader saw.

Methods

print() shows the answer, this run's calls, tokens and cost, where the evidence came from, and, when the answer is partial, why. When the model reported that the document contradicts the question's premise, it says so above the answer, as the model's report. as_json() serialises the answer (and premise, when it was checked) together with every prompt and response this run made (the steps of its trace that trace_steps names; for an answer from gr_compare(), its recipe's read alone; see trace above).

See also

answer_document() and gr_read() which return one, gr_compare() to compare several, is_not_found() to test the sentinel, as_json(), new_answer() to build one in a custom reader

gr_read(), answer_document(), as_json()

Other reading functions: gr_check_reader(), gr_read(), gr_read_spec(), gr_reader_signature(), gr_readers(), gr_register_reader(), is_not_found(), new_answer()

Examples

cl <- gr_mock_client(function(m, p) "Revenue was 45.2 million dollars.")
ans <- answer_document(gr_example(), "What was revenue?", "fast", client = cl)
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
ans$partial
#> [1] FALSE
ans$evidence[, c("chunk_id", "page", "score")]
#>   chunk_id page score
#> 1        1   NA    NA