Choosing how the model reads

Once a document is cut into chunks, something has to decide how the model works through them. Send everything at once? Ask each chunk separately and combine the answers? Find the few chunks most like the question and ignore the rest? These are reading strategies, which harnessr calls readers. They decide most of the cost of a run and much of the quality of its answer. The same chunks read two ways can take one request or dozens, and can find a fact or miss it.

This guide explains each reader in plain terms, shows how they differ on the same document, and gives a way to choose. vignette("harnessr") introduces the package and its ideas.

Setting up

The examples use a stand-in for a real model, so they run for free. This one answers each kind of request the way a model would: it scores an excerpt when asked to score and quotes when asked to quote, so every reader can do its normal work. You do not need to follow its details; it plays the part of the model.

library(harnessr)
old <- gr_options(verbose = FALSE, embedder = "lexical")

stand_in <- function(messages, params) {
  instructions <- messages[[1]]$content
  everything <- paste(vapply(messages, function(m) m$content, character(1)),
                      collapse = "\n")
  task <- if (is.null(params$schema_name)) "" else params$schema_name
  if (task == "relevance") {        # rerank: score one excerpt from 0 to 10
    excerpt <- messages[[length(messages)]]$content
    if (grepl("revenue", excerpt, ignore.case = TRUE)) {
      return('{"score": 9, "reason": "States revenue."}')
    }
    return('{"score": 2, "reason": "About something else."}')
  }
  if (task == "iterative_step") {   # iterative: can I answer yet?
    return('{"can_answer": true, "answer": "Revenue was 45.2 million dollars.", "next_query": ""}')
  }
  if (task == "reading_plan") {     # preview: which sections to read
    return('{"sections": [{"id": 1, "treatment": "read", "reason": "States revenue."}]}')
  }
  if (grepl("You extract evidence", instructions, fixed = TRUE)) {   # skim: quote
    quote <- regmatches(everything,
                        regexpr("[^.\n]*revenue of [0-9.]+ million dollars[^.\n]*", everything))
    return(if (length(quote)) quote else "NONE")
  }
  "Revenue was 45.2 million dollars."
}

embedder = "lexical" makes the readers that search for similar text use simple word matching instead of a paid embedding service; more on that below.

Now the document, cut into chunks along its section headings:

chunks <- gr_segment(gr_ingest(gr_example()),
                     list(method = "structural", max_tokens = 120))
gr_chunk_stats(chunks)
#>       method n total_tokens min median mean max over_cap
#> 1 structural 8          557  16   76.5 69.6 102        0

Eight chunks. A real report would have hundreds, and the differences below would be correspondingly larger.

Listing the readers

gr_readers()[, c("name", "signature", "cost_calls")]
#>            name             signature
#> 1      ensemble   ensemble|sum+1|none
#> 2       extract  all|N+conflicts|none
#> 3  hierarchical     all|N+tree+1|tree
#> 4     iterative topk|rounds*2|forward
#> 5    map_reduce       all|N+logN|tree
#> 6       preview    planned|1+s+1|none
#> 7        refine         all|N|forward
#> 8        rerank         topk|m+1|none
#> 9      retrieve           topk|1|none
#> 10       screen           head|1|none
#> 11         skim          all|N+1|none
#> 12        stuff            all|1|none
#>                                    cost_calls
#> 1                          sum of members + 1
#> 2  N (+1 per conflict with resolve = 'model')
#> 3                       N + fan-in levels + 1
#> 4                        up to 2 x max_rounds
#> 5                                  N + merges
#> 6                    1 + skimmed sections + 1
#> 7                                           N
#> 8                                       m + 1
#> 9                              1 + embeddings
#> 10                                          1
#> 11                                      N + 1
#> 12                                          1

cost_calls is how many requests a reader makes, as a formula in N, the number of chunks. A reader whose count grows with N makes a hundred times as many requests on a document a hundred times longer. But you pay for tokens, not requests, and a reader that looks at every chunk sends every token: stuff makes one request whatever the length, yet that request carries the whole document, so its cost grows with the document too. Each request also repeats the instructions and writes its own reply, so among readers that see everything, more requests cost more. The readers that look at a selection of chunks (the topk ones below) cost roughly the same however long the document is.

signature describes how a reader works, in three parts separated by |:

which chunks it looks at: all of them; the topk most relevant; the head (opening) of the document; the chunks a planned reading chose; or, for an ensemble, whatever its members looked at;
how many requests it makes, the same formula in short;
what carries over between requests: none (each request stands alone), tree (answers are combined in stages), or forward (each request builds on the one before).

The built-in readers all have different signatures, and ensemble refuses members that share one: two readers with the same signature would be the same method under two names.

Every reader on one document

Here is the same question put to the same eight chunks by each general-purpose reader. extract and screen are left out: they are built for literature reviews and need a schema or criteria.

read_with <- function(reader) {
  model <- gr_mock_client(stand_in)
  ans <- gr_read(chunks, "What was revenue in 2024?", model, reader)
  data.frame(reader = reader, requests = length(model$calls()),
             chunks_used = length(ans$chunks_used), partial = ans$partial)
}
readers <- c("stuff", "map_reduce", "refine", "skim", "hierarchical",
             "retrieve", "rerank", "iterative", "preview", "ensemble")
do.call(rbind, lapply(readers, read_with))
#>          reader requests chunks_used partial
#> 1         stuff        1           8   FALSE
#> 2    map_reduce        9           8   FALSE
#> 3        refine        8           8   FALSE
#> 4          skim        9           1   FALSE
#> 5  hierarchical        9           8   FALSE
#> 6      retrieve        1           6   FALSE
#> 7        rerank        9           2   FALSE
#> 8     iterative        1           6   FALSE
#> 9       preview        2           8   FALSE
#> 10     ensemble       11           8   FALSE

The stand-in gives every reader the same answer, and none of the runs degraded (partial is FALSE). What differs is the work: from one request to eleven for eight chunks, and from reading everything to relying on a single passage. On a long document the gap widens: the number of requests grows with the document for some readers and not at all for others, and the cost grows for every reader that reads the whole document.

The readers

`stuff`: everything in one request

Puts the whole document into a single prompt. It is the simplest reader, and usually the cheapest way to have the model read all of a document, with one set of instructions and one reply. Its cost still grows with the document’s length. The model sees everything at once, so it can connect facts from different places. The limit is the context window: if the document does not fit, stuff sends what fits, warns, and marks the answer partial (on_overflow = "error" makes it stop instead). Very long prompts also dilute attention; details in the middle of a long prompt are more easily missed.

Use it for documents comfortably inside the model’s context window. Recipe: "fast", which answer_document() uses by default for a document of up to 50,000 tokens, or less on a model with a small context window.

`map_reduce`: every chunk separately, then combined

Asks the question of each chunk on its own (“map”), then combines the answers in stages (“reduce”). Nothing is skipped, each request is small, and the per-chunk requests can run in parallel. It costs N requests plus the combining steps, and a fact that only makes sense across two chunks can be missed, because no single request sees both.

Use it for questions where every mention matters: “list every risk named in this report”. Recipe: "thorough", which answer_document() uses by default for a longer document.

`refine`: a draft revised chunk by chunk

Reads the chunks in order, keeping a running answer and revising it with each new chunk. Later text can correct earlier conclusions, which suits documents whose argument develops. It costs N requests that cannot run in parallel, and the running answer can drift.

Use it for narratives and arguments that build. Recipe: "narrative".

`skim`: quote first, then answer

For each chunk, asks the model to copy out any passages that bear on the question, then answers once from those passages alone. The final answer rests on quotations, which harnessr checks against the text (see below). It costs N + 1 requests; the per-chunk step can use a cheaper model through skim_model.

Use it for questions where you want the answer tied to exact wording. Recipe: "precise".

`hierarchical`: summaries of summaries

Summarises each chunk, then summarises groups of summaries (fan_in at a time), level by level, until everything fits in one prompt, and answers from that. It suits questions about a whole long document, such as “what is this report’s overall argument?”, at the price of detail lost in summarising. summary_model can use a cheaper model for the summaries; max_levels caps the depth.

Use it for long, structured documents where you want an overview. Recipe: "survey".

`retrieve`: only the most relevant chunks

Turns the question and every chunk into embeddings (lists of numbers that place similar text close together) and sends only the top_k chunks closest to the question, in a single request. The cost of the answer does not grow with the document, which makes it the usual choice for finding one fact in something long. The risk: a chunk that answers the question but does not look like it is never sent.

Use it for a specific fact in a long document. Recipe: "needle".

`rerank`: shortlist, then let the model judge

Makes a quick word-matching shortlist (rerank_candidates chunks, 20 by default), asks the model to score each for relevance from 0 to 10, and answers from the chunks scoring at least rerank_min_score. The model’s judgement is better than word matching, at the price of one scoring request per shortlisted chunk. skim_model can do the scoring more cheaply.

If no chunk could be scored, for example because the provider does not support the structured replies the scoring needs, it falls back to the word-matching order and warns. If only some chunks were scored, the answer is marked partial.

Use it for long documents where word matching alone picks badly, including scanned ones. Recipe: "scanned".

`iterative`: look, decide what is missing, look again

Retrieves a few chunks and asks the model whether it can answer yet, and if not, what it should look for next. Then it retrieves again, up to max_rounds times. It suits questions needing several facts from different places (“how did the drop in margin affect the dividend?”). ans$notes$queries records what it searched for in each round, and ans$notes$stop_reason why it stopped.

Use it for multi-step questions. Recipe: "research".

`preview`: plan, then read

Shows the model an outline of the document (section headings, sizes, and short excerpts) and asks which sections to read in full, which to skim, and which to skip. Then it reads accordingly. A section the plan does not mention is read, never skipped, so silence cannot lose part of the document; the plan is recorded in ans$notes$plan, so “sections 3 and 5 were not read” is visible. If no usable plan comes back, it reads everything and marks the answer partial.

Use it for long documents with clear sections, when most of them are irrelevant to the question.

`ensemble`: several readers, compared

Runs two or more different readers (members; by default retrieve and map_reduce) and asks the model to reconcile their answers. It costs the sum of its members plus one request, and pays off when a wrong answer is expensive. Members must have different signatures, because running the same method twice is not a second opinion.

Use it for high-stakes questions. Recipe: "consensus".

`extract` and `screen`

extract fills in a table of details (a schema) from every chunk, and screen decides whether a document meets a set of criteria. Both belong to the literature-review workflow in vignette("review").

Choosing a reader

your situation	reader	recipe	cost grows with document length?
the document fits comfortably in one request	`stuff`	`"fast"`	yes, though it is one request
one specific fact, long document	`retrieve`	`"needle"`	no (embeddings aside)
one fact, and word matching picks badly; scanned pages	`rerank`	`"scanned"`	no, fixed by `rerank_candidates`
every mention must be found	`map_reduce`	`"thorough"`	yes
the answer must rest on exact quotations; a short document with every sentence weighed	`skim`	`"precise"`	yes
an overview of a long document	`hierarchical`	`"survey"`	yes
an argument that develops through the text	`refine`	`"narrative"`	yes
several facts combined from different places	`iterative`	`"research"`	no, up to `max_rounds` (embeddings aside)
long document, most sections irrelevant	`preview`	none	depends on the plan
a wrong answer would be costly	`ensemble`	`"consensus"`	as its members

If unsure, run two or three on a document where you already know the answer and compare (see “Trying readers on your own documents” below).

Recipes

A recipe names an ingestion, a segmentation and a reader together, with their settings. The table above gives the recipe built on each reader. gr_recipes() returns them all:

do.call(rbind, lapply(names(gr_recipes()), function(n) {
  r <- gr_recipes(n)
  data.frame(recipe = n, segment = r$segment$method, reader = r$read$reader)
}))
#>       recipe    segment       reader
#> 1       fast  paragraph        stuff
#> 2    precise   sentence         skim
#> 3     needle   semantic     retrieve
#> 4   thorough  paragraph   map_reduce
#> 5     survey structural hierarchical
#> 6  narrative  paragraph       refine
#> 7    scanned       page       rerank
#> 8   research structural    iterative
#> 9  consensus  recursive     ensemble
#> 10    legacy  paragraph   map_reduce

answer_document() defaults to "auto", which reads a document of up to 50,000 tokens (less on a model with a small context window) with "fast", in one request, and a longer one with "thorough", one request per chunk. Both send every chunk. ans$recipe says which was used. gr_read_many(), gr_compare() and the review functions take one fixed recipe and refuse "auto".

Printing a recipe shows every setting on all three steps:

gr_recipes("thorough")
#> <gr_recipe 'thorough'>
#>   ingest  : clean=standard ocr=auto
#>   segment : paragraph (max 1200 tok, overlap 120, min 0)
#>   read    : map_reduce [all|N+logN|tree] model=(the client's)

A recipe is a starting point. Any of its settings can be overridden by name in the call, as in answer_document(file, question, "needle", top_k = 3), and gr_recipe() builds one from scratch (there is an example under “Trying readers on your own documents” below).

"legacy" reproduces the previous release’s behaviour (digit stripping, 3000-token paragraph chunks, no overlap), so you can measure the difference rather than assume it.

Choosing and placing chunks

retrieve, rerank and iterative choose which chunks the model sees. These settings control the choice and where the chosen chunks go; each says which readers it applies to.

top_k (retrieve, rerank, iterative) is how many chunks to send, 6 by default. For iterative it is how many each round adds: the chunks gathered in earlier rounds stay in the prompt as long as they fit. Too few and the answer can be missed; too many and you pay for text that does not help.

min_score (retrieve) drops chunks less similar to the question than a threshold, though the single best chunk is always kept.

mmr (retrieve, iterative) stops near-duplicates crowding each other out. Plain top-k picks the chunks most like the question; if three paragraphs say the same thing, all three get in. Below 1, mmr balances relevance against similarity to what is already picked (0.7 is a reasonable start):

repetitive <- paste(c(
  "Revenue was 45.2 million dollars in fiscal 2024.",
  "Total revenue reached 45.2 million dollars in the 2024 fiscal year.",
  "In fiscal 2024 the company recorded revenue of 45.2 million dollars.",
  "Headcount grew to 1,204 employees across nine clinical sites.",
  "The board approved a dividend of 0.42 dollars per share in March."),
  collapse = "\n\n")
pieces <- gr_segment(gr_ingest(repetitive), list(method = "paragraph", max_tokens = 40))
model <- gr_mock_client(stand_in)
picked <- function(mmr) {
  gr_read(pieces, "What was revenue?", model,
          list(reader = "retrieve", top_k = 3, mmr = mmr))$chunks_used
}
rbind("top-k (mmr = 1)" = picked(1), "mmr = 0.3" = picked(0.3))
#>                 [,1] [,2] [,3]
#> top-k (mmr = 1)    1    3    2
#> mmr = 0.3          1    4    5

Plain top-k spent all three places on the same fact; mmr = 0.3 kept the best one and used the other two for something different. It costs nothing extra.

context_order (retrieve, rerank, iterative) decides where the chosen chunks sit in the prompt. Models attend best to the beginning and end of a long prompt, so "edges" puts the best chunk first and the second-best last; "document" keeps the order they appear in the document, which reads more naturally when chunks are consecutive; the default "relevance" puts the best first. It never changes which chunks were chosen, only where they sit.

restate (every reader but extract and screen) repeats the question before the excerpts as well as after them, because a question asked once, after several thousand tokens of text, is a long way from the start of the prompt. "auto" does this when the text is long, "always" and "never" override.

Whether mmr, context_order or restate help depends on your documents and model; test them rather than assume.

Embeddings

retrieve, iterative and the semantic way of cutting chunks (vignette("segment")) need embeddings. By default harnessr asks the model provider for them, which costs a little and needs an internet connection. gr_options(embedder = "lexical") uses word-matching vectors computed locally instead: free, offline and exactly repeatable, but they match words rather than meaning, so “turnover” will not find “revenue”. Use lexical vectors for practice and testing, and real embeddings for real work.

gr_embedders()[, c("name", "deterministic", "description")]
#>      name deterministic
#> 1     api         FALSE
#> 2 lexical          TRUE
#>                                                    description
#> 1                 Embeddings endpoint on the client's base URL
#> 2 Hashed bag-of-words; free, offline, word overlap not meaning

Reasoning effort, stage by stage

The default models reason before they answer. That reasoning is billed as output and counts against the same cap as the reply, so it is part of what every request costs. effort in gr_read_spec() sets how much of it to do: one value, such as "low", for every request, or a vector named by stage. It is sent only to models harnessr lists as reasoning models (gr_model_info(model)$reasoning). By default nothing is sent, and each request goes out at the provider’s default.

Every request a reader makes carries a label naming its stage:

reader	one request per chunk	the rest
`stuff`	none	`stuff.answer`
`map_reduce`	`map.answer`	`reduce`
`refine`	`refine.draft`, then `refine.revise`	none (the last revision is the answer)
`skim`	`skim.extract`	`skim.consolidate` (only if the passages do not fit), `skim.answer`
`hierarchical`	`hier.summarise.L1`	`hier.summarise.L2` and up, `hier.answer`
`retrieve`	none	`retrieve.answer`
`rerank`	`rerank.score` (each shortlisted chunk)	`rerank.answer`
`iterative`	none	`iterative.step` (one a round), `iterative.final` (only if no step could answer)
`preview`	`preview.skim` (each skimmed section)	`preview.plan`, `preview.answer`
`ensemble`	as its members	as its members, then `ensemble.adjudicate`

The per-chunk requests are many and each does a small job, so they are where a lower effort saves most; ?gr_read_spec suggests a starting profile. A name covers its own label and every label that starts with it and a dot, so map covers map.answer, and an unnamed value covers the rest.

gr_trace() records every request a run makes. Here map_reduce reads the eight chunks at low effort and merges at high:

spec <- gr_read_spec("map_reduce", model = "gpt-5.6-luna",
                     effort = c(map = "low", reduce = "high"))
tr <- gr_trace()
ans <- gr_read(chunks, "What was revenue in 2024?", gr_mock_client(stand_in), spec,
               trace = tr)
with(as.data.frame(tr), table(stage, effort))
#>             effort
#> stage        high low
#>   map.answer    0   8
#>   reduce        1   0

model = "gpt-5.6-luna" names a reasoning model: with the stand-in’s own model nothing would be sent, and the trace would have no effort column. A request at "high" can spend the whole output cap reasoning and come back cut off, so give it room with max_answer_tokens. gr_compare() (below) counts recipes that differ only in effort as different, so you can test what a lower effort costs. vignette("runs") covers effort across a literature review and what each trace step records.

Asking for citations

With cite = TRUE the model is asked to cite the chunks it used, as [chunk 3]. Each chunk id leads back to a page and section through ans$evidence:

citing <- gr_mock_client(function(messages, params)
  "Revenue was 45.2 million dollars [chunk 2].")
ans <- gr_read(chunks, "What was revenue in 2024?", citing,
               list(reader = "stuff", cite = TRUE))
ans$answer
#> [1] "Revenue was 45.2 million dollars [chunk 2]."
ans$evidence[ans$evidence$chunk_id == 2, c("chunk_id", "section")]
#>   chunk_id section
#> 2        2 Summary

A citation to a chunk the model was never sent is caught and recorded in ans$notes$cited_unknown, and marks the answer partial. hierarchical does not support citations, because it answers from summaries rather than chunks.

Checking what an answer rests on

ans$evidence holds different things depending on the reader:

reader	`evidence$text` holds
`stuff`, `retrieve`, `rerank`, `iterative`	the chunks, word for word
`skim`	passages the model copied out
`map_reduce`	each chunk’s answer
`refine`, `hierarchical`	nothing (the answer is built from drafts or summaries)

Chunks sent word for word are true copies of the document by construction. The passages skim collects are what the model says the document contains, so harnessr checks every one against the chunk it came from:

skimmed <- gr_read(chunks, "What was revenue in 2024?", gr_mock_client(stand_in), "skim")
gr_verify_evidence(skimmed)[, c("chunk_id", "verified", "match")]
#>   chunk_id verified match
#> 1        2     TRUE     1

Here is a model that invents one. The passage is fluent, on topic, attributed to a real chunk, and not in the document, so it cannot be caught by eye:

memo <- gr_segment(gr_ingest("Revenue rose to 45.2 million dollars.\n\nHeadcount grew to 1,204."),
                   list(method = "paragraph", max_tokens = 40))
inventing <- gr_mock_client(function(messages, params) {
  if (grepl("You extract evidence", messages[[1]]$content, fixed = TRUE)) {
    return("Revenue rose to 88.9 billion dollars on record demand.")
  }
  "Revenue was 88.9 billion dollars."
})
invented <- gr_read(memo, "What was revenue?", inventing, "skim")
gr_verify_evidence(invented)[, c("kind", "verified", "match")]
#>        kind verified match
#> 1 extracted    FALSE 0.333
invented$partial
#> [1] TRUE

match is 1 for an exact quotation once spacing, quote marks, dashes and capitals are folded away, since a faithful quotation can differ in those. A changed figure is not folded away. Below 1, match is the fraction of the passage carried by its longest consecutive run in the source, so where a change falls matters as much as how much changed: a changed last word leaves a run of nine in ten and scores 0.9, while a changed word in the middle splits the passage and scores about 0.5. Below about 0.3 there is no quotation left, only shared vocabulary. A run is used rather than word overlap because overlap cannot tell a quotation from a paraphrase built from the same words. A passage that does not verify is marked verified = FALSE and counted in ans$notes$unverified_evidence, and any one of them makes the answer partial.

gr_read_spec(verify = TRUE) runs the same check inside gr_read(): it writes the check’s columns onto $evidence, marks the answer partial when a quotation is not found, counts the quotations it checked in notes$quotes_checked, and records a read.verify step in the trace. It also holds iterative to quoting: each step must list the passages its answer rests on, word for word with their chunk numbers, and the loop stops only when every one is found in the chunk it names. A run that ends with none accepted keeps the refused answer in notes$unverified_answer and reports the answer as not in the document.

vspec <- gr_read_spec(reader = "skim", verify = TRUE)
checked <- gr_read(chunks, "What was revenue in 2024?", gr_mock_client(stand_in), vspec)
checked$notes$quotes_checked
#> [1] 1
checked$evidence[, c("verified", "match", "quote_status", "quote_sim")]
#>   verified match quote_status quote_sim
#> 1     TRUE     1     verbatim         1

The last columns come from a second matcher, recorded beside verified: quote_status, quote_sim, quote_coverage, quote_edits and quote_critical_ok. It expands ligatures, joins words hyphenated across a line, and scores the closest window of the source by character edits, with numbers and negations required to be identical. Its statuses are verbatim, near_verbatim, unverified, and not_checked for a passage too long to score. On a labelled set of 343 true and 162 false quotations it rejects 1.2% of true ones, against 36.4% for the existing check, and accepts none with a changed digit, number word, comparison sign or negation. It does accept 25 of 46 quotations with one content word swapped as near_verbatim, so it decides nothing in this release: verified and match still decide.

The other columns of ans$evidence are page, filled in only for a PDF, section, kind, and score, which only retrieve (similarity) and rerank (the model’s 0 to 10) set. This check and the citation check above are string comparisons on text you already have, so they cost nothing and always run.

Questions that assume something false

A question can take for granted something the document contradicts: “Why did revenue fall in 2024?” of a report in which revenue rose. A reader then either answers on the question’s terms or says the answer is not in the document, and either way what the document does say is lost. With premise = TRUE in gr_read_spec(), the model is told that if the excerpts state the opposite of something the question assumes, it should say so in one sentence and give what they state instead. That sentence is ans$answer and ans$premise, and is_not_found() is FALSE for it. Excerpts that are merely silent still count as not found.

A reader that answers in one request (stuff, retrieve, rerank, skim, hierarchical, preview) gives that request’s verdict. map_reduce passes a chunk that contradicts the question to its merge as a finding, and ans$notes$premise_chunks lists the chunks that said so, whatever the merge decided; ensemble does the same with its members (ans$notes$premise_members). ?gr_read_spec describes the rest.

This stand-in plays a model that finds the summary contradicts the question and the other chunks silent. FALSE_PREMISE: is how the model is asked to open such a reply:

loaded <- gr_mock_client(function(messages, params) {
  sent <- paste(vapply(messages[-1], function(m) m$content, character(1)),
                collapse = "\n")
  if (grepl("45.2 million", sent, fixed = TRUE)) {
    return("FALSE_PREMISE: Revenue rose to 45.2 million dollars, from 38.7 million.")
  }
  "NOT_IN_DOCUMENT"
})
ans <- gr_read(chunks, "Why did revenue fall in 2024?", loaded,
               gr_read_spec("map_reduce", premise = TRUE))
ans$premise
#> [1] "Revenue rose to 45.2 million dollars, from 38.7 million."
is_not_found(ans$answer)
#> [1] FALSE
ans$notes$premise_chunks
#> [1] 2

The verdict is the model’s reading of the excerpts, not something harnessr checks, so weigh it as you would any answer. The check is off by default because it makes the prompts longer, so a cache or a recorded run made without it does not answer a run with it. extract and screen ignore it.

Trying readers on your own documents

The best way to choose is to test on a few documents where you already know the answer. gr_compare() runs several configurations on one document, extracting the text once and sharing chunks between configurations that cut them the same way:

cmp <- gr_compare(gr_example(), "What was revenue in 2024?", list(
  gr_recipe("narrow", segment = list(method = "structural", max_tokens = 120),
            read = list(reader = "retrieve", top_k = 2)),
  gr_recipe("wide", segment = list(method = "structural", max_tokens = 120),
            read = list(reader = "retrieve", top_k = 6)),
  gr_recipe("all", segment = list(method = "structural", max_tokens = 120),
            read = list(reader = "map_reduce"))),
  client = gr_mock_client(stand_in))
cmp$summary[, c("recipe", "reader", "settings", "chunks_used", "not_found", "partial")]
#>   recipe     reader                settings chunks_used not_found partial
#> 1 narrow   retrieve top_k=2, max_tokens=120           2     FALSE   FALSE
#> 2   wide   retrieve          max_tokens=120           6     FALSE   FALSE
#> 3    all map_reduce          max_tokens=120           8     FALSE   FALSE

settings names whatever each configuration changed from the defaults, so rows that differ only in a number can be told apart. Two configurations that would do exactly the same work are run once, with a warning, rather than paid for twice.

When a provider cannot do structured replies

rerank, iterative and preview ask the model to reply in a fixed structure (JSON), which most current providers support. Where the structure is not supported or the reply cannot be read, each one falls back, warns, and marks the answer partial: rerank to word-matching order, iterative to a single retrieval, preview to reading everything. The answer is still produced; partial tells you it came by the fallback.

Every call that asks for JSON (these three, model segmentation, screening, claims, outlines and extraction) checks the reply’s types, enum values and required fields, through nested objects and arrays, before using it. A reply that fails is treated as one that did not parse: the reader takes its failure path, the reply is not cached, and the reason says where the reply broke the schema, as in $.groups[0]: expected array, got 1. gr_extract() still reads each field’s value itself, so “1,204” in an integer field is still 1204; only the shape of its replies is checked.

When the answer is not what you expected

The answer object and its trace record enough to find the cause without guessing. In order of how often each is the cause:

symptom	check	likely cause
`NOT_IN_DOCUMENT`, but you can see the answer in the file	`nrow(ans$evidence)`, then `gr_chunk_stats()`	the chunk holding it never reached the model. Lower `max_tokens`, raise `top_k`, or switch to a reader whose `signature` starts `all\\|`
the answer is right but thin	`ans$notes$chunks` against `length(ans$chunks_used)`	most chunks answered `NOT_IN_DOCUMENT`. That is usually correct; if not, the chunk boundaries are cutting the evidence in half, so add `overlap_tokens` (`vignette("segment")`)
`ans$partial` is `TRUE`	`print(ans)`, then `ans$notes` and `print(ans$trace)`	`failed_calls` (transport), `dropped_chunks` (did not fit), `call_cap_reached` or `cost_cap_reached` (`vignette("runs")`), `unread_pages` (no text layer and no OCR), or a merge that fell back to joining the answers
`ans$notes$unverified_evidence` is set	`gr_verify_evidence(ans)`	the model wrote a quotation that is not in the chunk it is attributed to. `match` says how far off: near 1 is a typo, near 0 is invention
`ans$notes$cited_unknown` is set	that value against `ans$chunks_used`	the answer cited a chunk that was never sent to it
figures, dates or percentages are missing	`doc$stats$clean_log`	a cleaning step removed them. `remove_numbers` is off by default; the `legacy` preset turns it on (`vignette("ingest")`)
a scanned PDF comes back nearly empty	`doc$stats$chars` and `doc$stats$unread_pages`, and any `gr_ocr_unavailable` warning	OCR did not run or is not installed. Force it with `gr_ingest_spec(ocr = "always")`, and check `tesseract` and `magick` are present
every chunk is the whole document	`nrow(doc$blocks)`	the file has no blank lines between paragraphs, so there is nothing to split on. Use `method = "sentence"` or `"fixed"`
the run costs far more than expected	`gr_readers()$cost_calls`	the reader makes one request per chunk and `max_tokens` is small. Run `gr_chunk_stats()` first; it is free
answers change between identical runs	the `temperature` setting, unset by default	set `temperature = 0`. Reasoning models ignore it (`gr_model_info(m)$supports_temperature`)
`NOT_IN_DOCUMENT` for a question that assumes something the document contradicts	`ans$premise`, with `premise = TRUE`	without the check a reader can only answer or say the excerpts are silent; see “Questions that assume something false” above

Test for the marker with is_not_found(ans$answer) rather than grepl("NOT_IN_DOCUMENT", ...): a real answer can quote the marker, and models decorate it (**NOT_IN_DOCUMENT.**).

Two habits make all of this cheaper: run gr_chunk_stats() before you spend anything, and keep gr_options(max_cost_usd = ...) set to something you would not mind paying by accident.

vignette("segment"): cutting documents into chunks, and comparing chunkings for nothing before you read.
vignette("review"): reading many documents, and literature reviews with extract and screen.
vignette("runs"): limits, the trace, caching and replaying a run.
vignette("extending"): registering a reader of your own.
?gr_read_spec: every reading setting, with its default and allowed range.

Setting up

Listing the readers

Every reader on one document

The readers

stuff: everything in one request

map_reduce: every chunk separately, then combined

refine: a draft revised chunk by chunk

skim: quote first, then answer

hierarchical: summaries of summaries

retrieve: only the most relevant chunks

rerank: shortlist, then let the model judge

iterative: look, decide what is missing, look again

preview: plan, then read

ensemble: several readers, compared

extract and screen