gr_arm() wraps one recipe as an hr_arm(), so recipes can be compared
with hr_benchmark() on a reference of questions about documents, such as
one from gr_qa_reference(). gr_arms() builds one arm per recipe.
Arguments
- recipe
A recipe: a name from
gr_recipes(),"auto", a reader name, agr_recipe()or a named list ofingest,segmentandread.- settings
A named list of overrides applied to the recipe, as the
...ofanswer_document()takes them (for examplemodel,top_k,max_tokens). An unknown name is an error when the arm is built.- name
The arm's name. By default the recipe's name.
- recipes
A character vector of recipe names, or a list of recipes. The names of a named list or vector are used as the arms' names.
Value
gr_arm() returns an hr_arm(). gr_arms() returns a list of
them, named by arm.
Details
Each run reads the item's document and answers its question: with
answer_document() when the document is a file path, a web address or
text, or with gr_read() when it is already cut into chunks by
gr_segment(). It returns the gr_answer, which the benchmark's scorer
then judges; gr_scorer_answer() is the scorer made for it.
The run uses the benchmark's client and records every request into the
trace the benchmark made for that run (ctx$trace), so the run's tokens,
cost and refusals are counted and held to the benchmark's limits, and the
answer's trace is that run's trace. That trace also gives every request
of the run the run's draw index (see draw_index in gr_call()), segmenter
and reader requests included, so the runs of one arm on one item are
separate draws under a gr_cache() or a gr_replay_client() rather than
one stored reply repeated, which hr_benchmark() would refuse.
A reader checks before it starts whether the run's call and spending
limits leave room for it. When they do not, the run returns a
"NOT_IN_DOCUMENT" answer marked partial, with call_cap_reached or
cost_cap_reached in its notes as gr_compare() records them, and
gr_scorer_answer() scores it as a "budget_stop".
A recipe given as "auto" chooses "fast" or "thorough" per document,
as answer_document() does; it needs a document that is not yet chunked.
The arm's params are the recipe's name and the settings, and its version
is the version of harnessr, since readers and prompts change between
versions.
See also
gr_qa_reference(), gr_scorer_answer(), hr_benchmark(),
hr_report(), gr_compare() for a single descriptive comparison
Other harness:
gr_qa_reference(),
gr_scorer_answer(),
hr_arm(),
hr_benchmark(),
hr_contrast(),
hr_drift_check(),
hr_mdd(),
hr_reference(),
hr_report(),
hr_scorer()
Examples
cl <- gr_mock_client(function(m, p) "Revenue was 45.2 million dollars.")
ref <- gr_qa_reference(
questions = c(revenue = "What was revenue in 2024?",
ceo = "Who is the chief executive?"),
documents = gr_example(),
answers = list("45.2 million", NA),
unanswerable = "ceo", split = "test")
arms <- gr_arms(c("fast", "thorough"))
bench <- hr_benchmark(arms, ref, runs = 5, client = cl, scorer = gr_scorer_answer())
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 1200 tokens, overlap 120).
#> Reading with 'map_reduce' (all|N+logN|tree) over 1 chunk(s).
bench
#> <hr_bench> 2 arm(s) x 2 item(s) in 1 cluster(s) x 5 run(s); split test; scorer answer_contains
#> arm accuracy runs wrong failed budget_stop usd
#> fast 0.500 10 5 0 0 0.0000
#> thorough 0.500 10 5 0 0 0.0000
#> Descriptive only: accuracy is the mean over runs within each item, then over labelled items, with no interval.