A client that answers from a recorded run

Replays the responses in a trace instead of calling a model. Give it the trace from a run and the same document and question, and you get that run back (same answers, same evidence, same merge decisions) with no API key, no network and no spend.

Usage

gr_replay_client(source, strict = TRUE, on_exhausted = c("error", "flag"))

Arguments

source: A gr_trace; a path to a file written by gr_trace_save(); the JSON text as_json() writes for a trace or for an answer (whose trace is used); or an already-parsed list in that shape. Parse a file yourself with jsonlite::fromJSON(path, simplifyVector = FALSE): the default simplification turns the steps into a data frame, which loses what a replay needs, and is refused with a message that says so.
strict: If TRUE (default), a prompt with no recorded response raises a gr_replay_miss error. That is usually what you want: a miss means the replay has diverged from the recording, and continuing would produce a result that looks like the original but is not. With FALSE a miss returns a failed gr_result instead, so a partially recorded trace still runs.
on_exhausted: What a call does when the recording holds fewer replies to its request than the replay has asked for. "error" (the default) raises an hr_replay_exhausted error: the replay is making more calls than the run it reproduces did. "flag" returns the last recorded reply again and counts it in $stats()$repeats, which is what every replay did before this argument existed.

Value

An object of class gr_replay_client, usable anywhere a gr_client() is. It also carries $stats() and $missed().

Details

This is what makes a published result checkable. Ship the trace next to the paper and a reader can reproduce the run rather than take it on trust. It is also the cheapest possible bug report: a trace file is a re-runnable recording of exactly what went wrong.

Matching

A response is matched on the exact prompt messages plus the model id the call asked for, the reasoning effort it was sent with when it had one (see effort in gr_read_spec()), the temperature it was made with once the session default was filled in (none for a model that takes none), the output cap it asked for, and its JSON schema. These make up the key harnessr-replay-v2, which each step of a trace records as replay_key. So a replay under another effort, temperature, cap or schema misses rather than passing for the recording. A replay under the same effort matches even where this session's registry does not list a model as a reasoning model and the recording session's did (a gateway model registered with gr_register_model() there and not here), and a call through an ellmer chat, which sends no effort, is matched on the one it asked for. The model id can differ from the model the trace records as answering: a gr_ellmer_client() answers with its chat's model whatever the recipe asked for, and its runs replay all the same.

A step recorded before replay_key existed (readgpt 0.6.0, and harnessr before this key) is matched on the old key, readgpt-replay-v1: the messages, the model and the effort only. A call that finds nothing under the new key is looked up there, and a reply found that way is used, with an hr_legacy_key warning once per recording in a session, since the replay cannot then show that the temperature, cap and schema were the recorded ones. $stats()$legacy_hits counts these replies.

The replay client's own model, which a read that names none asks for, is the model the recorded reads asked for. One trace can hold reads through clients built for different models, and a recording made by readgpt 0.5.0 asked for gr_options("model") on a read and for the client's model everywhere else; a call that asks for the replay client's model and finds nothing under it is answered from the one other such model that holds the same prompt. A model the recorded settings named (model, skim_model, summary_model) is never used that way, so a replay that leaves one out misses.

Repeated calls and draws

When a run issued the same request more than once (which happens at a temperature above zero, and in readers that revisit a chunk), the recorded responses are returned in the order they were produced. A call with a draw_index above 0 (see gr_call()) is answered with the reply recorded for that draw; in a recording that holds no draw indexes, draw k is the k-th reply recorded for the request. A call past the end of what was recorded, or for a draw that was never recorded, raises hr_replay_exhausted unless on_exhausted = "flag".

Embeddings

A trace records each request to an embeddings endpoint (a step labelled "embed.request", counted in calls and priced like any other request), but not the vectors that came back, so a replay has nothing to answer those requests with. A run that embedded (the semantic segmenter, the retrieve and iterative readers, rerank when word overlap finds nothing, and so the "needle" recipe) therefore replays only when the recording used a deterministic embedder and the replay uses the same one: the vectors are then computed again, exactly. Both conditions, because replaying an API-embedded run with a deterministic local embedder would compute vectors the original never saw while looking exact. Anything else, including a run recorded with the default "api" embedder (or a gr_mock_client()'s embed handler), warns with class gr_replay_no_embeddings and falls back to hashed lexical vectors. Those place semantic cuts and rank chunks differently, so such a replay usually sends prompts the recording does not hold: a strict replay stops with gr_replay_miss, whose message names this cause, and a non-strict one gets failed calls for them and does not give the recorded answer. Record a run you intend to publish or replay with gr_options(embedder = "lexical"), or with your own embedder registered as deterministic = TRUE.

Limits

A saved trace says whether a limit cut the run short (budget_stop, stop_reason), and a replay of such a run stops where it stopped when it runs under the limits it was recorded with. Replayed calls count against max_calls as they did when recorded. When the spending limit stopped the recorded run, each replayed call also counts what it cost when recorded against max_cost_usd (in the trace's replayed_usd; nothing is spent); for any other run it counts nothing, so a replay of an expensive run is not stopped by the replaying session's limit. A replay under a higher limit, or none, that asks for a call past the recorded stop gets a gr_replay_miss saying so, and one under a lower limit stops sooner, marked partial as the run would have been.

The recipe "auto" chose

A recording of answer_document() with recipe = "auto" holds the recipe the choice picked for each document and question, and a replay of the same document and question repeats that choice rather than making it again. What the choice rests on (the token count, the registered models, the client's model) can differ between the session that recorded a run and the one replaying it, and a different choice would send prompts the recording does not have.

What does not replay

Traces do not store the JSON schema a call requested, only a hash of it in the step's key. In a recording made before replay_key existed, two calls that differ only by schema, temperature or output cap share a reply.

Examples

# A run.
cl <- gr_mock_client(function(m, p) "Revenue was 45.2 million dollars.")
ans <- answer_document(gr_example(), "What was revenue?", "fast", client = cl)
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).

# The same run, from the recording, with no client and no key.
rp <- gr_replay_client(ans$trace)
again <- answer_document(gr_example(), "What was revenue?", "fast", client = rp)
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
identical(again$answer, ans$answer)
#> [1] TRUE

rp$stats()
#>   recorded distinct hits repeats misses legacy_hits
#> 1        1        1    1       0      0           0

# Through a file, which is how a run reaches someone else.
f <- tempfile(fileext = ".json")
gr_trace_save(ans$trace, f)
answer_document(gr_example(), "What was revenue?", "fast",
                client = gr_replay_client(f))$answer
#> Using cached ingestion for this document + settings.
#> Segmenting with 'paragraph' (cap 4000 tokens, overlap 0).
#> Reading with 'stuff' (all|1|none) over 1 chunk(s).
#> [1] "Revenue was 45.2 million dollars."