Register or inspect the active tokenizer

Register or inspect the active tokenizer

Usage

gr_set_tokenizer(name, fn = NULL)

Arguments

name

One of the built-in tokenizers ("heuristic", "words", "chars", "tiktoken") or a name previously registered with a custom function.

fn

Optional. A function taking a character vector and returning an integer vector of token counts. Supplying it registers a custom tokenizer under name.

Value

Invisibly, the name of the newly active tokenizer.

See also

gr_tokenizer(), gr_count_tokens(), gr_budget()

Other cost and token functions: gr_budget(), gr_count_tokens(), gr_estimate_cost(), gr_model_info(), gr_model_limits(), gr_models(), gr_register_model(), gr_tokenizer(), gr_truncate_tokens()

Examples

old <- gr_tokenizer()
gr_set_tokenizer("heuristic")
gr_count_tokens("the quick brown fox")
#> [1] 8

# Register your own; budgets recompute against it immediately.
gr_set_tokenizer("naive_words",
                 function(x) lengths(strsplit(trimws(x), "\\s+")))
gr_count_tokens("the quick brown fox")
#> [1] 4
gr_set_tokenizer(old)