Skip to contents

Count tokens in text

Source: R/core-tokenize.R
gr_count_tokens.Rd

Vectorised over text. Always returns a non-negative integer vector of the same length; blank and NA entries count as 0.

Usage

gr_count_tokens(text, model = NULL)

Arguments

text

Character vector.

model

Optional model id; used only by tokenizers that are encoding-specific (e.g. "tiktoken").

Value

Integer vector of token counts: a conservative upper bound under the default tokenizer.

Details

The default "heuristic" tokenizer is a deliberate over-estimate, not an exact count: it classifies each word, sums the per-script contributions, and adds a small per-message framing allowance. Overcounting wastes a little context; undercounting produces a hard API failure after you have paid for the request. For exact counts install reticulate plus Python tiktoken and call gr_set_tokenizer("tiktoken").

See also

gr_set_tokenizer(), gr_truncate_tokens(), gr_budget()

Other cost and token functions: gr_budget(), gr_estimate_cost(), gr_model_info(), gr_model_limits(), gr_models(), gr_register_model(), gr_set_tokenizer(), gr_tokenizer(), gr_truncate_tokens()

Examples

gr_count_tokens(c("the quick brown fox", "", "a much longer sentence than that one"))
#> [1]  8  0 12

# Compare tokenizers on the same text.
old <- gr_tokenizer()
vapply(c("heuristic", "words", "chars"), function(t) {
  gr_set_tokenizer(t); gr_count_tokens("the quick brown fox jumps")
}, integer(1))
#> heuristic     words     chars 
#>         9         5         7 
gr_set_tokenizer(old)