Vectorised over text. Always returns a non-negative integer vector of the
same length; blank and NA entries count as 0.
Value
Integer vector of token counts: under the default tokenizer, an estimate built to err high (see above for the one kind of text where it can fall short).
Details
The default "heuristic" tokenizer is a deliberate over-estimate, not an
exact count: it classifies each word, sums the per-script contributions, and
adds a small per-message framing allowance. For numbers, tables, code and
short lines it also counts the pieces a BPE tokenizer splits text into first
(digit groups, punctuation runs, line breaks, indentation) and keeps the
larger figure, line by line. Overcounting wastes a little context;
undercounting produces a hard API failure after you have paid for the
request. Rare long words (technical and medical vocabulary) can still come in
under the real count, by about a tenth for a table of medical terms; that is
what the safety_margin of gr_budget() is for.
For counts from OpenAI's own tokenizer install reticulate plus Python
tiktoken and call gr_set_tokenizer("tiktoken"). Given model, it counts
in that model's encoding. Without one, which is how the package's own budgets
call it, it counts in both encodings current OpenAI models use (cl100k_base
and o200k_base) and keeps the larger, so the count holds for any of them.
Models from other providers tokenize differently, and for them it is an
estimate without the heuristic's padding. Text that spells a special token
(<|endoftext|>, in a paper about language models) is counted as the plain
text it is.
See also
gr_set_tokenizer(), gr_truncate_tokens(), gr_budget()
Other cost and token functions:
gr_budget(),
gr_estimate_cost(),
gr_model_info(),
gr_model_limits(),
gr_models(),
gr_register_model(),
gr_set_tokenizer(),
gr_tokenizer(),
gr_truncate_tokens()
Examples
gr_count_tokens(c("the quick brown fox", "", "a much longer sentence than that one"))
#> [1] 8 0 12
# Compare tokenizers on the same text.
old <- gr_tokenizer()
vapply(c("heuristic", "words", "chars"), function(t) {
gr_set_tokenizer(t); gr_count_tokens("the quick brown fox jumps")
}, integer(1))
#> heuristic words chars
#> 9 5 7
gr_set_tokenizer(old)