Truncate text to at most n tokens

Truncation happens at a word boundary where the text has whitespace, and at a character boundary where it does not (base64, a data URI, a CJK run, a minified line); otherwise the cap would be unenforceable for exactly the inputs that most need it. Returns "" for empty input and never returns NA.

Usage

gr_truncate_tokens(text, n, marker = " ...[truncated]")

Arguments

text

A single string.

n

Maximum token count.

marker

Appended when truncation occurred; set "" to suppress.

Value

A single string. "" when n <= 0 or the input is blank. This is not treated as an error.

See also

gr_count_tokens()

Other cost and token functions: gr_budget(), gr_count_tokens(), gr_estimate_cost(), gr_model_info(), gr_model_limits(), gr_models(), gr_register_model(), gr_set_tokenizer(), gr_tokenizer()

Examples

gr_truncate_tokens(paste(rep("alpha beta gamma", 40), collapse = " "), 20)
#> [1] "alpha beta gamma alpha beta gamma alpha beta ...[truncated]"
gr_truncate_tokens("short enough already", 100)
#> [1] "short enough already"