Exported because it is part of the extension API: gr_segment() rejects
anything that does not inherit "gr_chunks", so a custom segmenter
registered with gr_register_segmenter() cannot be written without this.
Using it also gets you the shared invariants for free (blank units dropped,
chunk_id assigned in order, tokens and characters measured, provenance
recycled to match), so your segmenter behaves like the built-ins wherever
the rest of the package touches it.
Usage
new_chunks(
text,
method,
spec,
page = NA_integer_,
section = NA_character_,
block_id = NA_integer_,
extra = list()
)Arguments
- text
Character vector of chunk texts. Blank entries are dropped.
- method
Your segmenter's name. Record a fallback here if you took one (
"semantic->paragraph");gr_chunk_stats()surfaces it.- spec
The
gr_segment_specpassed to your segmenter.- page, section, block_id
Provenance, one value per chunk or one recycled value. Leave as
NArather than guessing: wrong provenance sends a reader to the wrong page with full confidence.- extra
Named list of segmenter-specific detail, kept on
$extra.
Value
A gr_chunks.
Details
The token cap is NOT enforced here. gr_segment() checks it after your
function returns and re-splits anything oversized, so a segmenter that
ignores spec$max_tokens produces a warning and correct chunks rather than
an HTTP 400.
See also
gr_register_segmenter(), gr_chunks, gr_segment(),
gr_chunk_stats(), new_answer()
Other segmentation functions:
gr_chunk_stats(),
gr_chunks,
gr_register_segmenter(),
gr_segment(),
gr_segment_spec(),
gr_segmenters()
Examples
# One chunk per bullet, with the source block recorded.
doc <- gr_ingest("Findings:\n\n- Revenue rose.\n- Costs fell.\n- Margin widened.")
#> Ingested 2 block(s), ~21 tokens (0 chars removed by cleaning).
ch <- new_chunks(trimws(strsplit(doc$text, "\n(?=-)", perl = TRUE)[[1]]),
method = "by_bullet", spec = gr_segment_spec(max_tokens = 100),
block_id = 1L)
gr_chunk_stats(ch)
#> method n total_tokens min median mean max over_cap
#> 1 by_bullet 4 28 6 7 7 8 0