Segment a document into chunks

Segmentation is deliberately separate from ingestion: one extraction can feed many different segmentations, so comparing chunking strategies costs one file read, not one per strategy.

Usage

gr_segment(doc, spec = NULL, client = NULL, trace = NULL)

Arguments

doc

A gr_document from gr_ingest(), or anything gr_ingest() accepts.

spec

A gr_segment_spec, a segmenter name, or a named list.

client

A gr_client, needed only by semantic, proposition and contextual(context_source = "llm").

trace

Optional gr_trace. When none is given one is made, so the model calls a segmenter makes are held to gr_options(max_calls =, max_cost_usd =) and recorded, and it is returned as $trace.

Value

A gr_chunks object. If the requested segmenter could not run it falls back and records the fallback in $method, e.g. "semantic->paragraph" when no client was supplied. proposition and contextual chunks carry a source_text column: the document text each chunk was made from, without the rewrite or the context line, which is what a quote from the chunk is checked against. When no trace was passed, $trace is the one this call recorded into. page keeps blocks that have no page number, in a document whose other blocks have one, as chunks of their own with page NA, and warns.

See also

gr_segmenters(), gr_segment_spec(), gr_chunk_stats(), gr_chunks

Other segmentation functions: gr_check_segmenter(), gr_chunk_stats(), gr_chunks, gr_register_segmenter(), gr_segment_spec(), gr_segmenters(), new_chunks()

Examples

doc <- gr_ingest(gr_example())
#> Using cached ingestion for this document + settings.

# The same document, three boundary hypotheses. No API calls.
do.call(rbind, lapply(c("fixed", "paragraph", "sentence", "structural"),
  function(m) gr_chunk_stats(gr_segment(doc, list(method = m, max_tokens = 120)))))
#> Segmenting with 'fixed' (cap 120 tokens, overlap 0).
#> Segmenting with 'paragraph' (cap 120 tokens, overlap 0).
#> Segmenting with 'sentence' (cap 120 tokens, overlap 0).
#> Segmenting with 'structural' (cap 120 tokens, overlap 0).
#>       method n total_tokens min median  mean max over_cap
#> 1      fixed 5          528  49  120.0 105.6 120        0
#> 2  paragraph 6          539  48   92.0  89.8 116        0
#> 3   sentence 6          534  47   92.5  89.0 107        0
#> 4 structural 8          557  16   76.5  69.6 102        0