Register a segmentation strategy

Axis 2 is a registry, so "where does meaning break in this document?" is a question you can answer for your own material rather than choosing from a fixed list. A registered segmenter is a first-class one: it appears in gr_segmenters(), can be named in a gr_recipe(), and goes through the same token-cap enforcement and reporting as the built-ins.

Usage

gr_register_segmenter(
  name,
  fn,
  description = "",
  cost = c("free", "embedding", "llm"),
  needs_client = FALSE
)

Arguments

name

Segmenter name, used in specs and recipes. Re-registering an existing name replaces it.

fn

Function of (doc, spec, client, trace) returning a gr_chunks. Build the return value with new_chunks(); gr_segment() rejects anything else. doc is a gr_document; spec carries max_tokens, overlap_tokens and min_tokens, which pack_units-style helpers respect for you.

description

One-line description, shown by gr_segmenters().

cost

"free", "embedding" or "llm": what one run spends, so a UI can warn before it is spent.

needs_client

Whether the segmenter requires a client. gr_segmenters() reports it, so a UI can check before offering the strategy. When TRUE and no client is supplied, gr_segment() calls fn and then warns with class "gr_segment_fallback", unless fn raised a warning of that class itself, so one fallback gives one warning. Raising your own is better, because it can name what fn fell back to: warning(warningCondition("No client; using 'paragraph'.", class = "gr_segment_fallback")). Record the downgrade in the returned method ("mine->paragraph") so it survives into gr_chunk_stats().

Value

Invisibly, name.

See also

new_chunks() to build the return value, gr_segmenters(), gr_segment(), gr_segment_spec(), gr_recipe()

Other segmentation functions: gr_chunk_stats(), gr_chunks, gr_segment(), gr_segment_spec(), gr_segmenters(), new_chunks()

Examples

# One chunk per bullet list. Build the result with the same helper the
# built-ins use, so the token cap and reporting still apply.
gr_register_segmenter("by_bullet", description = "one chunk per bullet",
  fn = function(doc, spec, client, trace) {
    units <- unlist(strsplit(doc$text, "\n(?=[-*])", perl = TRUE))
    new_chunks(units, "by_bullet", spec)
  })
subset(gr_segmenters(), name == "by_bullet")
#>        name cost needs_client          description
#> 1 by_bullet free        FALSE one chunk per bullet