An ingested document

Returned by gr_ingest().

Fields

blocks

Data frame of cleaned text blocks with provenance: text, page, section, kind, block_id. page is set only by the PDF extractor; kind is one of "body", "heading", "code", "table", "footnote", "ocr".

text

All blocks joined with blank lines.

source

Absolute path, the web address a document was fetched from, or "<inline text>" when the input was a string of text.

spec

The gr_ingest_spec() used.

stats

blocks, chars, chars_removed, tokens, pages, clean_steps, clean_log (characters removed per cleaning step, useful when cleaning ate more than you expected) and unread_pages (pages that never became text, such as scanned pages read without OCR; an answer drawn from the document is marked partial when there are any).

warnings

Character. What readgpt warned about while extracting and cleaning, named by the warning's class. Kept with the document, so a copy served from the ingestion cache still carries them.

Methods

print() shows the source, block count and token total; as_json() serialises the stats and every block with its provenance.

See also

gr_ingest() which returns one, gr_ingest_spec(), gr_extractors(), gr_cleaners(), as_json(), gr_segment() for the next axis

gr_ingest(), gr_ingest_spec(), gr_cleaners()

Other ingest functions: gr_clean(), gr_cleaners(), gr_extractors(), gr_ingest(), gr_ingest_spec(), gr_register_cleaner(), gr_register_extractor()

Examples

doc <- gr_ingest(readgpt_example())
#> Using cached ingestion for this document + settings.
doc$stats$tokens
#> [1] 573
vapply(doc$stats$clean_log, function(s) s$chars_removed, integer(1))
#>        page_numbers         hyphenation       control_chars           ligatures 
#>                  12                   0                   0                  -1 
#> collapse_whitespace 
#>                   0