Skip to content

DocumentSplitter

pixeltable.iterators.DocumentSplitter 

DocumentSplitter(
    document: str,
    *,
    separators: str,
    limit: Optional[int] = None,
    overlap: Optional[int] = None,
    metadata: str = "",
    html_skip_tags: Optional[list[str]] = None,
    tiktoken_encoding: Optional[str] = "cl100k_base",
    tiktoken_target_model: Optional[str] = None
)

Iterator over chunks of a document. The document is chunked according to the specified separators.

The iterator yields a text field containing the text of the chunk, and it may also include additional metadata fields if specified in the metadata parameter, as explained below.

Chunked text will be cleaned with ftfy.fix_text to fix up common problems with unicode sequences.

Parameters:

  • separators (str) –

    separators to use to chunk the document. Options are: 'heading', 'paragraph', 'sentence', 'token_limit', 'char_limit', 'page'. This may be a comma-separated string, e.g., 'heading,token_limit'.

  • limit (Optional[int], default: None ) –

    the maximum number of tokens or characters in each chunk, if 'token_limit' or 'char_limit' is specified.

  • metadata (str, default: '' ) –

    additional metadata fields to include in the output. Options are: 'title', 'heading' (HTML and Markdown), 'sourceline' (HTML), 'page' (PDF), 'bounding_box' (PDF). The input may be a comma-separated string, e.g., 'title,heading,sourceline'.