WithEnableLineNumber enables line number prefixing for chunk content. When enabled, each line in the chunk will be prefixed with line numbers. This op...
WithLines sets the line trigger for chunking. When set to a positive value, chunks will be created every N lines. If the N lines content is smaller t...
WithSeparatorAsBoundary switches the separator semantics from "trigger every occurrence" (default) to "use the separator as a preferred cu...
WithEnableLineNumber enables line number prefixing for chunk content.
When enabled, each line in the chunk will be prefixed with line numbers.
This option works with all chunking modes and respects ChunkSize limits.
WithLines sets the line trigger for chunking. When set to a positive value,
chunks will be created every N lines. If the N lines content is smaller than
ChunkSize, it will be kept intact. If larger than ChunkSize, it will be split
according to ChunkSize (ChunkSize is a hard limit).
Example:
aireducer.NewReducerFromFile("file.txt", aireducer.WithLines(10), aireducer.WithChunkSize(1024)) // This will create chunks every 10 lines, but if 10 lines exceed 1024 bytes, // they will be split at 1024 byte boundaries.
WithSeparatorAsBoundary switches the separator semantics from "trigger every
occurrence" (default) to "use the separator as a preferred cut boundary".
When true, the reducer fills up to ChunkSize and, inside each chunk, cuts at
the LAST separator occurrence in the window so that pre-structured blocks
stay intact. Combine with WithSeparatorTrigger(sep) + WithChunkSize(n).