chunk()

The chunk() function splits a given text into chunks of a specified size. It provides options for customizing the chunking process and returns the resulting chunks as rows in a table-like format.

Syntax

SELECT * FROM chunk(raw_text, type => ChunkingType)

Parameters

Parameter	Type	Optional	Description	Possible Values	Sample Value
raw_text	String	No	The raw text which needs to be chunked	Any String	'In a quaint village...'
type	String	No	Unit of chunking	Char, Word, Sentence, Paragraph	'Char'
chunk_size	Int	Yes	Number of units to be present in a Chunk	Any non-negative integer	100
overlap	Int	Yes	Number of units to overlap between consecutive chunks	Any non-negative integer	20
trim	Bool	Yes	Whether to trim whitespace from the start and end of each chunk	true, false	true

Usage

Chunking Raw text into Char with Chunk Size

SELECT * FROM chunk('In a quaint village nestled in the heart of the countryside, there lived a young girl named Lily. She was known throughout the village for her vibrant imagination and her love for adventure. Every day, Lily would set out to explore the lush forests and rolling hills that surrounded her home, always eager to discover something new and exciting.

One particularly sunny morning, Lily decided to venture deeper into the woods than she ever had before. As she walked, she stumbled upon a hidden grove filled with the most beautiful flowers she had ever seen. The colors were so vivid and the petals so delicate that Lily couldnt help but marvel at their beauty. She spent hours in the grove, carefully examining each flower and breathing in their sweet fragrance.',
    type => 'Char',
    trim => true,
    chunk_size => 200);

text	index
In a quaint village nestled in the heart of the countryside, there lived a young girl named Lily. She was known throughout the village for her vibrant imagination and her love for adventure.	0
Every day, Lily would set out to explore the lush forests and rolling hills that surrounded her home, always eager to discover something new and exciting.	1
One particularly sunny morning, Lily decided to venture deeper into the woods than she ever had before.	2
As she walked, she stumbled upon a hidden grove filled with the most beautiful flowers she had ever seen.	3
The colors were so vivid and the petals so delicate that Lily couldnt help but marvel at their beauty.	4
She spent hours in the grove, carefully examining each flower and breathing in their sweet fragrance.	5

Chunking Raw text into Word with Chunk Size and Overlap

SELECT * FROM chunk('In a quaint village nestled in the heart of the countryside, there lived a young girl named Lily. She was known throughout the village for her vibrant imagination and her love for adventure. Every day, Lily would set out to explore the lush forests and rolling hills that surrounded her home, always eager to discover something new and exciting.

One particularly sunny morning, Lily decided to venture deeper into the woods than she ever had before. As she walked, she stumbled upon a hidden grove filled with the most beautiful flowers she had ever seen. The colors were so vivid and the petals so delicate that Lily couldnt help but marvel at their beauty. She spent hours in the grove, carefully examining each flower and breathing in their sweet fragrance.',
    type => 'Word',
    chunk_size => 30,
    overlap => 10);

text	index
In a quaint village nestled in the heart of the countryside there lived a young girl named Lily She was known throughout the village for her vibrant imagination and her	0
known throughout the village for her vibrant imagination and her love for adventure Every day Lily would set out to explore the lush forests and rolling hills that surrounded her	1
explore the lush forests and rolling hills that surrounded her home always eager to discover something new and exciting One particularly sunny morning Lily decided to venture deeper into the	2
particularly sunny morning Lily decided to venture deeper into the woods than she ever had before As she walked she stumbled upon a hidden grove filled with the most beautiful	3
stumbled upon a hidden grove filled with the most beautiful flowers she had ever seen The colors were so vivid and the petals so delicate that Lily couldnt help but	4
and the petals so delicate that Lily couldnt help but marvel at their beauty She spent hours in the grove carefully examining each flower and breathing in their sweet fragrance	5

Chunking Raw text into Sentence

SELECT * FROM chunk('In a quaint village nestled in the heart of the countryside, there lived a young girl named Lily. She was known throughout the village for her vibrant imagination and her love for adventure. Every day, Lily would set out to explore the lush forests and rolling hills that surrounded her home, always eager to discover something new and exciting.

One particularly sunny morning, Lily decided to venture deeper into the woods than she ever had before. As she walked, she stumbled upon a hidden grove filled with the most beautiful flowers she had ever seen. The colors were so vivid and the petals so delicate that Lily couldnt help but marvel at their beauty. She spent hours in the grove, carefully examining each flower and breathing in their sweet fragrance.',
    type => 'Sentence');

text	index
In a quaint village nestled in the heart of the countryside, there lived a young girl named Lily	0
She was known throughout the village for her vibrant imagination and her love for adventure	1
Every day, Lily would set out to explore the lush forests and rolling hills that surrounded her home, always eager to discover something new and exciting	2
One particularly sunny morning, Lily decided to venture deeper into the woods than she ever had before	3
As she walked, she stumbled upon a hidden grove filled with the most beautiful flowers she had ever seen	4
The colors were so vivid and the petals so delicate that Lily couldnt help but marvel at their beauty	5
She spent hours in the grove, carefully examining each flower and breathing in their sweet fragrance	6

Chunking Raw text into Paragraph

SELECT * FROM chunk('In a quaint village nestled in the heart of the countryside, there lived a young girl named Lily. She was known throughout the village for her vibrant imagination and her love for adventure. Every day, Lily would set out to explore the lush forests and rolling hills that surrounded her home, always eager to discover something new and exciting.

One particularly sunny morning, Lily decided to venture deeper into the woods than she ever had before. As she walked, she stumbled upon a hidden grove filled with the most beautiful flowers she had ever seen. The colors were so vivid and the petals so delicate that Lily couldnt help but marvel at their beauty. She spent hours in the grove, carefully examining each flower and breathing in their sweet fragrance.',
    type => 'Paragraph');

text	index
In a quaint village nestled in the heart of the countryside, there lived a young girl named Lily. She was known throughout the village for her vibrant imagination and her love for adventure. Every day, Lily would set out to explore the lush forests and rolling hills that surrounded her home, always eager to discover something new and exciting.	0
One particularly sunny morning, Lily decided to venture deeper into the woods than she ever had before. As she walked, she stumbled upon a hidden grove filled with the most beautiful flowers she had ever seen. The colors were so vivid and the petals so delicate that Lily couldnt help but marvel at their beauty. She spent hours in the grove, carefully examining each flower and breathing in their sweet fragrance.	1