SPAWN TASK

The SPAWN TASK statement is used to define a task that will be executed in the background. This allows user to run periodic tasks, such as creating embeddings for a pdf file.

Syntax

SPAWN TASK task_name
BEGIN
    ....
END 
[EVERY duration unit]
[WITH MAX_POOL_SIZE max_pool_size];

Options

Option	Description	Possible Values	Mandatory
task_name	The name of the task to be created.	String	Yes
sql_operation	The SQL operation that will be executed when the task is triggered.	Valid SQL operation	Yes
EVERY duration unit	Specifies the frequency of the task execution. If omitted, the task runs once.	duration: Integer unit: second, minute, hour, day, week, month, year	Optional
max_pool_size	The maximum number of concurrent executions of the task. Defaults to 1 if not specified.	Integer	Optional

Example

SPAWN TASK generate_embeddings
BEGIN
    INSERT INTO pdf_embeddings
    SELECT p.id, embed(content) from pdfs as p LEFT JOIN pdf_embeddings as pe on p.id = pe.id
    WHERE p.id != pe.id
    ORDER by p.id
    LIMIT 10
END 
EVERY 5 second
WITH MAX_POOL_SIZE 5;

Spawned Task: generate_embeddings with id: ab41c508-86ce-4e1d-aa24-64474b7dc8eb

Task Logs

Once you successfully create a task, you can check the tasks and their status in langdb.tasks and langdb.task_logs

SELECT * FROM langdb.tasks;

id	name	sql	schedule_duration	schedule_unit
e711a5a8-83bc-4d5f-a462-f3b29cc1bbb1	generate_embeddings	INSERT INTO embeddings_fallback_data SELECT p.id, embed(question), p.question FROM popqa AS p LEFT JOIN embeddings_fallback_data AS	5	SECOND
		pe ON p.id = pe.id WHERE p.id <> pe.id ORDER BY p.id LIMIT 10

SELECT * FROM langdb.task_logs;

id	task_id	success	log	timestamp
68992881-da34-4703-b2c4-7eb93888ae1f	e711a5a8-83bc-4d5f-a462-f3b29cc1bbb1	true		2024-05-22 06:51:53
fab504c5-ddf8-4c60-a996-9aadec64b4dc	e711a5a8-83bc-4d5f-a462-f3b29cc1bbb1	true		2024-05-22 06:51:38
10b82a1b-6c09-46ca-b8c6-f1b640ab24ee	e711a5a8-83bc-4d5f-a462-f3b29cc1bbb1	true		2024-05-22 06:51:38
3c70fb8a-315f-4525-b49f-79b3836363a7	e711a5a8-83bc-4d5f-a462-f3b29cc1bbb1	true		2024-05-22 06:51:40
24ff108d-fe73-43ed-aba3-a7dbaad80647	e711a5a8-83bc-4d5f-a462-f3b29cc1bbb1	true		2024-05-22 06:51:45
05fd4c67-66fe-49a9-b94c-bb6bd5306b59	e711a5a8-83bc-4d5f-a462-f3b29cc1bbb1	true		2024-05-22 06:51:50