Introduction
First, i’d like to give you some definitions:
xTask: the tasks as defined in Xano
a task: an event that is triggered at a certain date & time to execute a process
a job: a process to be executed by a task
The actual way to manage tasks in Xano is pretty limited:
xTasks aren’t executed in parallel. if you set a task to be recurrent, If one xTask isn’t finished before it is supposed to fire again, then it’s not fired again.
A xTask will always do the same thing. Hence, you need to create as many xTasks as exist different jobs to get done. Depending on the plan you’re on, you can be quickly limited.
Some xTasks need to be fired on a minute frequency, some others on an hourly or daily basis. In Xano, you have to set a frequency per xTask. So you have to create a xTasks based on its frequency.
if you execute several jobs in one xTask but one is longer than expected, the required frequency for the others is not met and they become late.
As you can see, there are some limitations on xTasks.
That’s why in this article, i propose we will build a robust scheduler to allow you to organize and schedule the execution of various tasks, even programming recurring tasks, whatever the task duration is, without blocking other tasks to be executed.
What is a scheduler?
The goal of a scheduler is to arrange, control, and manage the execution of tasks. These tasks can be one-time actions or recurring jobs. A scheduler might offer a range of functionalities and allow a wide variety of actions and processes.
Its main functionalities are:
create and schedule a task
execute the task when planned
log task execution result
Designing the Scheduler
Global design
Here’s, at an abstract level, what the Scheduler does:
You create a task that is to execute a job
You schedule the task
When the time has come, you execute the job the task is supposed to do
you log the result of the task
Requirements
Some requirements about the Scheduler:
if a task (which can trigger several jobs) takes longer than its trigger frequency, it must not, as much as possible, prevent other tasks to be triggered
if a task is suspended, it mustn’t prevent other tasks to be triggered
Fine-tuned design
So, based on xTasks limitations and our requirements, we’ll have to create a Scheduling system that allows tasks to be executed in parallel when they are based on the same frequency, and if possible, prevent a maximum of tasks from getting stuck if one takes longer.
So first, we’ll need several “clock” engines to be able to run in parallel and trigger tasks.
Second, for a given frequency, we’ll need to balance tasks between these clocks. If one task takes longer, the other “clock” engines are able to run other tasks.
Then to prevent a task from being surely “late”, we’ll create “clock engines” based on several frequencies.
Implementing the Scheduler
Diving into Task creation and scheduling
To schedule a job to run at a specific date, you require two essential components:
A repository for your task that includes both the timing instructions for when the job should be executed and the necessary information for executing the job itself.
A scheduling system that triggers the task execution at the designated time.
Task repository
The best Task repository, in the context of Xano, is a table where all the needed informations are stored.
Let’s create this table called Tasks👍
Scheduling system
The scheduling system will be made out of several components:
a function to create a new task
a function to execute the job associated to the task
a scheduling mechanism to check and execute the right tasks on right time
Function task_new
Let’s call it: task_new
it needs several parameters:
pJob (text) : the job that needs to be run when task is executed
pParameters (json): : the JSON that will contain the data needed by the job
pExecutionDate (timestamp) : the date & time to execute the task
pTimer (enum) : the timer that will execute the task
its purpose is to add an entry to the Tasks table to handover the task to the scheduler.
Step1: all the DB operations inside this function will be executed “at once” at the end of it.
Step1.1: inserting the new task in the DB with first set of values
step 1.2: determining the scheduler id to trigger the task.
If the timer is minute then we calculate the schedulerId according to the new task Id modulus 3 (result is 0,1 or 2). Else it is set to 0.
I’ll explain later, when dealing with the scheduling mechanisms, why this calculation specifically. It’s based on the incremental id from the table Tasks.
Step 1.3: setting the scheduler Id
Function task_runJob
Let’s call it: task_runJob
This function will trigger the according sub-function related to the job. As a parameter, it receives the Task stored in the Tasks table.
it’s algorithm is very simple:
Based on the Tasks.tsk_job label, it executes the according function, transmitting to this function the .tsk_parameters JSON field needed by this function to achieve its purpose.
This if-statement is to be reproduced as many times as you have different jobs to be executed, each corresponding to a .tsk_job.
Then it returns a result to the scheduler that will update the task’s status accordingly.
A few things to consider:
The result of this function, as the result coming back from all the job_* functions is structured almost like an API response
The status will be stored into the .tsk_status field and the result in the .tsk_result field.
The job_* functions if they are to be recurrent can call the task_new function to reschedule itself
Scheduling mechanism
Now, we have to implement the “clock engines” that will properly trigger each of these tasks, right on time.
For this, we’re going to use Xano’s Tasks.
But remember that a xTask can’t be triggered while the same previous one is not finished, no matter the frequency. And one of our requirements is to prevent, as much as possible, tasks from getting stuck if a previous one is not finished.
Here's how to ensure that the maximum number of tasks are triggered when they're supposed to be.
We’re going to create 5 xTasks:
3 to handle 1 minute timer
1 to handle 1 hour timer
1 to handle 3hrs timer
Minute xTask
Let’s create a new task and name it: cron_m00
And here’s its function stack:
It is scheduled to be triggered every minute.
Here’s the description of the algorithm, step by step:
Step1: Querying all the tasks which .tsk_status is pending, execution date & time is in the past,.tsk_timer = minute and .tsk_schedulerId = 0. Which means: all the task to be executed each minute, affected to scheduler Id 0, that are waiting to be triggered and which execution date & time is before now. This allows tasks that hadn’t been executed (when cron_m00 was disabled for maintenance for example). Note that if the task was suspended, it is not taken into account.
Step2: a loop on each task to set each task to the processing status
This is done in the case that one day, Xano authorizes a xTask to get triggered if previous one is not finished.
Step 3: a loop on each task to execute its job
Step 3.1: we create a variable that contains the current time
Step 3.2: we call the task_runJob with the current task information as its input parameter.
Step 3.3: once the job is done for this task, we calculate its duration
step 3.4: we edit the task with more informations about its execution and result
Now that we have created this xTask, we duplicate it twice and rename them, respectively cron_m01 and cron_m02.
The only modifications needed in these xTasks are on the first step where you modify the scheduler Id, respectively to 1 and 2.
WIth this plus the setting of a SchedulerId to each task based on Tasks.id modulus 3 allows to balance the tasks amongst 3 “clock engines”. That way, each minute, 3 tasks are executed: if one takes too much time.
For cron_m01:
For cron_m02:
Then you clone cron_m02 and name it cron_h00. The modifications to do in this new task are :
modify the Query All Criterias:
Change the timer frequency to Every Hour:
Duplicate one last time this cron_h00 and name it cron_3h00. The modifications to do in this new task are :
modify the Query All Criterias:
Change the timer frequency to Every 3 Hours:
Don't forget to enable each of your xTasks!
Using this Scheduler
Now, each time you want to schedule a specific task, either one-time task or a recurring task you have to:
Create a job_<jobName> function that will handle the job execution
You modify the task_runJob function to add the new .tsk_job to be handled (cf. the if-statement to be cloned as needed in the function)
You use the task_new function to add any new task with the good parameters:
pJob = a name that’ll be used to set a precise process to be executed
pTimer = minute for jobs that take a few seconds
pTimer = hour for jobs that take a few minutes
pTimer = 3h for jobs that take less than 3 hours
Here’s the final schema for the Scheduler system:
Please, leave a comment or any feedback, correction, improvements, ... :)