Scheduling Tasks Dynamically

Question

Hello All,
I have a use case where I have to schedule my pipeline dynamically based on entries from an input table? What’s the best way to implement this? For Example: If I have a pipeline loadSalesTable, I have to look to my input scheduling table to decide what day and time to trigger the pipeline and accordingly I have to schedule it.
I don’t think we have any straightforward approach using Scheduled or Triggered task, looking for a work around. Thanks in advance.

bojanvelevski · Answer

Hi there @Senthilnaga91.
The Snaplogic Snaps will be suitable for this use case. You have all  CRUD operations at your disposal. Let me know if you still need help on this.
Regards,
Bojan

pratapkr_gayen · Answer

I think your requirement is to trigger the a pipeline when ever there is a change in record in source table.
For that you have to create pipeline to save the current count  in SLDB and compare the count of record in current execution. This pipeline to compare and check the record count  mismatch need to be schedule 15 or 30 min and if there is count mismatch, call the pipeline to load data to Sales table.

senthilnaga91 · Answer

No, let me be more precise. Our client has given a schedule to run what pipeline on which day and time (nnot a fixed schedule). It’s maintained in a RDBMS table. So, to trigger a pipeline, the process has to look into the scheduling table and trigger the pipleline on a particular day and time accordingly.
Example: There is a record in scheduling table,
PipeName,Date,Time
LoadSalesTable,06-27-2023,05:00 PM CST
LoadEmployeeTable,06-27-2023,04:00 PM CST
So, my question is how to automate the process to trigger the pipeline ‘LoadSalesTable’ on 06-27-2023,05:00 PM CST ?
(PS: The schedule for ‘LoadSalesTable’ might change in the future and the process should accommodate without manual intervention).

senthilnaga91 · Answer

I have provided more details in my previous reply. Please refer.

Forum Discussion

Scheduling Tasks Dynamically

5 Replies

Recent Discussions

Javascript to promote top level lists

Google Sheets Subscribe questions

Basic string transformations not working

Can we generate XML file in pretty print format using native snapLogic snaps?

Multipart Reader failure - 'content-type' was not found