This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
FAQ
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Scheduling Tasks Dynamically

Go to solution
Senthilnaga91
New Contributor III

‎06-23-2023 10:12 AM

Hello All,

I have a use case where I have to schedule my pipeline dynamically based on entries from an input table? What’s the best way to implement this? For Example: If I have a pipeline loadSalesTable, I have to look to my input scheduling table to decide what day and time to trigger the pipeline and accordingly I have to schedule it.

I don’t think we have any straightforward approach using Scheduled or Triggered task, looking for a work around. Thanks in advance.

0 Kudos
Reply
1 ACCEPTED SOLUTION

Go to solution
Abhishek_Soni37
Contributor
In response to Senthilnaga91

‎06-26-2023 11:34 PM

@Senthilnaga91 ,

As @bojanvelevski mentioned you can use the Metadata snap pack to achieve it,

image

Solution Approach:

  1. Create all tasks that you want to run dynamically as Scheduled.
  2. Read data from the source table with info on when to run a pipeline.
  3. Read the specific Task with the help of the Metadata snap pack.
  4. Assign new values and update the Task with the Metadata snap pack.

Set values below in the pipeline parameter:

source_proj: orgName/ProjectPath
target_proj: orgName/ProjectPath
taskToUpdate: Name of the task

Sample pipeline:
UpdateTaskInfo.slp (3.7 KB)

Let us know if this worked.

Cheers 🙂

View solution in original post

Reply
5 REPLIES 5

Go to solution
Senthilnaga91
New Contributor III
In response to PratapKr_Gayen

‎06-26-2023 07:59 AM

No, let me be more precise. Our client has given a schedule to run what pipeline on which day and time (nnot a fixed schedule). It’s maintained in a RDBMS table. So, to trigger a pipeline, the process has to look into the scheduling table and trigger the pipleline on a particular day and time accordingly.

Example: There is a record in scheduling table,
PipeName,Date,Time
LoadSalesTable,06-27-2023,05:00 PM CST
LoadEmployeeTable,06-27-2023,04:00 PM CST

So, my question is how to automate the process to trigger the pipeline ‘LoadSalesTable’ on 06-27-2023,05:00 PM CST ?

(PS: The schedule for ‘LoadSalesTable’ might change in the future and the process should accommodate without manual intervention).

0 Kudos
Reply