cancel
Showing results for 
Search instead for 
Did you mean: 

Trigger pipeline based on table updates

Roger667
New Contributor III

HI all,
I have a sql server upstream data source, there is a table in that which gets updated when the data is refreshed. I based on the updates in that particular table and colum i want the ingestion pipeline to be triggered. Is there any way i can implement it. Does this requires constant running of the pipeline to monitor the table?
Thanks!

1 REPLY 1

koryknick
Employee
Employee

@Roger667 - you can execute a pipeline via REST API through a triggered task.  You can create the triggered task for the pipeline in Designer using the Create Task button in Designer:

koryknick_0-1706705057286.png

Once created, you can see the URL to call to execute the pipeline by going into Manager for your project and click the dropdown next to the task name:

koryknick_1-1706705145329.png

You can use the Cloud URL if you need to execute the task outside your firewall or can use the Secured URL if within the firewall:

koryknick_2-1706705248149.png

To authorize the execution, you can either use basic auth (username/password) or the bearer token listed.

Once that is created, you will need to find the steps necessary on the SQL Server side to call the triggered task URL.

However, you may want to consider a different approach since calling a pipeline may take several seconds to complete depending on pipeline instantiation timing and the execution time of your pipeline.  If the table you put this type of trigger into has high volumes, it could cause serious performance issues in your table updates.

Hope this helps!