cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Trigger Task from Pipeline

alex_panganiban
Contributor

Is there a way that I can trigger a task from a pipeline?

I currently have a pipeline that passes a file path to a Pipeline Execute snap. There might over 2500 file paths that I need to pass into the Pipeline Execute, which will clearly be over and above the 64 depth limitation of the snap. I hesitate to send use the batch size feature of the Pipeline Execute because each file can contain over 1 million records to process. My gut tells me that this amount of throughput through a single instance of a pipeline would be too much.

Rather than using the Pipeline Execute, Iโ€™d like to pass my file paths to a task, essentially turning my child pipeline into a parent pipeline and enable me to get around the 64 depth limitation.

Is this possible?

4 REPLIES 4

dmiller
Admin Admin
Admin

You can call the task URL: Invoking a SL pipeline with a REST call


Diane Miller
Community Manager

alchemiz
Contributor III

Hi Kuya,

Good day, I had a similar requirement before and I took the path in using the ultra task high availability api

Snaplex must have at least one node setup as a feed-master

  • Design an ultra pipeline that will serve as a wrapper to the child pipeline
  • Create ultra task for the pipeline wrapper

** Requirement was to send a JSON payload, the pipeline wrapper will be accepting JSON content-type by invoking REST Post from the pipeline runner

With ultra task, this pipeline can be upscale depending from your requirement (ultra task instance) with high availability for request to be served (max in flight)

Attached PoC pipelines

Runner - main pipeline that will invoke the ultra task pipeline
Runner_2022_09_04.slp (26.1 KB)

Do Something - the child pipeline will be executed
Do Something_2022_09_04.slp (5.3 KB)

open_api_save_random_image - the ultra pipeline (can also be the child pipeline)
open_api_save_random_image_2022_09_04.slp (22.5 KB)

Ultra task instance was set to 3
image

Hope this helps

Thanks,
EmEm

alex_panganiban
Contributor

Hi EmEm, so happy to connect with you again. Hope youโ€™re doing well. Salamat kuya for the nice wrapper solution.

My plan next week is to discontinue using my taskโ€™s CloudURL and begin using the SecuredURL that points to our groundplex snaplex. I believe by doing this, I can resolve the issue of being limited to 10 concurrently running tasks. Based on what Iโ€™ve been reading, I shouldnโ€™t have concurrency limitations when Iโ€™m running on the groundplex. Do you have any experience using this method?

If this method works, that will be great. Iโ€™d like to see if this works before introducing ultra pipeline tasks. Weโ€™re not using those right now so Iโ€™d probably have to present a really good use case for it and (maybe?) the added expense that comes along with it.

Iโ€™ll let you know how it goes. Btw, I like your description label. MVP Rock Star!. Thatโ€™s so cool!

Only when invoking the cloud url of the triggered task has this limitations

image

No limits when invoking the on-premise url