Forum Discussion

arvindnsn's avatar
arvindnsn
Contributor
7 years ago

Triggered Task using cURL

Hello,

I am calling a Triggered Task (which does a ETL process) using a cURL command from a Shell script. Based on the output from the shell script, I would trigger the next process or exit with failure.

I am using the following in the shell script.

curl -H ‘Authorization: Bearer xxxx’ ‘http://xxxxx/xxx/xxx’ -v > $LOG 2>&1 <<END_SCRIPT

The shell script executes and exits in 3 to 5 seconds whether it is a success or a failure. If it is a success, the Log file has a HTTP Status = 200 and if it is failure, the HTTP Status != 200.

If it is a success, the shell script exists in 4 seconds but the ETL Insert is actually taking 3 mins to finish even after the shell script exits with success and since the shell script of task1 exits successfully, the task 2 is triggered thru another shell script which has a cURL command.

My issue is the dependent task should only start after the completion of the first task(only after all the records are inserted into the Oracle table ).

Below is the extract of the log file masking the host.

  • About to connect() to “xxx.com”
  • Trying xx.xx.x.xxx…
  • Connected to xxxx.com (xx.xx.x.xxx) port x(#0)

GET /api/1/xxx/xxx/xxxxxxxxxxxxxxxxxxxxx?xxx=2018
HTTP/1.1
User-Agent: curl/7.29.0
Host: host.host.com
Accept: /
Authorization: Bearer xxxxxxxxxxxx

< HTTP/1.1 200 OK
< Date: Tue, 23 Oct 2018 14:15:29 GMT
< Content-Length: 0
< Server: Jetty(9.4.0.v20161208)
<

2 Replies

  • tlikarish's avatar
    tlikarish
    Employee
    7 years ago
    Copy LinkReport

    Are there any unterminated outputs (see the section Synchronous and Asynchronous Execution of Triggered Tasks) in the pipeline that the task executes? That could cause it to return the response before the pipeline finishes executing.

    Another thought might be to move the flow control into a new pipeline. If your ETL process has two parts, you could use the pipe-exec snap to call the first and then the second. Something like

    Then point your task at that pipeline.

    • arvindnsn's avatar
      arvindnsn
      Contributor
      7 years ago
      Copy LinkReport

      Having One unlinked Output made the difference. If i have one unlinked output, the cURL command waited for the process to complete and return a output message and exit from the shell script.

      Thanks and much appreciated for providing a solution.