Forum Discussion

pranjbhatt's avatar
pranjbhatt
New Contributor III
11 months ago
Solved

Execute each child pipeline multiple times

Hi, 

I have a parent pipeline that is receiving an array of client_code.  I have mapped and routed it to two child pipelines and would like to execute the each of the child pipeline for each of the code. 

Let's say for 3 incoming client_codes. I want to execute each out of two child pipeline 3 times (so six parallel executions). Following is screenshot of parent pipeline.

 

 

6 Replies

    • pranjbhatt's avatar
      pranjbhatt
      New Contributor III

      Hi, 

      I am calling the client_code as JSON Object  {"clients" : [{ "client_code": "OHSU"}, {"client_code":"scj"}]}
      in pipeline parameter before mapping, but it converts into a string before passing to JSON Splitter Snap. 

      This throws error. 

  • SpiroTaleski's avatar
    SpiroTaleski
    Valued Contributor

    pranjbhatt 

    Why you are passing the object as a pipeline parameter? 

    You should split the clients array, and then pass the array elements as inputs to the child pipelines. 

    • pranjbhatt's avatar
      pranjbhatt
      New Contributor III

      Hi SpiroTaleski 
      I am receiving client details and their tokens in pipeline parameters via airflow. They are in JSON object now, as follows- 

       

      {
          "clients": [
              {
                  "client_code": "OHSU",
                  "client_token": "xxxxxx"
              },
              {
                  "client_code": "scj",
                  "client_token": "xxxxxx"
              }
          ]
      }

       

       But, JSON Splitter snap still throws error, as pipeline parameters defaults it to String. Is there a way to parse above JSON before sending to child pipelines?

    • pranjbhatt's avatar
      pranjbhatt
      New Contributor III

      One last question, Is it possible to process all client_codes in parallel? My current setup is passing the JSON details using router to several pipeline execute commands that call API child execution parallel. But I want to call parallel per client basis as well. (parallel execution of each document)