Forum Discussion

swright's avatar
swright
New Contributor III
5 years ago
Solved

What is the best way to loop through data from a file... like an csv?

I have a csv file with person ID numbers in one column. I need to use the id number to act on person records in the cloud via REST.

What is the best way to setup a pipeline to get an ID, do some processing, then execute a REST snap, then go back and do the same for the next ID and continue looping through the IDs?

Thanks,
Scott

  • sg_sripathi's avatar
    sg_sripathi
    5 years ago

    @swright, if you refer to below pipeline. It will make two REST API call to two different end point (URL constructed using input) and with HTTP body content as you see in the example. So basically the REST snap will take care of processing each document for you.

11 Replies

  • pkk_1's avatar
    pkk_1
    New Contributor
    5 years ago

    Hi Scott,

    You can separate out the logic and processing in different pipelines.
    1] Pipeline 1 : Master/Parent pipeline that reads the csv file and extracts the records and pass ID as pipeline parameter to child Pipeline 2. If there are huge records, you can enable re use and configure the pool size as required.
    2] Pipeline 2: To act on the ID received in the pipeline parameter and execute REST snap

    Thank you

    • swright's avatar
      swright
      New Contributor III
      5 years ago

      Hi Pooja,

      I tried that with File Reader > CSV Parser > Sequence > Mapper > Pipeline Execute.

      It didn’t pass the parameter to the child pipeline. Is there something special that I need to do to send the parameter to the child?

      Thanks,

      Scott

      • pkk_1's avatar
        pkk_1
        New Contributor
        5 years ago

        Hi Scott,

        In Pipeline execute’s Parameter Name you need to set the ID(can be any name) and in the Parameter Value enable the expression and access the ID that you want to read.

        Do share the sample csv file.

        Thanks,
        Pooja

  • sg_sripathi's avatar
    sg_sripathi
    New Contributor III
    5 years ago

    @swright, not sure why you would require a complex setup. Is the processing complicated to be had in a separate pipeline. Most of the snaps process at document level, so you don’t have to loop, including REST snaps.

    • swright's avatar
      swright
      New Contributor III
      5 years ago

      Hi Sripathi,

      I’m new to Snaplogic. I would love to have a simple setup. The CSV parser snap produces a json file with a separate array element for each id. I need to create endpoints for each ID to and pass the endpoints to REST snaps. It seems like I need to execute the REST snap once for each url that is created using the PersonNumber from the csv. In many programing languages this would be done using a loop. It would take the first PersonNumber, get/create the endpoints for the REST snaps and run the REST snaps then move on to the next PersonNumber and do the same with each PersonNumber until there are no more PersonNumbers in the array.

      I don’t know how to do this. Are there any examples?

      Thanks,

      Scott

      • sg_sripathi's avatar
        sg_sripathi
        New Contributor III
        5 years ago

        @swright, if you refer to below pipeline. It will make two REST API call to two different end point (URL constructed using input) and with HTTP body content as you see in the example. So basically the REST snap will take care of processing each document for you.