Forum Discussion

kumar25's avatar
kumar25
New Contributor II
2 years ago
Solved

Append input data vertically

Hi,

I have 3 input streams, like below sample

Input-1

ASERUFVH816783KGGVKLHO

KJGFYGCVJVJHJ7859444MBJHJHGFK

JBJGFR565675GGJFJFJGFKFFTDDK67

 

Input-2

KJGLFUDGCLHGIFGFKTDEDDXJGCCHG46533856

VHJFHTDGFXCGHCHKGDDDDHGCJVJGT6764468MVNVCHGFH

5444755687VHJJFFHGCVJGVJGF55764674

 

Input-3

HVLJFVBLGUFTFYFJVBKGFDDYDVHJBKJ44568459

BGJKFTYEDCJVHJGUFYYTESRFGCHVJHLHIGF455656

VHJFTRYTRFJHVJKFTYDDCGVJGLIYFYTEYD763653356

 

I have to join these vertically. When i am using Union Snap they are not getting added in a sequence, they are randomly added one after the other. And when i use Join Snap with Merge option, they are getting added horizontally.Can someone let me know how to append the input streams one below the other without changing the sequence.

  • koryknick's avatar
    koryknick
    2 years ago

    kumar25 - Do you just need something like the following?

    input-1 line1
    input-2 line1
    input-3 line1
    input-1 line2
    ...etc?

    I've attached an example pipeline how to accomplish this.  Basically, it's adding variables to track the input the data is coming from and the record number of each record.  Once the data is combined using Union, sort the data on the record number and input view.  Finally, remove the temporary values used for this process.  Note the use of the "Passthrough" option in the Mappers so I don't need to know what the record layout is.

    Hope this helps!

    PS - I appreciate Aleksandar_A 's contribution; however, I do warn against using the Gate snap with the "All input documents" setting.  If your input dataset is large, it can consume considerable resources on your execution nodes, causing other pipelines to pause to wait on resources, or worst case can crash the node depending on other activity.  Gate is a powerful snap and can be used very effectively, just remember:

     

     

  • Aleksandar_A's avatar
    Aleksandar_A
    2 years ago

    Just change the order of the Sort Paths in the Sort by recnum and input Sort Snap from the koryknick's pipeline.

     

4 Replies

  • Aleksandar_A's avatar
    Aleksandar_A
    Contributor III
    2 years ago

    Hello kumar25,

    You can try by introducing the Gate Snap to gather all inputs, followed by concatenating them using the array concat function within a Mapper Snap. Finally, split them using the JSON Splitter Snap.

    Below is a sample pipeline demonstrating this method.

    Take note of the snaplex memory usage, as the Gate Snap waits for all documents upstream.

    Please let me know if this helps you.

    Regards,

    Aleksandar.

  • koryknick's avatar
    koryknick
    Employee
    2 years ago

    kumar25 - Do you just need something like the following?

    input-1 line1
    input-2 line1
    input-3 line1
    input-1 line2
    ...etc?

    I've attached an example pipeline how to accomplish this.  Basically, it's adding variables to track the input the data is coming from and the record number of each record.  Once the data is combined using Union, sort the data on the record number and input view.  Finally, remove the temporary values used for this process.  Note the use of the "Passthrough" option in the Mappers so I don't need to know what the record layout is.

    Hope this helps!

    PS - I appreciate Aleksandar_A 's contribution; however, I do warn against using the Gate snap with the "All input documents" setting.  If your input dataset is large, it can consume considerable resources on your execution nodes, causing other pipelines to pause to wait on resources, or worst case can crash the node depending on other activity.  Gate is a powerful snap and can be used very effectively, just remember:

     

     

    • kumar25's avatar
      kumar25
      New Contributor II
      2 years ago

      koryknick , I need data to be appended as below.

       

      input-1 line 1

      Input-1 line 2

      Input-1 line 3

      Input-2 line 1

      Input-2 line 2

      Input-2 line 3

      Input-3 line 1

      Input-3 line 2

      Input-3 line 3

      • Aleksandar_A's avatar
        Aleksandar_A
        Contributor III
        2 years ago

        Just change the order of the Sort Paths in the Sort by recnum and input Sort Snap from the koryknick's pipeline.