Merging output of one document in next

I am working on a pipeline where I am reading data from CSV generator.

I have 2 types of data H line and I line for each one I need to do some mapping, which is fine. the final output I need to sent to SAP with a format H + I

So I need to merge the H line to each I line and then post to SAP.
Can someone help me achieve this? As all the values are in an independent document I am not able to do it easily. I tried Join and Union followed bu Group by but still not getting desired result.

Desired result:

/KS

Is there a key that you can use to join the data? If not could you add an artificial key and then join the two documents?

I tried using merge operation to add sequence but I get the error,

Snap has the following errors. Please open the info box and edit the highlighted fields.
Failure: Invalid JSON-path: $group[*].DOCUMENTHEADER, Reason: Field not found in JSON object: DOCUMENTHEADER, Resolution: Check the path syntax

I tried to use join operation, the output is

How to merge the documents?

Do H and I refer to Header and Line data?

Do you only have one H and multiple I lines?

If so, copy the incoming stream and process them independently and merge join them after you are done processing.

Hope it makes sense.

Let us know if this is not what is expected.

Thanks for the input.

Yes H and I are for header and line items.
I have done similar thing where I am routing H and I to different mapper and processing them individually. Once done I did a merge join and getting one H+I and I in 2 documents. I want H+I+I in one document.

/KS

1 Like

My understanding about the question is you have a csv with a header and line(s). you are processing them differently and after that you want the data in CSV style first line is header and rest is content (line/s).

If so I would like to suggest two different solutions :

  1. use a json splitter right after the CSV snap. you can configure it as:
    image

this can take sometime during insertion but each line will have a header


  1. after doing union you can use a script snap. Here is my jython (python) code. You can use JS or ruby:

#Import the interface required by the Script snap.
from com.snaplogic.scripting.language import ScriptHook
import java.util

class TransformScript(ScriptHook):
def init(self, input, output, error, log):
self.input = input
self.output = output
self.error = error
self.log = log

#The “execute()” method is called once when the pipeline is started
# and allowed to process its inputs or just send data to its outputs.
def execute(self):
self.log.info(“Executing Transform script”)
wrapper = java.util.ArrayList()

   while self.input.hasNext():
       try:
           # Read the next document, wrap it in a map and write out the wrapper
           in_doc = self.input.next()
         
           #wrapper['original'] = in_doc
           
           header = in_doc.get('DOCUMENTHEADER')
           
           line_acc = in_doc.get('ACCOUNTGL')
           line_cur = in_doc.get('CURRENCYAMOUNT')
           
           if (header != None):
              wrapper.add(0, header)
           if (line_acc != None):
               wrapper.add(line_acc)
           if (line_cur != None):
               wrapper.add(line_cur)

       except Exception as e:
           errWrapper = {
               'errMsg' : str(e.args)
           }
           self.log.error("Error in python script")
           self.error.write(errWrapper)
           
   self.output.write(in_doc, wrapper)
   self.log.info("Finished executing the Transform script")

#The Script Snap will look for a ScriptHook object in the “hook”
#variable. The snap will then call the hook’s “execute” method.
hook = TransformScript(input, output, error, log)

the output will be an array which has header at 0th index follwed by lines

Hope this helps!

1 Like

Try doing this. Of course I used all dummy variables but here are the main steps.

  1. Read Incoming file into 2 parsers
  2. Read the first one with header H
  3. Read the second one with I
  4. Do any Transformations specific to H and I on that specific route
  5. On the I route, merge all of them into a single array
  6. Merge join them in the end
  7. After the join, use JSON splitter on the I and add each individual H item using the “Including Path”
  8. You can do further massaging of the data after the JSON splitter.

Look at this example.

H and I_2017_08_15.slp (8.5 KB)

I get this error,

/KS

Solved the issue and I can see one header and 2 line items in one document.
Thanks for the help.

1 Like

Can you kindly share the final solution

Output 1

Output 2

1 Like