How to write each document to a file rather than all documents to a single file

[mohamadelmardin]

I am reading multiple messages using JMS consumer into one big binary file then using a binary to document snap I am breaking it into multiple documents. How can I write each document to a separate file with timestamp rather than writing all the documents at the end to one file?

I know there is a way to do that using ForEach pipeline where you can send each document seperately but I don’t want to use a master pipeline and a child pipeline. I want all operations to be done in one single pipeline.

[nganapathiraju]

@mohamadelmardini

While it is not possible to individually break the document messages into separate individual files on the same pipeline, I dont understand why you want to increase the I/O resources for lets say 10000 messages.

It is plain and clear that writing so many messages into so many files require that many I/O streams to be opened on the network. It is very process intensive.

I am not getting your concept here. What is the reasoning?

Another way of implementing the Queue, is write the whole message to a database table.

id, message
1, JSON.stringify(jms.message1)
2, JSON.stringify(jms.message2)
3, JSON.stringify(jms.message3)

When you want to process them, you can individually read the message, parse it and do whatever you want to further process it. Hope it makes sense.

[mohamadelmardin]

@nganapathiraju
the reasoning is because each one of the incoming message on the JMS queue is a large XML Sterling file containing EDI 850 Purchase Order. So when you use JMS consumer it will read all the messages on the queue and put it all into one binary file. So I had to break it because I have business requirements to operate on each message separately because each one contain an 850 PO that needs to be sent to EDW and vendors. The problem is if I use the ForEach execute pipeline to do it in order to operate on each one of them then I won’t be able to use the JMS Consumer and JMS Acknowledge for message assurance in case the message is lost. This is according to the reply received from support@snapplogic.com when we run into the issue of JMS acknowledge configuration. You can refer to it for further reading: (#18557) [GameStop] JMS Acknowledge Snap issue:
https://snaplogic.zendesk.com/hc/en-us/restricted?return_to=https%3A%2F%2Fsnaplogic.zendesk.com%2Fhc...

Therefore the only way to use JMS acknowledge is with the same JMS consumer in the same pipeline not by using ForEach child pipeline however I need to use the ForEach in order to be able to operate on each document separately. To sum it up, I need a solution where I can break and write each JMS message to a single file while using JMS acknowledge for message assurance in the same pipeline.

Does that make sense?

[nganapathiraju]

Ok I get it.

Did you look at the Workaround I suggested. You can use DB snaps to write and the output of the DB snap will include the original content which can be used to acknowledge to JMS.

You can achieve that in one pipeline too.

[Bhavin]

The problem is if I use the ForEach execute pipeline to do it in order to operate on each one of them then I won’t be able to use the JMS Consumer and JMS Acknowledge for message assurance in case the message is lost

I think there is a way to handle this, ACK snap would look for JMSMessageID to ack or noack a message, so you can still use a parent/child pipeline where in Parent would pass

$JMSMessageID
$content

to a child as pipeline parameters and within child pipeline you’ll process these messages and if all messages are processed succesfully you’ll return the status message along with $JMSMessageID to the parent which will then ACK/NOACK a given message, check out attached pipelines, I am using SQS but the process remains same for your usecase too.

1.1-SNS-SQS-Subscriber_2017_06_16.slp (11.5 KB)
1.2-Process-SQS-Message_2017_06_16.slp (7.2 KB)

by the way could you please share how you have configured JMS for your requirements? I would like to document your JMS account configure so that we can reuse this knowledge for such future requests, you can remove your JMS server and port related details, I am interested to see JMS props along with jars used to connect to the broker, and if you cannot than no worries 🙂