How to write each document to a file rather than all documents to a single file

I am reading multiple messages using JMS consumer into one big binary file then using a binary to document snap I am breaking it into multiple documents. How can I write each document to a separate file with timestamp rather than writing all the documents at the end to one file?

I know there is a way to do that using ForEach pipeline where you can send each document seperately but I don’t want to use a master pipeline and a child pipeline. I want all operations to be done in one single pipeline.

@mohamadelmardini

While it is not possible to individually break the document messages into separate individual files on the same pipeline, I dont understand why you want to increase the I/O resources for lets say 10000 messages.

It is plain and clear that writing so many messages into so many files require that many I/O streams to be opened on the network. It is very process intensive.

I am not getting your concept here. What is the reasoning?

Another way of implementing the Queue, is write the whole message to a database table.

id, message
1, JSON.stringify(jms.message1)
2, JSON.stringify(jms.message2)
3, JSON.stringify(jms.message3)

When you want to process them, you can individually read the message, parse it and do whatever you want to further process it. Hope it makes sense.

@nganapathiraju
the reasoning is because each one of the incoming message on the JMS queue is a large XML Sterling file containing EDI 850 Purchase Order. So when you use JMS consumer it will read all the messages on the queue and put it all into one binary file. So I had to break it because I have business requirements to operate on each message separately because each one contain an 850 PO that needs to be sent to EDW and vendors. The problem is if I use the ForEach execute pipeline to do it in order to operate on each one of them then I won’t be able to use the JMS Consumer and JMS Acknowledge for message assurance in case the message is lost. This is according to the reply received from support@snapplogic.com when we run into the issue of JMS acknowledge configuration. You can refer to it for further reading: (#18557) [GameStop] JMS Acknowledge Snap issue:
https://snaplogic.zendesk.com/hc/en-us/restricted?return_to=https%3A%2F%2Fsnaplogic.zendesk.com%2Fhc%2Fen-us%2Frequests%2F18557

Therefore the only way to use JMS acknowledge is with the same JMS consumer in the same pipeline not by using ForEach child pipeline however I need to use the ForEach in order to be able to operate on each document separately. To sum it up, I need a solution where I can break and write each JMS message to a single file while using JMS acknowledge for message assurance in the same pipeline.

Does that make sense?

Ok I get it.

Did you look at the Workaround I suggested. You can use DB snaps to write and the output of the DB snap will include the original content which can be used to acknowledge to JMS.

You can achieve that in one pipeline too.

The problem is if I use the ForEach execute pipeline to do it in order to operate on each one of them then I won’t be able to use the JMS Consumer and JMS Acknowledge for message assurance in case the message is lost

I think there is a way to handle this, ACK snap would look for JMSMessageID to ack or noack a message, so you can still use a parent/child pipeline where in Parent would pass

$JMSMessageID
$content

to a child as pipeline parameters and within child pipeline you’ll process these messages and if all messages are processed succesfully you’ll return the status message along with $JMSMessageID to the parent which will then ACK/NOACK a given message, check out attached pipelines, I am using SQS but the process remains same for your usecase too.

1.1-SNS-SQS-Subscriber_2017_06_16.slp (11.5 KB)
1.2-Process-SQS-Message_2017_06_16.slp (7.2 KB)

by the way could you please share how you have configured JMS for your requirements? I would like to document your JMS account configure so that we can reuse this knowledge for such future requests, you can remove your JMS server and port related details, I am interested to see JMS props along with jars used to connect to the broker, and if you cannot than no worries :slight_smile:

@Bhavin
Thank you this what I ended up doing and the closest thing to my solution. I have one clarification though regarding using two JMS Acknowledge snaps (Acknowledge, Reject) instead of just one. Isn’t the other JMS Acknowledge snap for ‘Reject’ redundant? Why cannot we use just one Acknowledge snap and if the message processed successfully then it will be acknowledged but the message wasn’t processed and didn’t make it till the end then the message won’t be acknowledged therefore it will remain in the queue for future processing. I don’t see what difference would it make to add a Reject snap with Recover message box checked since the first Acknowledge snap can cover both scenario.

Regarding your request to share the JMS account configuration I will be more than happy to share it below :slight_smile:

Isn’t the other JMS Acknowledge snap for ‘Reject’ redundant?
yes, mine was geared more towards “asthetics” :slight_smile:

Regarding your request to share the JMS account configuration I will be more than happy to share it below
much obliged

1 Like

Getting back to your original question, you can write separate documents with the JSON-Formatter, XML-Formatter, and DocumentToBinary snaps. You can then configure the FileWriter snap with an expression for the file name, for example:

'out_' + Date.now() + '.json'

Or, if you just want a sequence number, you can use the ‘snap.in.totalCount’ variable:

'out_' + snap.in.totalCount + '.json'

So, every document generated by the formatter snap will be written to its own file with the name computed by the expression.

To configure the formatters:

  • JSON Formatter - Select the ‘Format each document’ option.
  • XML Formatter - Clear the ‘Root element’ field. Note that the input document should only have a single field that represents the root element.
  • Document To Binary - Set ‘Encode or Decode’ to DOCUMENT.

@mohamadelmardini For breaking up each document from JMS queue we are following the same method as they suggested above,do we still having issue breaking up the documents ?also the JMS acknowledge we were able to acknowledge the messages based JMS Message ID as suggested above only thing we didn’t do is having the reject acknowledge .

Sandeep the original solution wasn’t working when I tested it out. I updated the approach and it is working now. You can check it out at DEV environment.

All is good now.