While it is not possible to individually break the document messages into separate individual files on the same pipeline, I dont understand why you want to increase the I/O resources for lets say 10000 messages.
It is plain and clear that writing so many messages into so many files require that many I/O streams to be opened on the network. It is very process intensive.
I am not getting your concept here. What is the reasoning?
Another way of implementing the Queue, is write the whole message to a database table.
id, message
1, JSON.stringify(jms.message1)
2, JSON.stringify(jms.message2)
3, JSON.stringify(jms.message3)
When you want to process them, you can individually read the message, parse it and do whatever you want to further process it. Hope it makes sense.
@nganapathiraju
the reasoning is because each one of the incoming message on the JMS queue is a large XML Sterling file containing EDI 850 Purchase Order. So when you use JMS consumer it will read all the messages on the queue and put it all into one binary file. So I had to break it because I have business requirements to operate on each message separately because each one contain an 850 PO that needs to be sent to EDW and vendors. The problem is if I use the ForEach execute pipeline to do it in order to operate on each one of them then I won’t be able to use the JMS Consumer and JMS Acknowledge for message assurance in case the message is lost. This is according to the reply received from support@snapplogic.com when we run into the issue of JMS acknowledge configuration. You can refer to it for further reading: (#18557) [GameStop] JMS Acknowledge Snap issue:
https://snaplogic.zendesk.com/hc/en-us/restricted?return_to=https%3A%2F%2Fsnaplogic.zendesk.com%2Fhc%2Fen-us%2Frequests%2F18557
Therefore the only way to use JMS acknowledge is with the same JMS consumer in the same pipeline not by using ForEach child pipeline however I need to use the ForEach in order to be able to operate on each document separately. To sum it up, I need a solution where I can break and write each JMS message to a single file while using JMS acknowledge for message assurance in the same pipeline.
Does that make sense?
Ok I get it.
Did you look at the Workaround I suggested. You can use DB snaps to write and the output of the DB snap will include the original content which can be used to acknowledge to JMS.
You can achieve that in one pipeline too.
The problem is if I use the ForEach execute pipeline to do it in order to operate on each one of them then I won’t be able to use the JMS Consumer and JMS Acknowledge for message assurance in case the message is lost
I think there is a way to handle this, ACK snap would look for JMSMessageID to ack or noack a message, so you can still use a parent/child pipeline where in Parent would pass
$JMSMessageID
$content
to a child as pipeline parameters and within child pipeline you’ll process these messages and if all messages are processed succesfully you’ll return the status message along with $JMSMessageID to the parent which will then ACK/NOACK a given message, check out attached pipelines, I am using SQS but the process remains same for your usecase too.
1.1-SNS-SQS-Subscriber_2017_06_16.slp (11.5 KB)
1.2-Process-SQS-Message_2017_06_16.slp (7.2 KB)
by the way could you please share how you have configured JMS for your requirements? I would like to document your JMS account configure so that we can reuse this knowledge for such future requests, you can remove your JMS server and port related details, I am interested to see JMS props along with jars used to connect to the broker, and if you cannot than no worries 🙂
@Bhavin
Thank you this what I ended up doing and the closest thing to my solution. I have one clarification though regarding using two JMS Acknowledge snaps (Acknowledge, Reject) instead of just one. Isn’t the other JMS Acknowledge snap for ‘Reject’ redundant? Why cannot we use just one Acknowledge snap and if the message processed successfully then it will be acknowledged but the message wasn’t processed and didn’t make it till the end then the message won’t be acknowledged therefore it will remain in the queue for future processing. I don’t see what difference would it make to add a Reject snap with Recover message box checked since the first Acknowledge snap can cover both scenario.
Regarding your request to share the JMS account configuration I will be more than happy to share it below 🙂
Isn’t the other JMS Acknowledge snap for ‘Reject’ redundant?
yes, mine was geared more towards “asthetics” 🙂
Regarding your request to share the JMS account configuration I will be more than happy to share it below
much obliged
