I am evaluating ultra pipelines for converting existing pipelines to ultra (with the architecture one unconnected input view AND one or more unconnected output views).
According to the press release:
“Ultra Pipelines also bring guaranteed delivery of data requests. SnapLogic accomplishes this by correlating inbound and outbound messages and ensuring that if a request fails, it is picked up immediately by another instance and restarted without performance delays. With Ultra Pipelines, guaranteed delivery runs across the entire transaction process.”
This is my understanding of the above statement:
Internally, incoming HTTP requests to the FeedMaster end-point are turned into documents (inbound messages) and fed to the ‘Ultra’ pipeline task instances via the FeedMaster queue. The FeedMaster queue is the embedded ActiveMQ queue within the FeedMaster. Queued messages in the feedmaster queue are persisted. Messages taken from the FeedMaster queue (outbound messages) are delivered in a continuous fashion to instances of the ultra task (which are running on the other nodes, not the feedmaster node). Messages still in the queue will be delivered to the task instances if the feedmaster went down and was restarted. If it were not possible for a task instance to be started with the outbound message item from the queue, for some reason (i.e. ‘request failure’), retry will happen.
By ‘correlation’, my understanding is that messages that have entered the queue will be processed, guaranteed, because the implementation keeps track of messages that have not been processed and those that have been handed over to the task instances.
Please let me know if my understanding is correct. Also is there a way to test this out? Is there a way to inspect the embedded ActiveMQ queue?