Forum Discussion

binupaul's avatar
binupaul
New Contributor II
8 years ago

Ultra pipeline guaranteed delivery

I am evaluating ultra pipelines for converting existing pipelines to ultra (with the architecture one unconnected input view AND one or more unconnected output views).
According to the press release:
“Ultra Pipelines also bring guaranteed delivery of data requests. SnapLogic accomplishes this by correlating inbound and outbound messages and ensuring that if a request fails, it is picked up immediately by another instance and restarted without performance delays. With Ultra Pipelines, guaranteed delivery runs across the entire transaction process.”

This is my understanding of the above statement:
Internally, incoming HTTP requests to the FeedMaster end-point are turned into documents (inbound messages) and fed to the ‘Ultra’ pipeline task instances via the FeedMaster queue. The FeedMaster queue is the embedded ActiveMQ queue within the FeedMaster. Queued messages in the feedmaster queue are persisted. Messages taken from the FeedMaster queue (outbound messages) are delivered in a continuous fashion to instances of the ultra task (which are running on the other nodes, not the feedmaster node). Messages still in the queue will be delivered to the task instances if the feedmaster went down and was restarted. If it were not possible for a task instance to be started with the outbound message item from the queue, for some reason (i.e. ‘request failure’), retry will happen.
By ‘correlation’, my understanding is that messages that have entered the queue will be processed, guaranteed, because the implementation keeps track of messages that have not been processed and those that have been handed over to the task instances.

Please let me know if my understanding is correct. Also is there a way to test this out? Is there a way to inspect the embedded ActiveMQ queue?

ultra

1 Reply

  • nganapathiraju's avatar
    nganapathiraju
    Former Employee
    8 years ago

    Here is what correlation means for Ultra

    For every message or document that is processed in an Ultra pipeline, there is a Correlation ID that is maintained and needs to be maintained and if that ID is lost, the documents loses its parent lineage and therefore cannot be further processed in the next downstream Ultra compatible snap.

    There are only certain snaps that are Ultra compatible and the remaining snaps are either incompatible or compatible with restrictions. All of these are documented in the doc.snaplogic.com

    Those that are incompatible can be moved to a child pipeline and can be called using a pipeline execute.

    The main premise is that ULTRAs process one document at a time so you can not split and do whatever you want.

    Hope that makes sense.