Ultra Pipeline Functionality

I’ll try to answer your questions:

1. The transaction rate really depends on what you’re going to be doing in the pipeline. If you are contacting local database/REST servers as part of handling requests, you can probably sustain a pretty high rate. But, if the pipeline needs to talk to cloud services, you’ll be bounded by the amount of time it takes to communicate over the internet, which can be quite high.
   There is, of course, going to be overhead incurred by the ultra pipeline infrastructure. The request message is written to disk so that we can replay it in case of a node failure in the GroundPlex. There is an extra hop from the Feed-Master to the the GroundPlex node so that we can scale by adding more nodes and be resilient to a GroundPlex node failing. The overhead is relatively constant, though, so it should work if you can tolerate that cost.

2. Ultra Tasks are intended to be resilient to pipeline and whole GroundPlex node failures. The pipeline instances that will be handling requests are spread across nodes, so that if one goes down the other instances will continue processing. The pipeline instances on those failed nodes will then be respawned on other nodes to get you back to capacity.
   As you mention, pipeline design does play a part. The ultra pipeline instances should be able to survive an outage of the SnapLogic cloud servers, but if your pipeline is reading from or writing to SLFS, you’ll experience problems.

3. Not really. You will need a load-balancer to spread load across the Feed-Masters, so I think we would recommend setting up throttling on that when possible. The Feed-Master will start rejecting requests if it’s space for storing requests fills up.

4. Requests will queue up on the feed-master until they can be serviced by one of the ultra pipeline instances. A 503 will be returned by the Feed-Master if it has no more space for requests.

5. I’m not quite sure what you mean here. The document sent into the pipeline that contains most everything in the original request, so you should be able to do what you want.

6. The pipeline should be able to do all aspects of request handling, including authentication. For example, you can parse the ‘Authorization’ header and check against a DB to see if it’s valid, responding with a 401 if it’s not or passing the doc down the rest of the pipeline to do the real handling.