ksalem
New Contributor III
Joined 8 years ago
User Profile
User Widgets
Contributions
Pipeline Execute Pool size
I am trying to use the pipeline execute snap to launch child pipelines. currently the throughput of this is relatively slow, to improve this I would like to increase the pool size to allow multiple childs to run at the same time however regardless of changing this setting it seems the run continues to only process one document at a time. for example please see screenshots below: In the screenshots above i would expect 10 documents to be on the input section, with 10 instances of the DELME pipeline running, however only 1 is running at a time?SQS -pipeline stalling
Hi All, I am facing a peculiar issue with a new pipeline being built. the pipeline is setup to consume an SQS message > apply some processing and auditing(Via MySQL DB) > acknowledge the SQS message The pipeline runs correctly when the SQS Consumer Snap is set to message count 1. however the intention is for this pipeline to run permanently and constantly consume messages from SQS queue. when setting the message count property to -1 the pipeline does appear to work correctly until it reaches the first audit step where a 'MySQL Execute' is intended to insert an audit entry into a table (just a basic insert). in this instance the pipeline reaches this snap and indicates an input document of 1 but no output document or error document. (The MySQL execute is set to pass through and is not set to ignore empty results, so an output of some kind should always be expected): see attached screenshot I was unable to understand why this works when the SQS message count brings in exactly 1 message, but when set to -1 seems to fail silently here. the pipeline continues to run and further messages are picked up by the sqs but they are always held up at the MySQL step. Many thanks, Kareem.Re: Filter expression evaluate against all input documents
Thanks @cjhoward18 I did initially look at this kind of solution however not all secretary are given a position of secretary some may be assistant or receptionist. the other problem is that not all secretaries report to architects. If i was to write a script i would iterate through the document filtering firstly on architects and add them to a seperate list then if they have a secretery ID I would append those secretaries to that list. Im wondering if this can be done without the script snap though as I would prefer to use the out the box components to make the support aspect of this easier however it may be a case that script is the simplest way to go
