Thanks for the valuable inputs.
So do you mean if I keep pool size as 1 then it will initiate one instance of the pipeline as per the input document and it will not allow even start another instance which is a different pipeline?
What my understanding was if I send 5 input documents, the document has the name of the pipeline to be executed out of which 2 documents have p1-pipeline and 3 has p2-pipeline. Here I would like both p1 and p2 to start running parallel but process their inputs in sequence.
so is this how it should it work? If not can you help me how can I enable such processing? I tried the below,
I am extracting documents from DB and after sorting them as per pipeline name and timestamp sending it to pipeline execute with “reuse” enabled,
here is what my pipeline execute will receive as input, and below is the configuration,
I get error in pipeline execute
{error:Invalid configuration, stacktrace:com.snaplogic.snap.api.SnapDataException: Invalid configuration\n at com.snaplogic.snaps.flow.PipeExec.process(PipeExec.java:722)\n at com.sna...}
"error": "Invalid configuration"
"stacktrace": "com.snaplogic.snap.api.SnapDataException: Invalid configuration\n at com.snaplogic.snaps.flow.PipeExec.process(PipeExec.java:722)\n at com.snaplogic.snaps.flow.PipeExec.processSafely(PipeExec.java:699)\n at com.snaplogic.snaps.flow.PipeExec.execute(PipeExec.java:628)\n at com.snaplogic.snaps.flow.PipeExec.executeForSuggest(PipeExec.java:1273)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeSnap(SnapRunnableImpl.java:677)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.executeForSuggest(SnapRunnableImpl.java:556)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.doRun(SnapRunnableImpl.java:735)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.access$000(SnapRunnableImpl.java:105)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl$1.run(SnapRunnableImpl.java:330)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl$1.run(SnapRunnableImpl.java:326)\n at java.security.AccessController.doPrivileged(Native Method)\n at javax.security.auth.Subject.doAs(Subject.java:422)\n at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)\n at com.snaplogic.cc.snap.common.SnapRunnableImpl.run(SnapRunnableImpl.java:325)\n at com.snaplogic.snap.threadpool.SnapExecutorService$SnapRunnableWrapper.run(SnapExecutorService.java:86)\n at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n at java.util.concurrent.FutureTask.run(FutureTask.java:266)\n at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n at java.lang.Thread.run(Thread.java:745)\n"
"reason": "The pipeline path changed between documents when reuse was enabled"
"resolution": "Change the pipeline expression so that the same value is returned for every input document"
"snap_details": {label:Pipeline Execute}
"label": "Pipeline Execute"
"original": {RUNID:3716be8b-aa2b-4158-be89-792302c2a570, PIPELINENAME:CheckRunID, PAYLOAD:{pipelineName=P_NewExceptionTest}, ERRORTYPE:Technical, ERRORTIMESTAMP:2017-09-07T08:19:05.883Z, EXTRAP...}
"RUNID": "3716be8b-aa2b-4158-be89-792302c2a570"
"PIPELINENAME": "CheckRunID"
"PAYLOAD": "{pipelineName=P_NewExceptionTest}"
"ERRORTYPE": "Technical"
"ERRORTIMESTAMP": "2017-09-07T08:19:05.883Z"
"EXTRAPARAMETERS": "NONE"
Awaiting for your inputs.
You have to be careful with the reuse checkbox.
From documentation:
Parameters
Pipeline parameter values can only be changed if this flag is not enabled. In other words, reusable executions cannot have different pipeline parameter values for different documents.
Pipeline name
When this property is an expression, the Snap will need to contact the SnapLogic cloud servers to load the pipeline information for execution. Also, if reuse is enabled, the result of the expression cannot change between documents.
As you can see from the error, it has a problem with the expression value being changed when the reuse is enabled.
This is happening for the 4th document in your case.
“reason”: “The pipeline path changed between documents when reuse was enabled”
“resolution”: “Change the pipeline expression so that the same value is returned for every input document”
“snap_details”: {label:Pipeline Execute}
“label”: “Pipeline Execute”
What is your use case here? What are you trying to achieve?
The result I want to achieve is as shown in the output view of “sort” snap the documents are sorted as per pipelinename and timestamp this input will be sent to pipeline execute and we want to have 3 instances each for unique pipelines and these instances should process the payload filed one by one.
Example,
if I send 5 input documents, the document has the name of the pipeline to be executed out of which 2 documents have p1-pipeline and 3 has p2-pipeline. Here I would like both p1 and p2 to start running in parallel but process payload in sequence.
Can you run it without the reuse checkbox and see if that is what you want to achieve?
