Unable to handle echo message

[JensDeveloper]

Hi all,

I’m encountering this error for the first time but we have a ultra task api pipeline running.
And today when Iwas testing out the api it gave the error message:

Somehow knows what that means.
I also saw in the dashboard that we sometime lose connection to the nodes so I found on the community.

Lost contact with Snaplex node Designing Pipelines

I have a very small pipeline (3 snaps) that I’m reading from a SQL table and writing the results to a new table in the same DB. I keep getting an error: Lost contact with Snaplex node while the pipeline was running. The select statement is pulling 250+M records and I’m using the Azure bulk insert to write. In order to avoid this error I keep having to reduce the batch size, from 20K to 10K to now 8K. Any thoughts on what could be causing the error?

But i already have implemented a pipeline execute. and the CPU usage is very low

someone knows the first error?

Regards

Jens

[JensDeveloper]

Hi everyone,

Update on the topic. After long investigation and checking every snap, documentation and ultra task support. I found out that there was nothing wrong with my pipeline but that the API portal can’t handle certain request even when it’s not so big. Because I tested all my API get,put, post methods in the browser with the request url and they all worked and instant response.

Conclusion: I think that the API portal has some issues regarding api calls getting lots of data or multiple requests. I will close my topic tomorrow.

Also If anyone has experience with the API and API portal/ Ultra task of SnapLogic. I’m all ears to learn more.

Regards

Jens

[ptaylor]

Can you please share some specifics about what you tried? It’s hard to help without any specifics.

[JensDeveloper]

Hi @ptaylor

So I will try me best to scheme my discovery.
Below my pipeline as ultra task enabled.

To summarize:

• First it will see which method it is and route it to GET/PUT/POST or OTHER
• After doing all my get requests and put/post request everything was fine
• Until i Discovered that SQL select returns per record one document
• For the json formatters it’s mandatory to enable ‘Format each document’ in ULTRA tasks
• Then I saw for my two get methods that get an array from SQL select it only showed the first record
• So i Investigated on the comunnity and came accross this topic: Merge multiple documents to an array in ultra pipeline - #19 by thilkip
• Here I implemented a pipeline execute. But when I finished creating the pipeline execute it always gave a response:
  
• Next I tried to find something online about it. But I only could find that it was possible the the node couldn’t allocate enough memory or the CPU was to high. I checked the dasboard it only used like 1 percent CPU and less than 10mb Memory but I did for sure in the SQL select. Staging mode: in memory.
• then after researching all that I kinda was thinking why haven’t I tried the request URL in the browser.
• Somehow then all my API call requests worked and I get an instant response with the corresponding format that i made with the pipeline and pipeline execute.

I hope this explains it a bit more. If you would like we can also have a call in the future so I can explain more detailed step by step what I did.

But maybe there is another problem in my pipelines that causes the portal to give that response. Don’t know yet. My apoligize if my thinking or answer is wrong.

Note: I’m not so experienced in efficient pipeline creation for APIs still learning. Open for suggestions in the future.

[ptaylor · Accepted solution]

Thanks, Jens. The image of your pipeline is too low-res for me to be able to read the snap names. But I think I’m hearing that the crux of the problem is that your SQL Select returns multiple records so you’re trying to deal with that in your Ultra pipeline.

The key to doing that is to use a child pipeline and move the SQL Select to that pipeline, along with whatever snaps are needed to aggregate those multiple documents and return a single document as the result of the child pipeline. Then in the parent, do any additional processing needed on that child output document to create a single response document for this request. This works because a new instance of the child pipeline is created for every request, so you can use aggregating snaps there that you can’t use in the root Ultra pipeline. Make sense?