Copy snap creating output that is multiplied by number of records

[clarrivey]

I have a pipeline that creates files based upon 490 records in an Oracle table. I create the output by doing a select on static headings that I create and union it with actual records from the table and then I sftp the file. I wanted to also be able to email the file, so snapLogic support suggested adding a Copy snap and sending one to sftp and the other to email, so I added a Copy snap right after the Oracle Execute snap that then went to a CSV Formatter snap and then to a File Writer snap to sftp. The copy also went to a CSV Formatter and then a File Writer to create a file in my directory and finally an email snap to send the file. All of this worked EXCEPT that the listing I received was 491 records (which included a header) TIMES 490!!! For whatever reason, it repeated the copy 490 times!!! Any ideas as to why???

[koryknick]

Thank you for the pipeline statistics - that is helpful to explain. I see that you already added the Tail snap in, but it looks like you added it after the Oracle Execute 3 snap, which I believe you are using to generate your file output including the header. You should add the Tail immediately following the Oracle Insert snap.

Here is why: The pipeline is reading from Oracle and writing to “NBRJOBS_new.csv” file, then reading a file and performing Oracle Inserts (one document at a time). Keep in mind that the Oracle Insert snap passes the update information for EACH document to the output view… so 490 document in is 490 document out of that snap. This means that the Oracle - Execute Update SWEADTE snap is being executed 490 times (once for each input document), and I don’t believe this is what you want.

By adding the Tail snap before that Oracle - Execute Update SWEADTE snap, you are limiting the number of documents going into that snap to 1, so the snap will execute only once.

This is an exercise in understanding the behavior of each snap in the pipeline. The documentation of the snaps should help with that.

[clarrivey]

Thank you Kory, that was VERY helpful. It is now working as expected. I am much obliged!

-Chris

[ForbinCSD]

@clarrivey – glad you finally solved the problem. I can’t tell from reading the above whether you ran afoul of one of Snaplogic’s fundamental “gotchas”, but one that I’ve seen happen with our team here is the danger of thinking of snaps as individual program steps instead of programs themselves.

Interestingly enough, this is a trap that experienced programmers seem to fall into more often than “citizen implementors” who don’t have a formal programming background.

It helps me to think of each snap as a separate program, each of which can be running in parallel, and the connection between each pair of snaps is a queue. A pile of documents get spit out of a query snap or other data-producing snap, and they conceptually sit in a queue as the next snap processes each in sequence and puts the output docs one by one into the queue for the snap that follows. It is a true pipeline and there can be different docs found in different parts of the pipeline being processed simultaneously.

I’ve found this mental model to be particularly helpful in avoiding the “rat in the snake” problem (lots of data in one big glob that can only get through the pipeline one snap at a time, when there’s no need for it to do so). It also helps me to reason about decisions in the pipeline where it actually does need to wait on all the data to arrive so it can aggregate things.

Hope This Helps!