PayalS
New Contributor II
Joined 8 years ago
User Profile
User Widgets
Contributions
Re: Help with DIFF SNAP
@koryknick @Spiro_Taleski Thanks for your valuable comments 🙂 🙏 I just wanna share the good news that I am finally able to crack this now. PFB workarounds I did to make it a success: Updated Error pipeline to insert records in the DB. filtered data connection specific errors. Used exit snap with threshold value as ‘0’. Created a separate account with batch size as 1 for this error pipeline. By this, my error pipeline will stop the parent pipeline only for particular errors, and will continue the pipeline in all other data failures. Thanks, Payal Srivastava7.1KViews1like1CommentRe: Help with DIFF SNAP
Thnx @koryknick I tried this as well. This was the very recent thing I did. I made the batch size to 1 thinking it will reach to EXIT snap making my pipeline stop. But, unfortunately, this also dint work. I dont know if I am missing anything or Snaplogic is behaving weird. Because, in the logs, I could see record is flowing to the exit snap, but still not stopping the pipeline 😢1.9KViews0likes2CommentsRe: Help with DIFF SNAP
@koryknick I am using error table to log the error. I tried using Oracle insert to log the error, then used filter for data connection specific error description and then used Exit snap, so that as soon as any data connection error logged into the table it will reach to exit causing pipeline to stop. But, this also doesn’t seems to be working as EXIT is not causing pipeline to fail. Its waiting for entire stream to get processed and by that time, records from source B reaching to DIFF and thus making it to the deletion flow 🙁 Not accomplishing my task 😢1.9KViews0likes4CommentsRe: Help with DIFF SNAP
@koryknick This is already unchecked in my design but still not working, as its again stopping that link from Source A. But again, Source B continuing sending the data to DIFF and ultimately to delete link, thus causing emptying B. ☹️ koryknick: Another option may be to un-check the “Ignore empty result” and put a Router that looks for an empty result set. If empty, send to the Exit snap; otherwise process as usual. I tried multiple approaches, but not getting quick fix to it. I ultimately want to send a STOP signal to my pipeline in case of any source connection issue, but after logging the error in the error table.1.9KViews0likes6CommentsRe: Help with DIFF SNAP
Thnx @koryknick for your reply. Much appreciated 🙏 I tried this by explicitly filtering the connection errors, after inserting in the Oracle Snap, and putting a exit snap after this. But, this is only stopping that flow, not stopping the parent pipeline because of which Source B continuing sending the data to DIFF and thus deleting all the data from B. koryknick: You can examine the error details in your Error Pipeline and call the Exit snap immediately if it is a connection error.1.9KViews0likes0CommentsRe: Help with DIFF SNAP
Thanks Spiro for the quick reply. Much Appreciated. 🙏 Yeah, I too thought the same approach, but then in our project we do have a total count of 70 pipelines (which are already created and we identified this bug later). Now, with this change, we will have to modify all 70 pipelines and along with that we will have to create another set of 70 pipelines. Therefore, I was thinking if we could have any alternate approach to fix the same. May be by just modifying the error pipeline or may a small fix which can be done only to already created 70 pipelines.3.3KViews0likes2CommentsHelp with DIFF SNAP
Hi, I am looking for some help regarding DIFF Snap. Below is the scenario, I am working on: I have two Source DBs - A & B. DIFF snap is being used to identify the eligible data for Insert, Update and Delete, and accordingly operations performed in Source B as Target DB. e.g. Lets say 10 records are there in A 20 records are there in B then DIFF will identify 10 records for deletion from B. Now, the problem I am facing is, if due to any data issue or connection failure, if no records are coming from Source A, then DIFF is identifying all records from B to flow in the deletion link, which eventually deleting all the data from B. I dont want this to happen. I want my pipeline to stop/fail, if any such error happend in the source, so that target cannot be empty in any case. I am using Error Pipeline, which is tracking the error fine but then not stopping the pipeline. Is there a way, I can track the error and stop the pipeline in such scenarios of connection failure or Data Issue ? Quick Help will be really appreciated. Thanks in Advance, Payal SrivastavaSolved8.7KViews0likes14Comments