koryknick
Employee
Joined 9 years ago
User Profile
User Widgets
Contributions
Re: Inserting large data in servicenow
deepanshu_1 - You can relieve some of the memory requirement off your snaplex nodes by removing the Group By N and JSON Splitter snaps and use the Batch Size option in the Pipeline Execute snap instead. This accomplishes exactly the same result without the memory consumption of combining large sets of records into a single document as an array. If you are still experiencing slowness, you can follow-up with your ServiceNow admins to see if anything can be done on that side. I believe that ServiceNow is not really meant for batch operations, so inserting millions of records into ServiceNow is probably your bottleneck.Re: How can I pop the last element from one array and append it to another?
adam_gataev - you have the right terminology. Just use a Mapper as follows: Here I'm just using two Array methods: concat() and pop() to complete your goal in one expression. The Array.pop() removes the last element and returns that element. The Array.concat() creates a new array with the new element added in. Note that I'm also using the "pass-through" option on the Mapper settings. This allows any other elements in the input doc to flow through to the target path without specifying them. Hope this helps!Re: Combine CSV document into a new file
marenas - I would not recommend using Gate to combine the data files - it can cause excessive memory consumption for very large files since the data has to be stored completely in memory. I recommend the attached approach. The trick here is in the Mapper on the bottom path and the second input view added to the CSV Parser. If you review the documentation, you will see that the second input view allows you to specify a header and also datatypes, if you choose. I simply added the header in the Mapper. Then in the Union, it combines the data in the way you are looking for. One thing to note is that Union will take documents as they come from each input view - meaning in this case that if both CSV Parsers are sending a large volume of records, you will see them intermixed - it does not wait for all of the documents on the first path before consuming the documents from the second path. There are easy fixes for this, but thought I would mention it in case it is a requirement that the data ordering be preserved between the input files. Hope this helps!Re: Trying to 'flatten' array using a mapper and pull out values
Auroth - I believe what you are looking for is: jsonPath($, "$tdline_array..TDLINE") jsonPath is a tool that lets you perform powerful scans through JSON. In this case, I'm using the descent operator (..) to search for TDLINE regardless of where it exists in the JSON document. If you're already comfortable with Object and Array functions, this is your complement to those expressions. Hope this helps!
